Summary
Can Airbyte be used to move unstructured data from Google Cloud Storage to S3 with limited destination output formats?
Question
Hello Team, I was wondering if we can use Airbyte to move unstructured data. I have a requirement to move files(pdfs, htmls etc.) from Google Cloud Storage to S3. But looks like destination output format supports one of CSV, JSON, Parquet, AVRO only
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["move-unstructured-data", "google-cloud-storage", "s3", "airbyte-connector", "destination-output-format"]
Hi, Can someone please help here?
Hello Ravi, at the moment you can consume some unstructured data from some sources like Google Drive but the connector will transform the unstructured data to an Airbyte Message (id, data) and escapsulate the data inside a Json object and send to your destination. It looks you’re looking only to move the fils to from A to B, is that right?
Yes. Thats right. ATM my goal is to see if I can move documents across cloud storages as is. I also have requirement for structured data movement(my primary requirement), but that works for sure in Airbyte