S3 destination out not human-readable

Connector Questions & Issues
,
1

hey, we’re currently using the s3 destinations to export our data from postgres to s3 as csv. The file output however is not really human-readable. In the postgres destination you actually can use dbt to transform your data. In s3 this is not supported but rather you can flatten the data blob to multiple columns which however fails for complex tables. What’s the best practice in case to do it? Won’t it better if dbt would be allowed in this case?

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 4GB
  • Deployment: Docker
2

Hey @datitran,
DBT relies on the computational power of the underlying datawarehouse. S3 is not a data warehouse, hence it’s not possible to use DBT on top of it. Could you share an example of the problematic CSV files you get with your current setup?

3

Hmm ideally it would make sense to have some kind of transformation layer but yeh I understand it that this was not your goal of the s3 destinations which basically was just meant to dump everything to the data lake. I can’t share examples as they are customer data but yeh we just want to normalize the data and just get rid off all airbyte specific columns like _airbyte_ab_id and _airbyte_emitted_at

4

Unfortunately this kind of transformation is not available within Airbyte. As you wrote, Airbyte is meant to dump raw data in your data lake, an additional transformation layer would be required to transform the output in the way you expect.

5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.