S3 destination out not human-readable

hey, we’re currently using the s3 destinations to export our data from postgres to s3 as csv. The file output however is not really human-readable. In the postgres destination you actually can use dbt to transform your data. In s3 this is not supported but rather you can flatten the data blob to multiple columns which however fails for complex tables. What’s the best practice in case to do it? Won’t it better if dbt would be allowed in this case?

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 4GB
  • Deployment: Docker

Hey @datitran,
DBT relies on the computational power of the underlying datawarehouse. S3 is not a data warehouse, hence it’s not possible to use DBT on top of it. Could you share an example of the problematic CSV files you get with your current setup?

Hmm ideally it would make sense to have some kind of transformation layer but yeh I understand it that this was not your goal of the s3 destinations which basically was just meant to dump everything to the data lake. I can’t share examples as they are customer data but yeh we just want to normalize the data and just get rid off all airbyte specific columns like _airbyte_ab_id and _airbyte_emitted_at

Unfortunately this kind of transformation is not available within Airbyte. As you wrote, Airbyte is meant to dump raw data in your data lake, an additional transformation layer would be required to transform the output in the way you expect.