Is it Possible to Stop Unnesting Repeated Records in Seperate Tables? (Shopify Connector)

Q&A
,
1

Hi everyone,

Is there a way to configure Airbyte so it does not unnest repeated records in separate tables for Shopify? I already have the code to do it and it would avoid the proliferation of tables. Thanks!

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Debian Linux 10
  • Memory / Disk:
  • 8 vCPUs (GCP c2-standard-8 - Intel Cascade Lake)
  • 32 Gib RAM
  • disk: 100 + 200 GB
  • Deployment: Docker
  • Airbyte Version: 0.35.5-alpha
  • Shopify source: 0.1.36
  • Bigquery destination: 0.6.12
  • Step: Setting up a new Shopify import to BigQuery
2

Today you need to have a custom dbt for that @louism. You can export the dbt normalization project and remove all nested tables from there only keeping the high level ones.
I created an issue in Github Normalization: unnest only first level · Issue #11979 · airbytehq/airbyte · GitHub to implement this feature.

1 Like
4

Thank you. Can you point me to some documentation as to where those normalizations reside and how to export them please?

5

I strongly recommend to you read the 3 parts tutorial about normalization:
https://docs.airbyte.com/operator-guides/transformation-and-normalization/

6

Thank you very much!