I’m trying to sync a stream with nested array data but the final table isn’t dedup.
Basic normalization doesn’t apply de-dedup in nested data.
See issue Destination MSSQL duplicate array records when using incremental append (nested tables) · Issue #9465 · airbytehq/airbyte · GitHub
As a workaround you can export the dbt project built by Airbyte and edit the nested table generation doing the dedup.
Is this behavior valid for all connectors? I am currently using Airbyte with the Shopify connector and can indeed observe this behavior that I didn’t expect when choosing the sync mode “Incremental | deduped + history”.
What is the reason of not deduplicating nested data? I don’t remember having to deal with this when using Fivetran.
Hello Jaafar, look your issue is similar to https://github.com/airbytehq/airbyte/issues/9465
Currently you need to dedup nested records by yourself probably exporting the normalization project and executing the dedup.
The main reason of this is: