Incremental sync successful but rows in nested table are Duplicated

I’m trying to sync a stream with nested array data but the final table isn’t dedup.

Basic normalization doesn’t apply de-dedup in nested data.
See issue Destination MSSQL duplicate array records when using incremental append (nested tables) · Issue #9465 · airbytehq/airbyte · GitHub

As a workaround you can export the dbt project built by Airbyte and edit the nested table generation doing the dedup.

Hi @marcosmarxm,

Is this behavior valid for all connectors? I am currently using Airbyte with the Shopify connector and can indeed observe this behavior that I didn’t expect when choosing the sync mode “Incremental | deduped + history”.

What is the reason of not deduplicating nested data? I don’t remember having to deal with this when using Fivetran.


Hello Jaafar, look your issue is similar to
Currently you need to dedup nested records by yourself probably exporting the normalization project and executing the dedup.
The main reason of this is:

nested streams can’t be deduped like their parents (as they may not share the same cursor/primary keys)