Incremental sync successful but rows in nested table are Duplicated

Troubleshooting
#1

I’m trying to sync a stream with nested array data but the final table isn’t dedup.

#2

Basic normalization doesn’t apply de-dedup in nested data.
See issue Destination MSSQL duplicate array records when using incremental append (nested tables) · Issue #9465 · airbytehq/airbyte · GitHub

As a workaround you can export the dbt project built by Airbyte and edit the nested table generation doing the dedup.

#3

Hi @marcosmarxm,

Is this behavior valid for all connectors? I am currently using Airbyte with the Shopify connector and can indeed observe this behavior that I didn’t expect when choosing the sync mode “Incremental | deduped + history”.

What is the reason of not deduplicating nested data? I don’t remember having to deal with this when using Fivetran.

Thanks!

#4

Hello Jaafar, look your issue is similar to https://github.com/airbytehq/airbyte/issues/9465
Currently you need to dedup nested records by yourself probably exporting the normalization project and executing the dedup.
The main reason of this is:

nested streams can’t be deduped like their parents (as they may not share the same cursor/primary keys)