Hello @Clovis,
Thank you for this great question.
You are in a partial failure scenario. In this context some records are successfully committed to the destination and the sync fails afterward on other records. Airbyte does not perform a clean up of the already synced data when a failure happens.
In incremental sync, our connectors usually use the notion of cursor in the state to start the read from the place they stopped in the last sync. Hence, the more frequent the cursor is checkpointed the less you are prone to have duplicate data in your destination.
I suggest two approaches to work around your problem:
- If you use normalization you can choose the Incremental Dedupe mode that will remove duplicates in the destination
- What is your current cursor field? Maybe using a more fine-grained cursor field might increase the state checkpointing interval and reduce the duplicates.