Summary
The user is experiencing an issue where deleted records from the source PostgreSQL DB are not being removed in the destination AWS Redshift Data Warehouse when using Airbyte for incremental syncs with deduped. They are seeking information on whether Airbyte supports automatically deleting records in the destination table and the status of the issue mentioned in a GitHub comment.
Question
hello there, I have set up airbyte with several connections that sync data from a PostgreSQL DB to an AWS Redshift Data Warehouse via AIrbyte, I am using incremental syncs with deduped for that.
Today I found that in one of my destination tables there were records that had been removed from the source table. The sync had been working all along so it seems that deleted records in the source table are not deleted in the destination table. I have been investigating it a little bit and I found this <Incremental + dedup is not cleaning up deleted records · Issue #10140 · airbytehq/airbyte · GitHub issue> from 2022. I am not sure if this was implemented or not, and whether is there anything I can do to ensure that deleted records ofn source are removed in the destination. If this is not supported it’s really a big issue because I cannot ensure that both source and destination are really in sync. Does anyone know about the status of this?
Many thanks
PS: Deleting records on destinations is <Syncs | Hightouch Docs in Hightouch> for example
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.