Sync of Deleted records in source to the target datasource

Connector Development
1

I have the following scenario and would like to understand how airbyte connectors handle it.

  1. Connection created between a source say MYSQL db to target POSTGRE db, to syn one table data.
  2. Initial sync loaded all records from the source to the destination table.
  3. Subsequent incremental syncs fetch the changed data into the destination table.
  4. Lets say few rows or records deleted from the source table.
    How does Airbyte connector handles these deleted rows from source to the destination so that both source and destination are in sync ?

Was it taken care or should we do anything from our end?

2

I @rnncredit,
If you use incremental sync, and in the ELT philosophy, the records that were already synced but then deleted from the source are kept in the destination.
If you use incremental sync + CDC on your source the records will be flagged as deleted in your destination.
If you can’t use CDC and really need the destination to mirror the source your only option is to use the full refresh overwrite sync mode.

3

@alafanechere Thanks for the response. Happened to read about some sources like mysql, postgre which supports CDC. Just curious to know about other source connectors like Salesforce which supports incremental, how does this connector specifies or inform about deleted records.

4

REST API can only inform of deleted records if they expose a flag field about the deleted status of records rather than not returning it. This is usually not the case and it’s not something our connector can handle because their only source of truth is the source API, not the destination. In other word, a connector never queries a destination and compare it to the source data to know if a record was deleted or not.

5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.