Summary
When syncing data from Redshift to MySQL using Full Refresh - Overwrite mode, the destination table was erased during a sync failure due to connection failure. Questioning if the data loss issue is related to the MySQL connector behavior or if there are other overlooked issues.
Question
Hi all, recently I encountered an issue that the destination table has been erased while the sync failed due to connection failure (authentication failed). I am syncing data from Redshift to MySQL using Full Refresh - Overwrite mode. I expect the data in the destination table will remain the same as it was in the last sync if there is any sync failure. During my research, I have found something mentioned in the Airbyte docs that some connector with Full-Refresh - Overwrite mode may indeed need to erase the existing data at the start of each full-refresh sync (https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/full-refresh-overwrite) but it doesn’t explicitly mention if MySQL connector falls in this case. I wanna know if the data lost issue is related to the connector OR I may have overlooked others issues. Thanks and much appreciated if anyone can give my some directions.
Verisons:
Airbyte: 0.50.19
MySQL: v0.2.0
Redshift: v0.4.0
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.