Issue with MySQL to Redshift connection data loss during sync

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User is experiencing data loss during sync from MySQL to Redshift despite setting tables to ‘full refresh’ mode.


Question

Hey! I have a problem with mysql -> redshift connection. What I am trying to do is pretty simple, I want to refresh few tables from small, well structure mysql database. The problem I have is that the first sync is going well, and the next ones are losing some data even though all tables are set to “full refresh” mode.

Anyone face this issue?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["mysql", "redshift", "connection", "data-loss", "sync", "full-refresh"]
2

I can also see that I am not the only one facing this issues:
https://discuss.airbyte.io/t/data-loss-issue-with-full-refresh-overwrite-sync-mode/7194

3

Any idea how to resolve it?

4

I don’t know what connector and airbyte versions you’re using, but could it possibly be related to this…?

Running a Full Refresh stream with a Resumable Full Refresh Source on pre-0.58.0 versions will cause all jobs after the first job to behave as Incremental.
https://airbytehq.slack.com/archives/C01A4CAP81L/p1714154634283469