Summary
The user is experiencing an issue where the MySQL to Snowflake sync is copying the entire dataset instead of using Change Data Capture (CDC) after migrating their Airbyte installation and backend database. They seek guidance on debugging this behavior.
Question
Mysql to Snowflake sync, copies the entire set of data despite using CDC on mysql source.
I have a custom open source Airbyte installation running on EKS and an external RDS database, I recently moved the Airbyte installation to a larger cluster and similarly migrated the backend RDS (Postgres) database to a different larger instance. I used pgdump and psql to copy over the data from the old RDS instance. Since the migration, multiple runs of Airbyte sync have reported extracting and copying over 11 million records on each run. How can I understand what is happening and debug the issue. Nothing has changed on the MySQL source, binlogs are still enabled and this worked sweet previously.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.