Summary
The user is facing an issue where the Mysql to Snowflake sync is copying the entire set of data despite using CDC on the Mysql source. The issue started after migrating Airbyte installation and backend RDS database to larger instances. The user wants to understand what is happening and how to debug the issue.
Question
Mysql to Snowflake sync, copies the entire set of data despite using CDC on mysql source.
I have a custom open source Airbyte installation running on EKS and an external RDS database, I recently moved the Airbyte installation to a larger cluster and similarly migrated the backend RDS (Postgres) database to a different larger instance. I used pgdump and psql to copy over the data from the old RDS instance. Since the migration, multiple runs of Airbyte sync have reported extracting and copying over 11 million records on each run. How can I understand what is happening and debug the issue. Nothing has changed on the MySQL source, binlogs are still enabled and this worked sweet previously.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.