I am not 100% sure though if the cursor field is timemodified as I am using CDC and don’t have the option of selecting the cursor field.
Interestingly, this link describes that the source does not necessarily need a suitable cursor field.
“On the other hand, CDC incremental replication reads a log of the changes that have been made to the source database and transmits these changes to the destination. Because changes are read from a transaction log when using CDC, it is not necessary for the source data to have a suitable cursor field.”
I’ve also tested the same connection on a bit older Airbyte 0.39.19-alpha instance with source MySql 0.5.11, destination Snowflake 0.4.28, and the incremental loading works.
The incremental syncs are still failing after upgrading to Airbyte 0.40.10. Connectors are the latest versions.
The initial historical sync also succeeded after failing once.
Here is the historical sync log that failed: (/tmp/workspace/46/0) historical-0-fail.log (4.1 MB)
Here is the historical sync log that succeeded: (/tmp/workspace/46/1) historical-1-success.log (7.6 MB)
Note: I’ve removed some “Records read” logs to reduce size.
Here is the incremental sync log that fails: failure-logs.log (5.0 MB)
This looks like it is failing because of the StandardSyncInput which contains the full catalog and the state. It is hard to estimate what will be the size for this specific connection. The state is only added on the second sync, which is why the initial sync works.
Could you try a few things? These could be fixes in the short term, but we will work on making a longterm fix.
Split the tables into 2 or more connections so as not to hit the maximum message size.
Tweak the dynamic.config and build temporal with a bigger BlobSize
I have about 65 tables of interest that are about 21 Gb in size.
To split the tables into 2 or more connections, I have selected only a few tables that add up to 1.74 Gb, which I believe is small, yet the incremental sync keeps failing.
I have a critical critical issue. The connections are passing but there is a huge data integrity issue.
The counts of the records are significantly off. Please could you and the team look into this urgently.
Thanks @alexnikitchuk I’ve updated to the latest version and I no longer get that error
@natalyjazzviolin I can’t seem to replicate the previous issue, but after updating to the latest version the incremental syncs still aren’t fully reliable
I’ve found an issue for the latest error you’re encountering. Looks like it happens sporadically just like in your case, and the only current fix is to reset the data and start a new sync, unfortunately. https://github.com/airbytehq/airbyte/issues/17372
My suggestion would be to update the MySQL source to 1 or higher! As the user in the issue is also on a 6.x version.
We stick to one issue per forum post for documentation purposes. If you’d like to discuss this further, please start a new thread!
Sorry to hear it’s still happening! I’ll ask one of the engineers on the databases team for their input and will post in the GitHub thread as well when I get new info!