Why my connection in MySQL to BigQuery truncated in 1m rows
logs-22653.txt (1.0 MB)
my connector version in 0.6.0
1 Like
Hi @ayyoub, this looks to be a known issue with the MySQL source connector when trying to sync with millions of records with high variance in record size: Source Mysql: syncing timeout · Issue #9784 · airbytehq/airbyte · GitHub.
Feel free to follow the discussion and contribute a fix if you’re able to.
@sajarin so this issue still not solved ? because for different table that has 1 millions it runs properly, maybe this is an anomaly
@ayyoub, I don’t believe there’s a solution yet, it’s a bit difficult to reproduce since it only seems to happen with large syncs and many rows. Sometimes it works for some tables but not for others. Could you share some more information about the resources you’ve allocated for your airbyte instance? Could you also share some information about the differences between the tables that have synced properly and the tables that haven’t properly? Furthermore, it may be possible to divide up the full refresh sync into multiple connectors, each syncing a few tables, all to the same destination. Have you considered this route?