Issue with MySQL to BigQuery connection missing records during full refresh sync

Summary

Issue with a small table (<20k records) ingested via full refresh in MySQL to BigQuery connection where random records fail to ingest into the destination. Manual re-sync resolves the issue temporarily.


Question

Hi all,

We have a MySQL –> BigQuery connection that ingest several tables set to sync every 2h. Some of them are ingested via Full refresh | Overwrite and others with Incremental | Append + Deduped. We have an issue with a small table (<20k records) ingested via full refresh that from time to time fails to ingest some random records into the destination for seemingly no reason. After a manual re-sync missing records appear again. Has anyone experienced something similar?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["mysql-connector", "bigquery-connector", "full-refresh", "missing-records", "sync-issue"]