MinaSession WriteTimeoutException for MySQL to BigQuery sync

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu 20.04
  • Memory / Disk: 128Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: What version are you using now? 0.40.2
  • Source name/version: MySql 0.6.5
  • Destination name/version: BigQuery 0.2.0
  • Step: Initial Sync

During the initial sync for a MySQL (CDC/Tunnel) → BigQuery connection with google storage staging and no destination table normalization:

  • the Incremental / Deduped + history sync fails repeatedly for a large MySQL table.
  • If I reset and uncheck the large table, the sync completes successfully.
  • If I do only the large table using Full Refresh append, it fails.
  • If I do only the large table using Incremental / Deduped + history, it also fails.

There are records written to the _airbyte_raw_$LARGE_TABLE bigquery table (although I lost track of when they were emitted).

I can’t disclose the full log because it’s security sensitive. I assume this must be a timeout issue somewhere because repeated runs failed at roughly the same time. But the click-to-download logs have not been helpful. For the last run (only large table using Incremental / Deduped + history, the one that seems most clearly relevant is,

2022-08-27 16:51:24 e[44msourcee[0m > 2022-08-27 16:51:23 e[33mWARNe[m o.a.s.c.u.l.LoggingUtils(warn):610 - exceptionCaught(MinaSession[local=/, remote=/]) WriteTimeoutException: null

which preceded by,

2022-08-27 16:51:27 e[44msourcee[0m > 2022-08-27 16:51:26 e[1;31mERRORe[m i.d.p.ErrorHandler(setProducerThrowable):35 - Producer failure

3 seconds and a few lines earlier. The sync for this table started with,

2022-08-27 16:36:27 e[32mINFOe[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):339 - Records read: 1000 (789 KB)

which was exactly fifteen minutes earlier.

There is no setting on the mysql server side that matches 15 minutes.

If you encounter this page and it is relevant to you the problem was the SSH tunnel. It works fine if I create it locally with port forwarding to the MySQL server then put the docker network in host mode. In the UI it always eventually failed.

Hey could you create a github issue so that team can look into it

Thanks for creating the issue! link here: https://github.com/airbytehq/airbyte/issues/17346