MinaSession WriteTimeoutException for MySQL to BigQuery sync

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu 20.04
  • Memory / Disk: 128Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: What version are you using now? 0.40.2
  • Source name/version: MySql 0.6.5
  • Destination name/version: BigQuery 0.2.0
  • Step: Initial Sync

During the initial sync for a MySQL (CDC/Tunnel) → BigQuery connection with google storage staging and no destination table normalization:

  • the Incremental / Deduped + history sync fails repeatedly for a large MySQL table.
  • If I reset and uncheck the large table, the sync completes successfully.
  • If I do only the large table using Full Refresh append, it fails.
  • If I do only the large table using Incremental / Deduped + history, it also fails.

There are records written to the _airbyte_raw_$LARGE_TABLE bigquery table (although I lost track of when they were emitted).

I can’t disclose the full log because it’s security sensitive. I assume this must be a timeout issue somewhere because repeated runs failed at roughly the same time. But the click-to-download logs have not been helpful. For the last run (only large table using Incremental / Deduped + history, the one that seems most clearly relevant is,

2022-08-27 16:51:24 e[44msourcee[0m > 2022-08-27 16:51:23 e[33mWARNe[m o.a.s.c.u.l.LoggingUtils(warn):610 - exceptionCaught(MinaSession[local=/127.0.0.1:37121, remote=/127.0.0.1:46620]) WriteTimeoutException: null

which preceded by,

2022-08-27 16:51:27 e[44msourcee[0m > 2022-08-27 16:51:26 e[1;31mERRORe[m i.d.p.ErrorHandler(setProducerThrowable):35 - Producer failure

3 seconds and a few lines earlier. The sync for this table started with,

2022-08-27 16:36:27 e[32mINFOe[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):339 - Records read: 1000 (789 KB)

which was exactly fifteen minutes earlier.

There is no setting on the mysql server side that matches 15 minutes.

2

If you encounter this page and it is relevant to you the problem was the SSH tunnel. It works fine if I create it locally with port forwarding to the MySQL server then put the docker network in host mode. In the UI it always eventually failed.

3

Hey could you create a github issue so that team can look into it

4

Thanks for creating the issue! link here: https://github.com/airbytehq/airbyte/issues/17346