Source MySQL: Incremental Deduped Sync Failure

Connector Questions & Issues
,
1

Hi Everyone,

I am facing an Incremental Sync Failure when syncing data from Mysql to Bigquery.

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Debian 11
  • Memory / Disk: 64Gb / 35GB
  • Deployment: GCP VM with Docker Compose
  • Airbyte Version: 0.39.35-alpha
  • Source name/version: MySQL (0.5.15)
  • Destination name/version: BigQuery (1.1.11)
  • Bigquery Loading Method: GCS Staging
  • BigQuery Client Chunk Size: 15MB
  • Step: Sync - Incremental Deduped History

Description:
Starting with I am pretty new to using Airbyte.
I setup a manual frequency refresh to sync data incrementally from Mysql source to Bigquery. The sync fails at the initial step where it needs to do the full replication of the source DB.

2022-07-15 20:49:11 source > 2022-07-15 20:49:11 ERROR i.d.c.m.AbstractReader(failed):219 - Failed due to error: Aborting snapshot due to error when last running 'SELECT * FROM `parkplus_fastag_issuance`.`fastag_passbook`': Streaming result set com.mysql.cj.protocol.a.result.ResultsetRowsStreaming@370f5040 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
2022-07-15 20:49:11 source > org.apache.kafka.connect.errors.ConnectException: Streaming result set com.mysql.cj.protocol.a.result.ResultsetRowsStreaming@370f5040 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries. Error code: 0; SQLSTATE: S1000

Also, I see few records which have been emitted have been inserted into the Bigquery destination partially but I see no committed records count in the error log message.
I let Airbyte run 3 attempts for the pipeline to see if it works.

image
image1222×299 43.7 KB

Attaching Attempt 3 log for reference. (edited out few logs in between to get the size below 8 MB)

Thanks !!

fpl-Attempt-3-logs.txt (7.5 MB)

2

Hi @mastersam987, thanks for your post. I think this is related to an issue with Debezium, there was a PR that was recently merged in master relating to this: https://github.com/airbytehq/airbyte/pull/14542

3

Thanks @sajarin , will look into it and update this.

4

Hi @sajarin, we have updated the connector but we are still facing the same issue.

5

Hi @mastersam987, thanks for reporting back. I’ve created an issue on github regarding this, please follow there for any updates. Hopefully, we can get this working for you soon.

6

Sure @sajarin, Thanks !!

7

@mastersam987 can you take a look at this issue?