Users experiencing sync failures with MySQL source to BigQuery Destination after migrating to DestinationsV2
Hello! Has anyone encountered A LOT of issues since their migration to DestinationsV2 ?
I’ve been having sync failures with MySQL source x BigQuery Destination ever since it went into production
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
stacktrace not being super helpful either:
io.airbyte.cdk.integrations.source.relationaldb.state.FailedRecordIteratorException: java.lang.RuntimeException: java.sql.SQLException: Error retrieving record: Unexpected Exception: java.io.EOFException message given: Can not read response from server. Expected to read 10 bytes, read 2 bytes before connection was unexpectedly lost. Nested Stack Trace: ** BEGIN NESTED EXCEPTION ** java.io.EOFException MESSAGE: Can not read response from server. Expected to read 10 bytes, read 2 bytes before connection was unexpectedly lost. STACKTRACE: java.io.EOFException: Can not read response from server. Expected to read 10 bytes, read 2 bytes before connection was unexpectedly lost. at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:67) at com.mysql.cj.protocol.a.SimplePacketReader.readMessageLocal(SimplePacketReader.java:137) at com.mysql.cj.protocol.a.SimplePacketReader.readMessage(SimplePacketReader.java:102) at com.mysql.cj.protocol.a.SimplePacketReader.readMessage(SimplePacketReader.java:45) at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readMessage(TimeTrackingPacketReader.java:62) at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readMessage(TimeTrackingPacketReader.java:41) at com.mysql.cj.protocol.a.MultiPacketReader.readMessage(MultiPacketReader.java:66) at com.mysql.cj.protocol.a.MultiPacketReader.readMessage(MultiPacketReader.java:44) at com.mysql.cj.protocol.a.ResultsetRowReader.read(ResultsetRowReader.java:75) at com.mysql.cj.protocol.a.ResultsetRowReader.read(ResultsetRowReader.java:42) at com.mysql.cj.protocol.a.NativeProtocol.read(NativeProtocol.java:1651) at com.mysql.cj.protocol.a.result.ResultsetRowsStreaming.next(ResultsetRowsStreaming.java:194) at com.mysql.cj.protocol.a.result.ResultsetRowsStreaming.next(ResultsetRowsStreaming.java:62) at com.mysql.cj.jdbc.result.ResultSetImpl.next(ResultSetImpl.java:1813) at com.zaxxer.hikari.pool.HikariProxyResultSet.next(HikariProxyResultSet.java) at io.airbyte.cdk.db.jdbc.StreamingJdbcDatabase$toUnsafeStream$1.tryAdvance(StreamingJdbcDatabase.kt:90) at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.kt:42) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at io.airbyte.integrations.source.mysql.initialsync.MySqlInitialLoadRecordIterator.shouldBuildNextSubquery(MySqlInitialLoadRecordIterator.java:115) at io.airbyte.integrations.source.mysql.initialsync.MySqlInitialLoadRecordIterator.computeNext(MySqlInitialLoadRecordIterator.java:82) at io.airbyte.integrations.source.mysql.initialsync.MySqlInitialLoadRecordIterator.computeNext(MySqlInitialLoadRecordIterator.java:39) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.kt:42) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at io.airbyte.cdk.integrations.source.relationaldb.state.SourceStateIterator.computeNext(SourceStateIterator.kt:30) at io.airbyte.cdk.integrations.source.relationaldb.state.SourceStateIterator.computeNext(SourceStateIterator.kt:17) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.kt:42) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.kt:42) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.kt:67) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.kt:42) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.kt:67) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.kt:42) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.kt:42) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) at io.airbyte.cdk.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.kt:278) at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.kt:335) at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:184) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:116) at io.airbyte.integrations.source.mysql.MySqlSource.main(MySqlSource.java:646) ** END NESTED EXCEPTION **
Did you migrate recently, or has this problem been occurring for a while?
I migrated this morning. I also upgraded all my source connectors to make sure there wasnt any compatibility issue.
It appears to be happening with my mySQL connections
for some reasons, it would work without issue for a few syncs, then fail for a sync (14+ attempts).
cancelling / restarting / refreshing didn’t fix anything, and at some point it started succeeding again… (like 2min ago…)
Did you follow the migration guide to update your previous connections?
as you can see it’s pretty hit and miss as far as sync success…
here’s the current sync and the logs of attempt 5/6 (still ongoing) : Source process read attempt failed
Facing the same problem for mysql to bigquery connection
apparently downgrading the BQ connector to 2.6.3 fixes the issue. I’m waiting for my syncs to finish before testing this.
Found also this one but didn’t solve my issue: https://github.com/airbytehq/airbyte/issues/39900#issuecomment-2204555158
downgrading to 2.6.3 did fix the issue
It didn’t in my case, also facing problems with mysql sources