Summary
The MySQL binlog source connector in Airbyte Cloud stopped syncing due to a java.lang.OutOfMemoryError, despite not processing an unusually large number of records.
Question
I have a airbyte cloud mysql binlog source connector that was working for 30 days. It suddenly stopped syncing. I checked the logs and I found this entry:
2024-11-24 08:18:50 info Malformed non-Airbyte record (connectionId = 7b35d967-xxxx-xxxx-b5b2-afce95522ec3): Terminating due to java.lang.OutOfMemoryError: Java heap space
However, looking at the failed sync log summary, it wasn’t trying to sync an unusually large amount of records.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
['mysql-binlog', 'sync-failure', 'out-of-memory', 'airbyte-cloud']
here is what happened immediately before:
2024-11-21 22:33:49 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 371.22 MB, allocated: 10 MB (10.0 MB), %% used: 0.026938294469231416 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
914
2024-11-21 22:33:49 destination INFO main i.a.c.i.d.a.AsyncStreamConsumer(start):89 class io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer started.
915
2024-11-21 22:33:49 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
916
2024-11-21 22:34:13 source INFO debezium-engine i.a.c.r.c.CdcPartitionReader$EventConsumer(accept):203 Shutting down Debezium engine: record indicates that WAL consumption has reached the target position.
917
2024-11-21 22:34:13 source INFO debezium-close i.d.e.EmbeddedEngine(stop):961 Stopping the embedded engine
918
2024-11-21 22:34:13 source INFO debezium-close i.d.e.EmbeddedEngine(stop):968 Waiting for PT10S for connector to stop
919
2024-11-21 22:34:13 source INFO debezium-engine i.d.e.EmbeddedEngine(stopTaskAndCommitOffset):769 Stopping the task and engine
920
2024-11-21 22:34:13 source INFO debezium-engine i.d.c.c.BaseSourceTask(stop):432 Stopping down connector
921
2024-11-21 22:34:14 source ERROR Nov 22, 2024 3:34:14 AM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run
922
2024-11-21 22:34:14 source ERROR : threadExecutor is shut down, terminating keepalive thread
923
2024-11-21 22:34:14 source INFO debezium-mysqlconnector-db-change-event-source-coordinator i.d.p.ChangeEventSourceCoordinator(streamEvents):325 Finished streaming
924
2024-11-21 22:34:14 source INFO blc-abc.db.io:3306 i.d.c.b.BinlogStreamingChangeEventSource$ReaderThreadLifecycleListener(lambda$onDisconnect$0):1206 Stopped reading binlog after 0 events, last recorded offset: {ts_sec=1732246418, file=mysql-bin-changelog.155738, pos=42002222, server_id=976564898, event=2}
925
2024-11-21 22:34:14 source INFO debezium-engine i.d.p.s.SignalProcessor(stop):127 SignalProcessor stopped
926
2024-11-21 22:34:14 source INFO debezium-engine i.d.s.DefaultServiceRegistry(close):105 Debezium ServiceRegistry stopped.
927
2024-11-21 22:34:14 source INFO pool-4-thread-1 i.d.j.JdbcConnection(lambda$doClose$4):983 Connection gracefully closed
928
2024-11-21 22:34:14 source INFO pool-5-thread-1 i.d.j.JdbcConnection(lambda$doClose$4):983 Connection gracefully closed
929
2024-11-21 22:34:14 source INFO debezium-engine i.a.c.r.c.CdcPartitionReader$ConnectorCallback(taskStopped):264 Debezium Task stopped
930
2024-11-21 22:34:14 source INFO debezium-engine o.a.k.c.s.FileOffsetBackingStore(stop):71 Stopped FileOffsetBackingStore
931
2024-11-21 22:34:14 source INFO debezium-engine i.a.c.r.c.CdcPartitionReader$ConnectorCallback(connectorStopped):256 Debezium connector stopped
932
2024-11-21 22:34:14 source INFO debezium-engine i.a.c.r.c.CdcPartitionReader$CompletionCallback(handle):240 Debezium engine has shut down successfully: Connector ‘io.debezium.connector.mysql.MySqlConnector’ completed normally.
933
2024-11-21 22:34:14 source INFO DefaultDispatcher-worker-2#global-round-1-partition-1-read i.a.c.r.c.CdcPartitionReader(run):115 Debezium Engine has shut down and relinquished control, summary: {debezium-version=3.0.0.Final, records-emitted=1045, records-discarded=0, heartbeats=6, tombstones=0, events=1051, events-without-source-record=0, source-records-without-position=0, event-values-without-position=0, close-reason=RECORD_REACHED_TARGET_POSITION}.
934
2024-11-21 22:34:14 source INFO DefaultDispatcher-worker-2#global-round-1-partition-1-read i.a.c.r.FeedReader(readPartitionWithResources):203 completed reading partition 1 for ‘global’ in round 1
935
2024-11-21 22:34:14 source INFO DefaultDispatcher-worker-2#global-round-1-partition-1-read o.a.k.c.s.FileOffsetBackingStore(start):63 Starting FileOffsetBackingStore with file /tmp/airbyte-debezium-state11483320900476066707/offset.dat
936
2024-11-21 22:34:14 source INFO DefaultDispatcher-worker-2#global-round-1-partition-1-read o.a.k.c.s.FileOffsetBackingStore(stop):71 Stopped FileOffsetBackingStore
937
2024-11-21 22:34:28 source INFO Malformed non-Airbyte record (connectionId = 7b35d967-xxxx-xxxx-b5b2-afce95522ec3): Terminating due to java.lang.OutOfMemoryError: Java heap space