MySQL Source: sporadic `java.io.EOFException: Failed to read remaining bytes` errors

alexnikitchuk · September 28, 2022, 8:29am

Is this your first time deploying Airbyte?: No
OS Version / Instance: Linux Ami 2
Memory / Disk: you can use something like 16Gb / 90 Gb
Deployment: Docker
Airbyte Version: 0.40.4
Source name/version: MySQL 0.6.8 (AWS RDS 8.0.mysql_aurora.3.02.0)
Destination name/version: Redshift 0.3.47 (S3 Staging)
Step: sync

Description: Sync sporadically starts to fail with the following error:

2022-09-28 00:12:25 e[1;31mERRORe[m i.d.p.ErrorHandler(setProducerThrowable):35 - Producer failure
io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1664298076000, eventType=TABLE_MAP, serverId=780484762, headerLength=19, dataLength=775, nextPosition=84654564, flags=0}
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1189) ~[debezium-connector-mysql-1.9.2.Final.jar:1.9.2.Final]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1234) [debezium-connector-mysql-1.9.2.Final.jar:1.9.2.Final]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:980) [mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599) [mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857) [mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1664298076000, eventType=TABLE_MAP, serverId=780484762, headerLength=19, dataLength=775, nextPosition=84654564, flags=0}
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:309) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:281) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:228) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:230) ~[debezium-connector-mysql-1.9.2.Final.jar:1.9.2.Final]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:952) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	... 3 more
Caused by: java.io.EOFException: Failed to read remaining 162 of 614 bytes from position 148781693. Block length: 162. Initial block length: 771.
	at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.fill(ByteArrayInputStream.java:115) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:105) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:48) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:27) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:281) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:228) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:230) ~[debezium-connector-mysql-1.9.2.Final.jar:1.9.2.Final]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:952) ~[mysql-binlog-connector-java-0.25.6.jar:0.25.6]
	... 3 more

The only way I managed to fix it is to reset data and start fresh sync. But after a while it fails again.
Any suggestion on how to deal with it?

Log is too big to attach here so here is the link to it

marcosmarxm · September 28, 2022, 3:02pm

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

harshith · September 28, 2022, 4:22pm

Hey after it fails was rerunning fix it?

alexnikitchuk · September 28, 2022, 4:45pm

hey @harshith nope, every rerun fails with the same exception. Only full refresh helps.

harshith · September 29, 2022, 6:02am

Got it. Can you create a issue in github so that team can look and get back to you?

alexnikitchuk · September 29, 2022, 8:11am

@harshith MySQL Source: sporadic `java.io.EOFException: Failed to read remaining bytes` errors · Issue #17372 · airbytehq/airbyte · GitHub

Topic		Replies	Views
Sync suddenly fails Connector Questions & Issues data-loading , connectors	2	354	September 16, 2022
Suddenly after two days some of the syncs are failing Connector Questions & Issues source-mysql , destination-bigquery	9	604	October 4, 2022
MySQL Source Issue Connector Questions & Issues source-mysql , getting-started , connectors	9	435	January 25, 2023
Failed to Connect MYSQL (Verification error(s) occurred for CHECK) Connector Questions & Issues source-mysql , mysql	3	351	December 21, 2022
Failing to ingest a "big" MySQL table Connector Questions & Issues source-mysql , destination-snowflake , data-loading , connectors , cdc	6	478	July 14, 2022

MySQL Source: sporadic `java.io.EOFException: Failed to read remaining bytes` errors

Related Topics