Summary
User is experiencing a socket exception while using the MySQL CDC connector for incremental data sync to S3. They inquire about skipping the initial snapshot for large tables and seek insights on the socket error encountered after prolonged execution.
Question
Hi All,
We were exploring the mysql CDC connector to sync data from a mysql table to s3 in cdc (incremental refresh manner).
As per the documentation, we checked for the first time it will perform the full dump (by executing select statements)
But while fetching we were facing the issue
java.net.SocketException
java.net.SocketException: Connection reset by peer```
details stacktrace is mentioned in reply on the same thread.
I have two queries :
1. Is there any way we can only pick cdc logs [like manually setting the offset in airbyte connection state?] and skip reading/making initial snapshot of the table (as some tables are really huge and reading data take more hours and we have other process to have one time dump of the tables, so don't want to do rework).?
Found a thread on the same <https://discuss.airbyte.io/t/syncing-huge-tables-10b-rows-from-mysql/3783>
<@U02T7NVJ6A3> Any idea plans to have this feature in the dev pipeline?
1. Any insights on the above error [we got the error after 10 hrs of execution on retry it again starts picking data from the start as per the logs]?
Any help would be much appreciated..
Thanks
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1707302760814389) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
['mysql-cdc-connector', 'incremental-refresh', 'socket-exception', 'initial-snapshot', 'airbyte-connection-state']
</sub>