Summary
Encountering memory issues during the normalization step when syncing data from MySQL source to Clickhouse destination using Incremental mode. Switching to Full Refresh resolves the issue, but Incremental mode restarts reading data from the beginning.
Question
Hello,
I am currently performing MySQL (Source) → Clickhouse (Destination) Sync. When performing the initial load, I encounter memory issues in Normalization Step
The MySQL source connector is utilizing CDC, and the sync mode is set to Incremental | append + deduped. When the sync mode is switched to Full Refresh, the loading process works fine. However, when I change the sync mode to Incremental and set the Transform method accordingly, it starts reading the data from the beginning again.
Could you assist with this?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.