Summary
Data synchronization from Postgres to Citus using CDC is fast for most records but slows down significantly for the last few records. Any insights on why this might be happening?
Question
Hi All - Im Syncing data from postgres to Citus using CDC - Any idea why the last few records always take so long to complete? it syncs 800K records in like 2mins but starts to hang on like the last 8 records?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["syncing-data", "postgres", "citus", "cdc", "slow-completion"]
I See this error in the logs
the LSN will not be flushed to the database source and WAL logs will not be cleared. User is expected to handle this outside Debezium
Hi Gerrit - this isn’t an error but a DBZ output indicating WAL logs will be acknowledged by an independent process. Airbyte retains logs until there is confirmation of delivery from the destination connector. Only once this delivery is confirmed does Airbyte go back and acknowledge (i.e. clear) WAL logs
Interesting about the slow down at the end of your syncs though, do you have an example log file we can review?
Hi <@U03AM8G2WHG> thanks for taking a look - heres the logs:smiley:
Once again the connectors seems to hang as soon as it almost finishes - when i check the destination I need just 3 more records.
It seems like the connection hangs - and doesnt sync the last 5 records - but once i make a change in the source it goes through and finishes the sync