Postgres to S3 CDC Sync Error

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User encounters a ConfigErrorException related to replication slot’s confirmed LSN during CDC sync from Postgres 16 replica to S3 after multiple executions.


Question

Hi , I set Airbyte to transfer data from Postgres 16 replica to S3 bucket with CDC. The 1st sync work ok but after 2-3 executions (3-5 mins pass between the execution) I’m getting the following error:

"internalMessage" : "io.airbyte.commons.exceptions.ConfigErrorException: Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency to prevent this from happening in the future
I set the WAL files to be unlimited , so it not look like its WAL files missing is the issue , Is anyone encounter this issue?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['postgres-16', 's3-bucket', 'cdc', 'configerrorexception', 'replication-slot', 'wal-retention']
2

What values do you have for max_slot_wal_keep_size, max_wal_size, wal_keep_size ?

3

max_slot_wal_keep_size = -1
max_wal_size - 2GB
wal_keep_size - 400G

4

Hmm, configuration looks good. Have you searched Slack or Github issues?
I think there were some issues like this.

5

No, not yet , i will