Implementing incremental sync from PostgreSQL via CDC without primary key to S3 with Full refresh sync mode


This message is asking how to implement incremental sync from PostgreSQL via CDC to S3 when the table does not have a primary key but has REPLICA IDENTITY = full, and the sync mode is set to Full refresh.


How can I implement incremental sync from PostgreSQL via CDC? I have a table without a primary key, but with REPLICA IDENTITY = full , and target is S3. And I have Full refresh only in Sync mode.

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["incremental-sync", "postgresql", "cdc", "s3", "full-refresh", "replica-identity"]

I think CDC makes sense only database-like destinations

it sends only changes

I create Postgres - to Postgres connection now and same result. Only Full refresh in replication settings

with CDC you need to do a first full refresh before it start sending only changes

Let me double check this, got too many things going

first full sync is complete but incremental sync not available

• sorry, my bad “Incremental sync is only supported for tables with primary keys. For tables without primary keys, use <|Full Refresh sync>.

from here

<@U06AQE1UMMH> Thanks!