Implementing incremental sync from PostgreSQL via CDC without primary key to S3 with Full refresh sync mode

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

This message is asking how to implement incremental sync from PostgreSQL via CDC to S3 when the table does not have a primary key but has REPLICA IDENTITY = full, and the sync mode is set to Full refresh.


Question

How can I implement incremental sync from PostgreSQL via CDC? I have a table without a primary key, but with REPLICA IDENTITY = full , and target is S3. And I have Full refresh only in Sync mode.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["incremental-sync", "postgresql", "cdc", "s3", "full-refresh", "replica-identity"]
2

I think CDC makes sense only database-like destinations

3

it sends only changes

4

I create Postgres - to Postgres connection now and same result. Only Full refresh in replication settings

5

with CDC you need to do a first full refresh before it start sending only changes

6

Let me double check this, got too many things going

7

first full sync is complete but incremental sync not available

8

• sorry, my bad “Incremental sync is only supported for tables with primary keys. For tables without primary keys, use <https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-overwrite|Full Refresh sync>.
”

9

from here https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#cdc-requirements

10

<@U06AQE1UMMH> Thanks!