Is this your first time deploying Airbyte?: No
OS Version / Instance: Ubuntu
Memory / Disk: 16Gb / 200GB
Deployment: Docker deployment
Airbyte Version: 0.39.20-alpha
Source name/version: airbyte/source-postgres/0.4.27
Destination name/version: airbyte/destination-redshift/0.3.46
Step: The issue is happening during sync (normalization)
Description:
I’m using the CDC method to ingest from Postgres to Redshift, it is working as expect at first
At one point airbyte detect less and less changed data and replication slot accumulate & consumed all the storage results in crashed the Postgres database
Below is the replication slot in the database
Even there are 457GB in the replication slot waiting Airbyte to consume them, In the Airbyte UI although the job was succeeded but still no records wrote into Redshift and the size of replication slot didn’t decrease at all
The log of the succeed sync but don’t seems to consume replication slot:
airbyte_logs.txt (167.1 KB)
Sorry about that! Do you mind updating to latest version of Postgres Source connector (0.4.31), reset your data and sync from start?
The reason is:
We have discovered a compatibility issue in which users attempting to use the versions 0.4.27 or later of the Postgres source connector in incremental CDC mode with Airbyte versions v0.38.3-alpha through v0.39.17-alpha will result in incremental state not being updated, causing these connections to always sync from the currently stored state cursor. To resolve this, please upgrade to the latest version of Airbyte in which this issue is resolved. We apologize for any inconvenience!
yes I will try. upgrade to Postgres Source connector (0.4.31) how about the Airbyte version? is there any recommended that go well with the Postgres Source connector (0.4.31) and Resshift Destination connector? The reason I ask is because the version I used now (0.39.20-alpha) doesn’t include in versions v0.38.3-alpha through v0.39.17-alpha which will result in incremental state not being updated.
Meanwhile could you kindly share the link to the issue as reference? thanks!
I think your current version of Airbyte should work but you need to upgrade the connector version to latest
