How to reflect schema changes in destination for Postgres to Postgres CDC?

Connector Questions & Issues
, , , ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: MacOS
  • Deployment: Local via docker-compose
  • Airbyte Version: What version are you using now?
  • Source name/version: Postgres/13.3
  • Destination name/version: Postgres/14.2
  • Step: The issue is observed during syncs of the Connection.
  • Description:

Hello all,

I am working with Airbyte locally through the provided Docker files and local setup documentation. I was interested in evaluating how the Postgres CDC operated, so I set up a Postgres to Postgres (through two additional Docker containers running the respective Postgres versions noted above) Connection in Airbyte.

One particular feature that is important for my use case is handling schema changes in the source and how to reflect in the target or destination. The Airbyte documentation seems to suggest that, even though schema changes cannot be detected automatically, that I should reset and resync the data through the expected flow conducted via the UI or similar.

However, upon attempting to do so, I find that I am unable to reflect column additions (further DDL changes such as column deletions were not evaluated yet) from the source in the target even when resetting and resyncing the data for a Connection.

The sync does not fail; the data is captured but the destination does not capture any new columns unless I apply one of two workarounds I’ve been able to use thus far:

  1. Completely set up a new Connection, specifically with a newly defined Source, so that the schema of the tables is re-inspected
  2. Manually update the catalog field for the Connection in the connection by connecting to the Airbyte meta DB directly and making the change.

Neither of these feels like the ideal flow to me. I thought at this point I would turn to the Airbyte community and ask how I should be approaching schema changes in a Postgres Source so that syncs can flow data as expected to a destination?

Any help or guidance on this topic would be much appreciated!

2

We do not support schema changes automatically for CDC sources. We recommend resetting and resyncing data if you make a schema change.

From Airbyte docs maybe it’s not the best instruction. You must go Replication Tab and Refresh Source Schema to capture the new fields consequently will reset your data.

3

Not sure how I missed the Refresh Source Schema! Thanks for the input; this got things working for me in short order.

4

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.