CDC Postgres (Deduped+history) to Bigquery: deleted_at field is not getting updated for deleted records

  • Is this your first time deploying Airbyte: No
  • OS Version / Instance: GCP
  • Memory / Disk: 16Gb
  • Deployment: Docker
  • Airbyte Version: 0.35.56-alpha
  • Source name/version: postgres 0.4.9
  • Destination name/version: Bigquery
  • Step: Deduped+history syncing
  • Description:

We鈥檙e using airbyte 0.35.56-alpha with postgres source connector 0.4.9 with CDC to replicate to bigquery as the destination.
We鈥檙e observing the same replication problem as - 馃悰 CDC MySQL (Deduped+history) to Bigquery: deleted_at field is not getting updated for deleted records. 路 Issue #8830 路 airbytehq/airbyte 路 GitHub, where if a row is deleted in source, the destination continues to display the row in the replicated table. The <table_name>_scd shows _is_airbyte_active column which reflects the deletion, which indicates that there鈥檚 a bug in the transformation logic that builds the main table.This is a blocker for us and would be happy to open a PR, if pointed in the right direction. Any advice?

Hi @ameyabhope, I encourage you to open a similar issue as the one you linked but specific to Postgres. This issue will be a better place to have a detailed implementation discussion with our connector team. The related code is probably in this python file: airbyte-integrations/bases/base-normalization/build/lib/normalization/transform_catalog/stream_processor.py

Thanks, added CDC Postgres (Incremental | Deduped+history) to BigQuery: deleted_at field is not getting updated for deleted records. 路 Issue #11805 路 airbytehq/airbyte 路 GitHub

Been trying to make the change myself, had quick question -
I set everything up locally and I think I need to run these steps to test locally (according to the developing locally doc)-

./gradlew airbyte-integrations:bases:base-normalization:build
VERSION=dev docker-compose up

This doesn鈥檛 seem to generate the build dir under airbyte-integrations/bases/base-normalization/. Any suggestions?

@ameyabhope did you follow this doc to work locally on the normalization?

Hi @alafanechere, I did.
I made a small change to one of the print commands to start off.
Ran the unitTests which were successful. Followed it by the :build step and eventually ran docker-compose up in dev mode. Still no luck getting my changes on the UI.

I guess the question is that would I need to change something other than print command to see if local development env is working?

After a bit more investigation I think that you need to:

  • Change the version of normalization version to dev in this file
  • Rebuild the platform images with: SUB_BUILD=PLATFORM ./gradlew build

If you want to disable the test run on platform build you can use SUB_BUILD=PLATFORM ./gradlew build -x test which will make the build run faster.

Thanks @alafanechere, I finally got my local dev setup working.

I tried a bunch of things without much luck. Could you tag @edgao here (I can鈥檛 seem to do it), so that we can colab and work on this?

Cool!
Please sync with @edgao directly on the GitHub issue :pray:

This was fixed please updated to Airbyte 0.39.20+ version.