- Is this your first time deploying Airbyte: No
- OS Version / Instance: GCP
- Memory / Disk: 16Gb
- Deployment: Docker
- Airbyte Version: 0.35.56-alpha
- Source name/version: postgres 0.4.9
- Destination name/version: Bigquery
- Step: Deduped+history syncing
- Description:
We鈥檙e using airbyte 0.35.56-alpha with postgres source connector 0.4.9 with CDC to replicate to bigquery as the destination.
We鈥檙e observing the same replication problem as - 馃悰 CDC MySQL (Deduped+history) to Bigquery: deleted_at field is not getting updated for deleted records. 路 Issue #8830 路 airbytehq/airbyte 路 GitHub, where if a row is deleted in source, the destination continues to display the row in the replicated table. The <table_name>_scd shows _is_airbyte_active column which reflects the deletion, which indicates that there鈥檚 a bug in the transformation logic that builds the main table.This is a blocker for us and would be happy to open a PR, if pointed in the right direction. Any advice?
Hi @ameyabhope, I encourage you to open a similar issue as the one you linked but specific to Postgres. This issue will be a better place to have a detailed implementation discussion with our connector team. The related code is probably in this python file: airbyte-integrations/bases/base-normalization/build/lib/normalization/transform_catalog/stream_processor.py
Been trying to make the change myself, had quick question -
I set everything up locally and I think I need to run these steps to test locally (according to the developing locally doc)-
./gradlew airbyte-integrations:bases:base-normalization:build
VERSION=dev docker-compose up
This doesn鈥檛 seem to generate the build dir under airbyte-integrations/bases/base-normalization/. Any suggestions?
@ameyabhope did you follow this doc to work locally on the normalization?
Hi @alafanechere, I did.
I made a small change to one of the print commands to start off.
Ran the unitTests which were successful. Followed it by the :build step and eventually ran docker-compose up in dev mode. Still no luck getting my changes on the UI.
I guess the question is that would I need to change something other than print command to see if local development env is working?
After a bit more investigation I think that you need to:
- Change the version of normalization version to
dev in this file
- Rebuild the platform images with:
SUB_BUILD=PLATFORM ./gradlew build
If you want to disable the test run on platform build you can use SUB_BUILD=PLATFORM ./gradlew build -x test which will make the build run faster.
Thanks @alafanechere, I finally got my local dev setup working.
I tried a bunch of things without much luck. Could you tag @edgao here (I can鈥檛 seem to do it), so that we can colab and work on this?
Cool!
Please sync with @edgao directly on the GitHub issue 
This was fixed please updated to Airbyte 0.39.20+ version.
Hi there from the Community Assistance team.
We鈥檙e letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you鈥檙e having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we鈥檒l follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.