- Is this your first time deploying Airbyte: No
- OS Version / Instance: GCP
- Memory / Disk: 16Gb
- Deployment: Docker
- Airbyte Version: 0.35.56-alpha
- Source name/version: postgres 0.4.9
- Destination name/version: Bigquery
- Step: Deduped+history syncing
We’re using airbyte
0.35.56-alpha with postgres source connector
0.4.9 with CDC to replicate to bigquery as the destination.
We’re observing the same replication problem as - 🐛 CDC MySQL (Deduped+history) to Bigquery: deleted_at field is not getting updated for deleted records. · Issue #8830 · airbytehq/airbyte · GitHub, where if a row is deleted in source, the destination continues to display the row in the replicated table. The
_is_airbyte_active column which reflects the deletion, which indicates that there’s a bug in the transformation logic that builds the main table.This is a blocker for us and would be happy to open a PR, if pointed in the right direction. Any advice?
Hi @ameyabhope, I encourage you to open a similar issue as the one you linked but specific to Postgres. This issue will be a better place to have a detailed implementation discussion with our connector team. The related code is probably in this python file:
Been trying to make the change myself, had quick question -
I set everything up locally and I think I need to run these steps to test locally (according to the developing locally doc)-
VERSION=dev docker-compose up
This doesn’t seem to generate the
build dir under
airbyte-integrations/bases/base-normalization/. Any suggestions?
@ameyabhope did you follow this doc to work locally on the normalization?
Hi @alafanechere, I did.
I made a small change to one of the print commands to start off.
Ran the unitTests which were successful. Followed it by the
:build step and eventually ran
docker-compose up in dev mode. Still no luck getting my changes on the UI.
I guess the question is that would I need to change something other than print command to see if local development env is working?
After a bit more investigation I think that you need to:
- Change the version of normalization version to
dev in this file
- Rebuild the platform images with:
SUB_BUILD=PLATFORM ./gradlew build
If you want to disable the test run on platform build you can use
SUB_BUILD=PLATFORM ./gradlew build -x test which will make the build run faster.
Thanks @alafanechere, I finally got my local dev setup working.
I tried a bunch of things without much luck. Could you tag @edgao here (I can’t seem to do it), so that we can colab and work on this?
Please sync with @edgao directly on the GitHub issue
This was fixed please updated to Airbyte 0.39.20+ version.
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.