Source Postgres - Keywords table from Postgres not performing incremental load

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: MacOS
  • Deployment: Hosted on google cloud
  • Airbyte Version: 0.40.9
  • Source name/version: Postgres 1.0.14
  • Destination name/version: BigQuery (denormalized typed struct) 0.1.11
  • Step: The issue is happening during sync
  • Description:

Hey Team,

I’m connecting our PostgresSQL to BigQuery denormalized using the Postgres source. I’m running into an issue with Keywords table , where during the initial run using the incremental load with id as the cursor, it’s establishing the cursor fields during the first run. But when I run it again instead of appending new values it is again establishing the cursors, and performing full-refresh append
There’s about 700K records, with around 600k log lines.

First run:

e30640bb_ad76_4a17_adb0_e97a492316d9_logs_44035_txt.txt (305.1 KB)

Second run:

e30640bb_ad76_4a17_adb0_e97a492316d9_logs_44039_txt.txt (1.4 MB)

Hey @Yash,

Thanks for your post, I’ll look into this but just to summarize: on your first sync between Postgres and Bigquery, the cursor is established with the id field as the cursor field but on the second sync, instead of using the previously established cursor, it nulls them and performs a full refresh sync. If I got anything wrong, feel free to correct me.

I’ll try to see if I can reproduce it or see if other users reported similar issues.

Hey @sajarin,
you are correct!

Hey @Yash,

I tried to see if I could reproduce the issue and I could not. I know it’s been a while since your heard from us, have you gotten this sync to work or is it still failing to perform the incremental load?