BigQuery Destination using incremental append

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: you can use something like 32Gb / 30 Gb
  • Deployment: Docker
  • Airbyte Version: 0.39.41-alpha
  • Source name/version: MySQL (0.6.1)
  • Destination name/version: BigQuery (1.1.14)
  • Step: The issue is happening after initial sync, during the subsequent runs
  • Description: I currently setup the replication to be incremental|append and when I run the sync almost immediately it sync the whole data as if it was a full refresh replication. For example on the first successful sync I get this in the summary

526.61 MB | 677,667 emitted records | 677,667 committed records | 19m 56s | Sync
In the next summary I get the same summary and it takes a similar amount of time.

I have set the binlog of the mysql to row and im using GCS for the bigquery destination.

Not sure if there’s anything I might be missing

Are you using MySQL CDC? did you chose the cursor for the table?

Yes and I couldn’t set a cursor because it says source defined

If you’re using CDC the cursor is chose automatically using the CDC system. Looks for your connection Airbyte is not able to select it properly and continue to run from the beginning.

So what do I do to solve this?

If you disable CDC (just for testing) the sync works?

@ajatau every sync is getting full data or only the first one ?