BigQuery Destination using incremental append

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: you can use something like 32Gb / 30 Gb
  • Deployment: Docker
  • Airbyte Version: 0.39.41-alpha
  • Source name/version: MySQL (0.6.1)
  • Destination name/version: BigQuery (1.1.14)
  • Step: The issue is happening after initial sync, during the subsequent runs
  • Description: I currently setup the replication to be incremental|append and when I run the sync almost immediately it sync the whole data as if it was a full refresh replication. For example on the first successful sync I get this in the summary

526.61 MB | 677,667 emitted records | 677,667 committed records | 19m 56s | Sync
In the next summary I get the same summary and it takes a similar amount of time.

I have set the binlog of the mysql to row and im using GCS for the bigquery destination.

Not sure if there’s anything I might be missing