BigQuery Destination using incremental append

Connector Questions & Issues
, , , ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: you can use something like 32Gb / 30 Gb
  • Deployment: Docker
  • Airbyte Version: 0.39.41-alpha
  • Source name/version: MySQL (0.6.1)
  • Destination name/version: BigQuery (1.1.14)
  • Step: The issue is happening after initial sync, during the subsequent runs
  • Description: I currently setup the replication to be incremental|append and when I run the sync almost immediately it sync the whole data as if it was a full refresh replication. For example on the first successful sync I get this in the summary

526.61 MB | 677,667 emitted records | 677,667 committed records | 19m 56s | Sync
In the next summary I get the same summary and it takes a similar amount of time.

I have set the binlog of the mysql to row and im using GCS for the bigquery destination.

Not sure if there’s anything I might be missing

2

Are you using MySQL CDC? did you chose the cursor for the table?

3

Yes and I couldn’t set a cursor because it says source defined

4

If you’re using CDC the cursor is chose automatically using the CDC system. Looks for your connection Airbyte is not able to select it properly and continue to run from the beginning.

5

So what do I do to solve this?

6

If you disable CDC (just for testing) the sync works?

7

@ajatau every sync is getting full data or only the first one ?