- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu
- Memory / Disk: 4Gb / 1 Tb
- Deployment: Kubernetes
- Airbyte Version: 0.44.3
- Source name/version: Salesforce / 2.0.12
- Destination name/version: BigQuery / 1.2.13
- Step: during sync
- Description:
I am currently working with a Salesforce Contact table.
The synchronization method is Incremental Sync - Deduped History
The Contact table is heavily updated and pulls in tens of thousands of records daily.
So I found that some records are missing.
-
Synchronization started on 2023-05-28 21:00:00 (State is {SystemModstamp": "2023-05-27T21:15:08.000+0000} at this time)
-
after synchronization, State is updated to {“SystemModstamp”: “2023-05-28T21:10:08.000+0000”}.
-
next synchronization, it was found that a few records were missing, records for which SystemModstamp had the following values: {“SystemModstamp”: “2023-05-28T21:10:08.000+0000”}.(I checked the raw table)
- 2023-05-28 21:15:07 UTC
- 2023-05-28 21:15:06 UTC
I can see why it is missing because the value of SystemModstamp, which is indeed a curl field, is the value after that at step2.
How can this be resolved?