Summary
The user is facing issues with MongoDB to BigQuery incremental jobs not working as expected, even after following recommendations. The logs show errors related to invalid resume tokens and saved offsets.
Question
Hi all!
I’m trying to setup a MongoDB connection with BigQuery but incremental jobs aren’t working as expected. All jobs are running in full refresh mode, even if there’s a cursor saved on the connection state. I already did all the recommendations listed on MongoDB’s <Mongo DB | Airbyte Documentation section> page regarding oplog and sync frequency but the problem persists.
I’m using the latest versions of MongoDB and BigQuery connectors to this date (1.2.10 and 2.4.10 respectively).
These are some of the entries in the last log:
2024-02-20 13:29:45 source > 2024-02-20 13:29:45 INFO i.a.i.s.m.c.MongoDbDebeziumStateUtil(isValidResumeToken):122 - Invalid resume token '8265D4A833000000012B022C0100296E5A1004FF7D341058A2499EAFD5936EEF9BD08F46645F6964006465D4A8338943AB8A3178AEB90004' present, corresponding to timestamp (seconds after epoch) : 1708435507. Initial snapshot will be performed for all streams.
2024-02-20 13:29:45 source > {"type":"TRACE","trace":{"type":"ANALYTICS","emitted_at":1.708435785668E12,"analytics":{"type":"db-sources-cdc-cursor-invalid","value":"1"}}}
2024-02-20 13:29:45 source > 2024-02-20 13:29:45 INFO i.a.i.s.m.c.MongoDbCdcInitializer(createCdcIterators):119 - Saved offset is not valid. Airbyte will trigger a full refresh.```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1708437836032839) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["mongodb", "bigquery", "incremental-jobs", "connector", "troubleshooting", "resume-token", "saved-offset"]
</sub>