MongoDB to BigQuery Incremental Job Issue

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User reports that incremental jobs for MongoDB to BigQuery are executing in full refresh mode despite having a saved cursor. The issue persists after following troubleshooting recommendations, and relevant log entries indicate problems with op log stats and invalid resume tokens.


Question

Hi all!

I’m trying to setup a MongoDB connection with BigQuery but incremental jobs aren’t working as expected. All jobs are running in full refresh mode, even if there’s a cursor saved on the connection state. I already did all the recommendations listed on MongoDB’s <MongoDb Connector | Airbyte Documentation section> page regarding oplog and sync frequency but the problem persists.

I’m using the latest versions of MongoDB and BigQuery connectors to this date (1.2.10 and 2.4.10 respectively).

These are some of the entries in the last log:


2024-02-20 13:29:45 source &gt; 2024-02-20 13:29:45 INFO i.a.i.s.m.c.MongoDbDebeziumStateUtil(isValidResumeToken):122 - Invalid resume token '8265D4A833000000012B022C0100296E5A1004FF7D341058A2499EAFD5936EEF9BD08F46645F6964006465D4A8338943AB8A3178AEB90004' present, corresponding to timestamp (seconds after epoch) : 1708435507.  Initial snapshot will be performed for all streams.

2024-02-20 13:29:45 source &gt; {"type":"TRACE","trace":{"type":"ANALYTICS","emitted_at":1.708435785668E12,"analytics":{"type":"db-sources-cdc-cursor-invalid","value":"1"}}}

2024-02-20 13:29:45 source &gt; 2024-02-20 13:29:45 INFO i.a.i.s.m.c.MongoDbCdcInitializer(createCdcIterators):119 - Saved offset is not valid. Airbyte will trigger a full refresh.```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1708437836032839) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['mongodb', 'bigquery', 'incremental-jobs', 'full-refresh', 'oplog', 'resume-token', 'airbyte-logs']
</sub>