MongoDB to BigQuery Transfer Timeout Issue

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

User is experiencing a timeout error when transferring a large collection (45 million rows) from MongoDB to BigQuery using Airbyte, while a smaller collection (14 million rows) transferred successfully.


Question

Hello team. I’m using open source version of AirByte hosted on AWS. I want to transfer data from MongoDB to BigQuery. I successfully transferred a collection having 14 million rows but when I tried with a collection having 45 million rows, the source connector is timing out again and again with this error

com.mongodb.MongoExecutionTimeoutException: PlanExecutor error during aggregation :: caused by :: operation exceeded time limit

Please help me is resolving this issue as soon as possible as it’s urgent for our team.
Thanks


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['mongodb', 'bigquery', 'timeout', 'airbyte', 'connector']
2

<@U071HD82KAB> heads up :wink: we moving Mongo to the new CDK sometime soon, right?

3

<@U069EMNRPA4> that may happen in the a quarter or two.

Please file a ticket, and we will get to this

4

and please provide the full log

5

<@U071HD82KAB> heads up :wink: we moving Mongo to the new CDK sometime soon, right?

6

<@U069EMNRPA4> that may happen in the a quarter or two.

Please file a ticket, and we will get to this

7

and please provide the full log