MongoDB to BigQuery Transfer Timeout Issues

Summary

User is experiencing timeout errors during data transfers from MongoDB to BigQuery after migrating from Airbyte Cloud to the open-source version on AWS. The issue occurs with large document streams, leading to transfer failures.


Question

Hi team,
I’m experiencing issues with my MongoDB to BigQuery data transfers. Previously, I was using Airbyte Cloud for this transfer, and it worked seamlessly. However, after hosting the Airbyte open-source version on AWS, I’m facing repeated MongoDB timeout errors.
Here are the details:
Source: MongoDB, with a stream of ~1 million documents.
Error: Transfer fails each time with a timeout error.
Other attempts: I tried with a different collection of ~1.3 million documents. In this case, it gets stuck consistently at 1.2 million documents before timing out.
I’ve attached a screenshot for reference. Could you help me resolve this?
Thanks!



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['mongodb', 'bigquery', 'timeout-error', 'data-transfer', 'airbyte-open-source', 'aws']