MongoDB to BigQuery data transfer timeout issues after hosting Airbyte on AWS

Summary

Experiencing MongoDB to BigQuery data transfer timeout errors after switching from Airbyte Cloud to self-hosted Airbyte on AWS. Transfer fails consistently with ~1 million documents from MongoDB. Stuck at 1.2 million documents with a different collection of ~1.3 million documents.


Question

Hi team,
I’m experiencing issues with my MongoDB to BigQuery data transfers. Previously, I was using Airbyte Cloud for this transfer, and it worked seamlessly. However, after hosting the Airbyte open-source version on AWS, I’m facing repeated MongoDB timeout errors.
Here are the details:
Source: MongoDB, with a stream of ~1 million documents.
Error: Transfer fails each time with a timeout error.
Other attempts: I tried with a different collection of ~1.3 million documents. In this case, it gets stuck consistently at 1.2 million documents before timing out.
I’ve attached a screenshot for reference. Could you help me resolve this?
Thanks!



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["mongodb", "bigquery", "data-transfer", "timeout-error", "airbyte-cloud", "self-hosted", "aws"]