Optimizing data transfer from MongoDB to BigQuery using Airbyte

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is experiencing slow data transfer from MongoDB to BigQuery using Airbyte and is seeking advice on optimization or alternative approaches.


Question

hey folks, I would like to ask you for best practice here, I need to move on a daily basis around 93M rows from Mongodb to BgiQuery, I tried with Airbyte… but I’m observing it’s slowing for those operations, is there any tweak/hack or I should think in using another approach? Thanks in advance!!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["mongodb", "bigquery", "airbyte", "data-transfer", "optimization"]
2

From our experience with a lot of data, most of the time is in the deduplication process. If you can do deduplication separately the sync will be much faster

3

We move 20M rows from Postgres to Snowflake daily and most of the time is spent deduplicating