Optimizing data transfer from MongoDB to BigQuery using Airbyte

Summary

The user is experiencing slow data transfer from MongoDB to BigQuery using Airbyte and is seeking advice on optimization or alternative approaches.


Question

hey folks, I would like to ask you for best practice here, I need to move on a daily basis around 93M rows from Mongodb to BgiQuery, I tried with Airbyte… but I’m observing it’s slowing for those operations, is there any tweak/hack or I should think in using another approach? Thanks in advance!!



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["mongodb", "bigquery", "airbyte", "data-transfer", "optimization"]

From our experience with a lot of data, most of the time is in the deduplication process. If you can do deduplication separately the sync will be much faster

We move 20M rows from Postgres to Snowflake daily and most of the time is spent deduplicating