Summary
When using Airbyte OSS with large data sources like MySQL and BigQuery, how to skip historical data during initial loading without creating views in the production database.
Question
Hello Folks,
I am using Airbyte OSS
, which I deployed on GCP Compute Engine. Then I created a pipeline with MySQL
as the data source and BigQuery
as the destination.
I have one table with a size of 2TB
, which is too large to load all the data at once. How can I skip historical data during the initial loading?
I have already asked Ask-AI, and they suggested creating a view, but we are not allowed to create views in the production database.
Has anyone experienced a similar case to mine? And how did you resolve it?
thank you
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.