Summary
User is experiencing slow historical sync from Klaviyo to BigQuery using Airbyte, suspecting sequential fetching of records. Inquires about the possibility of chunking the historical data for parallel processing.
Question
Hello experts
I’m using (self hosted) Airbyte to sync Klaviyo events to BigQuery
I’m running a historical sync (a few months, each day is around 1-4 GB) and it takes forever
I’m on GCP with a n1-standard-8
which should be more than enough.
We do not hit any Klaviyo rate limit, currently it fetches around 5000 records (25 requests) every 10-15 seconds, which give the assumption that there’s no parallel fetching from Klaviyo, it is completely sequential, hence the slowness.
Is this true? If yes, is there a way to chunk the historical time range and run sync in parallel for a few chunks?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.