Improving historical sync performance for Klaviyo events to BigQuery in Airbyte

Summary

The user is experiencing slow performance when syncing Klaviyo events to BigQuery using Airbyte. They suspect that the sequential fetching from Klaviyo is causing the slowness and are looking for a way to chunk the historical time range and run the sync in parallel for multiple chunks.


Question

Hello experts :wave:
I’m using (self hosted) Airbyte to sync Klaviyo events to BigQuery
I’m running a historical sync (a few months, each day is around 1-4 GB) and it takes forever :disappointed:
I’m on GCP with a n1-standard-8 which should be more than enough.
We do not hit any Klaviyo rate limit, currently it fetches around 5000 records (25 requests) every 10-15 seconds, which give the assumption that there’s no parallel fetching from Klaviyo, it is completely sequential, hence the slowness.

Is this true? If yes, is there a way to chunk the historical time range and run sync in parallel for a few chunks?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["klaviyo-events", "bigquery", "historical-sync", "sequential-fetching", "parallel-sync", "performance"]