Slow data fetching from Klaviyo to BigQuery through Airbyte

[slack-user-airbyte]

Summary

Airbyte connection fetching Klaviyo data to BigQuery has been slow, fetching only 35M records in 12 days. User seeks help to understand reasons for slow retrieval, potential rate limiting issues on Klaviyo’s end, and optimizations to speed up the process.

Question

Hello <#C021JANJ6TY|>,
I’m currently fetching Klaviyo data through Airbyte and loading it into BigQuery. The Airbyte connection has been running for the past 12 days to pull data for a single stream, but it only fetched around 35M records so far.
Can anyone please help me understand:
• Why might Airbyte be taking so long to retrieve the data?
• Could it be a rate limiting issue on Klaviyo’s end?
• Are there any best practices or optimizations we can apply to speed up the data fetching process?
Thanks in advance!

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

_{["klaviyo", "bigquery", "airbyte-connector", "data-fetching", "rate-limiting", "optimizations"]}

[slack-user-airbyte]

https://developers.klaviyo.com/en/docs/rate_limits_and_error_handling#rate-limits

> Unless otherwise documented, all API endpoints use one of the following rate limits:
> • XS: 1/s burst; 15/m steady
> • S: 3/s burst; 60/m steady
> • M: 10/s burst; 150/m steady
> • L: 75/s burst; 700/m steady
> • XL: 350/s burst; 3500/m steady

[slack-user-airbyte]

https://docs.airbyte.com/integrations/sources/klaviyo#performance-considerations

[slack-user-airbyte]

> All new API endpoints are rate limited on a per-account basis, and used a fixed-window rate limiting algorithm with two distinct windows: burst (1-second window) and steady (1-minute window). All API traffic will be subject to these rate limits, and will receive HTTP 429 errors in the event either a burst or steady rate limit is reached.
https://developers.klaviyo.com/en/docs/rate_limits_and_error_handling#rate-limits