Summary
Airbyte OSS running on Kubernetes via Helm Chart is pulling data in small batches, leading to prolonged runtimes when transferring large datasets. Increasing CPU and memory did not resolve the issue.
Question
Hi, we’re using Airbyte OSS, running it on Kubernetes via a Helm Chart, and we’re encountering issues with long runtimes.
The logs indicate that Airbyte is pulling the data in small batches, which leads to prolonged runtimes.
For example, transferring 5 million records from MySQL to Redshift takes about 30 minutes because the data is pulled in batches of 5000 records at a time. We’ve experienced a similar issue when trying to pull data from Facebook.
We’ve tried increasing the CPU and memory, but that didn’t improve the situation.
Has anyone encountered this issue and can help?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.