- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Docker
- Memory / Disk: 4GB & 1 CPU per pod
- Deployment: Kubernetes
- Airbyte Version: What version are you using now? 0.39.32-alpha
- Source name/version: Postgres 0.4.31
- Destination name/version: GCS 0.2.29
- Step: Sync
Following this thread Slack
I’m wondering how to speed up my airbyte jobs.
Here is my current performance (for my dev environment)
1.89 GB | 2,170,569 emitted records | 2,170,569 committed records | 11m 27s | Sync
Do you have best practices for increasing speed ? Is it even possible ?
I tried to set all the
JOB_MAIN_CONTAINER_CPU_REQUEST variables but with no effect.
Thanks for your help
Hey, you could check first the usage of the resources for the sync pod. If you find the usage is high and over the limit you could then increase that limit
My reading pod is using around 162MB and
0.0000533 of CPU.
My writing pod is using around 163Mb and
0.00000717 of CPU
This is way below the values set with
Got it. I meant if the problem is with the pulling/sending data and not with memory changing that variable won’t make much sense because you would already be having enough resources.
Thus it would be great if we can point to some data which says the slow Is because of lack of resources. Is it possible to check this?
Not sure to understand. I am really convinced that is not an issue related to lack of resources (at least for CPU/RAM)
What is the expected performance for the postgres connector ?
How can I identify the performance bottleneck of my job ?