How to speed up airbyte Jobs

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Docker
  • Memory / Disk: 4GB & 1 CPU per pod
  • Deployment: Kubernetes
  • Airbyte Version: What version are you using now? 0.39.32-alpha
  • Source name/version: Postgres 0.4.31
  • Destination name/version: GCS 0.2.29
  • Step: Sync
  • Description:

Following this thread Slack

I’m wondering how to speed up my airbyte jobs.
Here is my current performance (for my dev environment)

1.89 GB | 2,170,569 emitted records | 2,170,569 committed records | 11m 27s | Sync

Do you have best practices for increasing speed ? Is it even possible ?
I tried to set all the JOB_MAIN_CONTAINER_CPU_REQUEST variables but with no effect.

Thanks for your help

2

Hey, you could check first the usage of the resources for the sync pod. If you find the usage is high and over the limit you could then increase that limit

3

Hi @harshith

My reading pod is using around 162MB and 0.0000533 of CPU.
My writing pod is using around 163Mb and 0.00000717 of CPU
This is way below the values set with JOB_MAIN_CONTAINER_* values

4

Got it. I meant if the problem is with the pulling/sending data and not with memory changing that variable won’t make much sense because you would already be having enough resources.

Thus it would be great if we can point to some data which says the slow Is because of lack of resources. Is it possible to check this?

5

Hi @harshith

Not sure to understand. I am really convinced that is not an issue related to lack of resources (at least for CPU/RAM)

What is the expected performance for the postgres connector ?
How can I identify the performance bottleneck of my job ?

6

Hey you can look at this thread https://discuss.airbyte.io/t/mysql-source-connector-performance/1092

7

Hmm interesting

Thanks for the inputs.