High CPU Usage by airbyte-worker Container after Update to v0.50.51

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The airbyte-worker container is consuming high CPU resources after updating to v0.50.51, with usage scaling with the number of syncs running. Setting JOB_MAIN_CONTAINER_CPU_LIMIT variable does not lower the CPU usage.


Question

Hello everyone, was wondering if anyone else faced this issue.
We are running OSS version on e2-highmem-8 machine in GCP.
We are facing problems since we have updated to v0.50.51, namely the airbyte-worker container is hogging CPU resources and it’s scaling with the number of syncs running.
If I run two syncs at the same time the usage goes up to 400%, we also see our CPU locked at 100% usage in the morning where we ran most of our syncs, while most syncs look to be in pending state.

This seems like a bug, I have tried setting JOB_MAIN_CONTAINER_CPU_LIMIT variable but that does not seem to do anything to lower the CPU usage of the airbyte-worker

I have seen this thread already: https://airbytehq.slack.com/archives/C021JANJ6TY/p1706901124594629, but the response is not very helpful as this looks like a bug.
I was also able to replicate this on my local machine (ryzen 7 5800x), with a fresh install running just 3 connections CPU usage is 600%.

Any advice on sorting this would be highly appreciated.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["high-cpu-usage", "airbyte-worker", "v0.50.51", "syncs", "pending-state", "CPU-limit"]
2

Have you tried to configure more workers in separate containers?

In my project with colleague we noticed similar behaviour on Kubernetes cluster. We’ve just increased number of airbyte-worker pods and it solved the problem.

The other idea to spread syncs during the day with no overlap or minimal one.

3

I don’t think that’s the solution I am looking for. The syncs we are running are pulling couple thousand rows each, it should not be using that much CPU.
We did not see this issue on previous versions, we updated recently and noticed that our CPU is getting hammered.
We never reached 100% CPU usage before the update.

Anyway I thank you for the suggestion.