Adjusting CPU resources for specific Airbyte components in Kubernetes deployment

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is seeking help to adjust CPU resources for specific Airbyte components in a Kubernetes deployment to avoid unnecessary scaling. They have tried setting CPU requests and limits without success.


Question

Hey everyone,

I’m trying to deploy Airbyte on Kubernetes and everything seems fine, apart from allocated resources to the following components:
• call-heartbeat-server
• remote-stdin
• relay-stdout
• relay-stderr
I’ve seen this issue: https://github.com/airbytehq/airbyte/issues/35749, but setting JOB_MAIN_CONTAINER_CPU_REQUEST or JOB_MAIN_CONTAINER_CPU_LIMIT doesn’t do much. The problem that I have is that once a connection testing starts, it initializes the pods specified above and each of them consumes 2 CPUs each, in total being consuming over 6 CPUs, which seems quite a lot and could cause unnecessary scaling. Is there any way to overwrite the requests and limits CPU resources for those pods? Thanks in advance!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["kubernetes", "cpu-resources", "airbyte-components", "scaling", "pod-resources"]
2

<@U035912NS77> Hey, maybe you know something about relay-stderr, which is created with every source / destination check? It’s a sidecar container as far as I understand, and I managed to drop down resources for call-heartbeat-server and relay-stdout by providing

"SIDECAR_KUBE_CPU_LIMIT": "100m",
"SOCAT_KUBE_CPU_REQUEST": "50m",
 "SOCAT_KUBE_CPU_LIMIT": "100m",```
inside globals.env_vars - do you think it's possible to reach the `relay-stderr` this way too?
3

I was under the impression that it used the same resource limits as relay-stdout—possible there’s a bug related to that. I’d probably check through the GitHub issues first and see if anyone else is seeing the same. You could also check your Airbyte version and try upgrading to latest to see if the behavior is the same (which is often helpful if you think something might be a bug)

4

Yes, I am deploying via Helm and I already set these values in the global.jobs.resources.requests/limits path, like

    resources:

      requests:
        cpu: 100m
        memory: 256Mi

      limits:
        cpu: 100m
        memory: 1Gi```
It might be not enough, but the jobs themselves are not run yet, since I'm just creating connections and I'm seeing huge CPU spike during `Test the source` stage (during the Check stage?). I was thinking maybe this can be configured through `worker.kube-job-config.check.cpu-limit`  but I guess this isn't the stage or it's not picked up.
5

Here’s an example of how relay-stderr looks like: