Kubernetes Deployment Error: Resource Quota Violation

slack-user-airbyte · December 9, 2024, 5:04pm

Summary

User is encountering a Kubernetes client exception related to resource quotas while deploying Airbyte on Kubernetes. The error indicates that CPU and memory limits/requests are not specified for the connector-sidecar, init, and main containers. The user is using Helm chart version 1.0.0 and has attempted to set environment variables for resource limits without success.

Question

hello community,
I am new to airbyte and trying to deploy airbyte community edition on kubernetes.
I am able to deploy and all pods are up, but when I am try to create connection i am getting below error in workload-launcher.
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: <https://10.100.0.1:443/api/v1/namespaces/data-engineering/pods/e-mongodb-v2-check-33a37202-430d-4270-b979-6c008749a86d-0-lnidc?fieldManager=fabric8>. Message: pods "e-mongodb-v2-check-33a37202-430d-4270-b979-6c008749a86d-0-lnidc" is forbidden: failed quota: data-engineering: must specify limits.cpu for: connector-sidecar,init,main; limits.memory for: connector-sidecar,init,main; requests.cpu for: connector-sidecar,init,main; requests.memory for: connector-sidecar,init,main. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=e-mongodb-v2-check-33a37202-430d-4270-b979-6c008749a86d-0-lnidc, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "e-mongodb-v2-check-33a37202-430d-4270-b979-6c008749a86d-0-lnidc" is forbidden: failed quota: data-engineering: must specify limits.cpu for: connector-sidecar,init,main; limits.memory for: connector-sidecar,init,main; requests.cpu for: connector-sidecar,init,main; requests.memory for: connector-sidecar,init,main, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660) ~[kubernetes-client-6.12.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:640) ~[kubernetes-client-6.12.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589) ~[kubernetes-client-6.12.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549) ~[kubernetes-client-6.12.1.jar:?]

i am using helm chart version 1.0.0

i can not find any properties which can specify limits for connector-sidecar. i tried suppling below extraEnv in workload-launcher but it didnt helped.

  JOB_MAIN_CONTAINER_CPU_LIMIT: '200m'
  JOB_MAIN_CONTAINER_CPU_REQUEST: '250m'
  JOB_MAIN_CONTAINER_MEMORY_LIMIT: '1Gi'
  JOB_MAIN_CONTAINER_MEMORY_REQUEST: '256Mi'```
does anybody have faced somthing like this?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1731324661785739) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['kubernetes', 'helm', 'resource-quota', 'airbyte', 'workload-launcher']
</sub>

slack-user-airbyte · December 9, 2024, 5:08pm

Hi Shubham Kanungo,
I also faced same issue
you will have to update the job section in values.yaml something like below

    resources:
      requests:
         memory: 500Mi
         cpu: 300m
    
      limits:
         cpu: 300m
         memory: 750Mi```
The above envs are present in arbyte-env configmap

you can check env-configmap.yaml  file present present inside templates repo of airbyte-platform to get morew info on how envs are mapped to values.yaml
Thank you hope this helps

slack-user-airbyte · December 9, 2024, 5:08pm

Thanks for the reply <@U07S3DM6CTY>
I have already added this properties in http://global.jobs|global.jobs . but the error remains same.

slack-user-airbyte · December 9, 2024, 5:09pm

Describe the replica pod which is created while sync happens
check that the envs which you updated is really updated in describe section
If env is updated then it might be resource issue so your pod is pending
if env is not updated recheck your values.yaml config

slack-user-airbyte · December 9, 2024, 5:09pm

env are updated Rajath.

slack-user-airbyte · December 9, 2024, 5:09pm

Then I am not sure
when I updated envs I was able to do it with helm installation in kubernates
I also tried abctl command to install
(here inside docker container cluster is running)
with abctl also I am getting same issue as yours
But helm installation I was able to do

slack-user-airbyte · December 9, 2024, 5:09pm

For helm installation I am using airbyte version 0.64.5
for abctl installation it is automatically taking latest version 1.1.0
Either there might be extra configuration in latest version or there is a bug in latest version
I was able to make it work with 0.64.5
So let me know if you find anything useful
Thank you

slack-user-airbyte · December 12, 2024, 3:21pm