Deploy Kubernetes: JOB_KUBE_TOLERATIONS with no effect

Platform, Deploy & Infra Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Linux
  • Memory / Disk: -
  • Deployment: Kubernetes
  • Airbyte Version: What version are you using now? 0.39.32-alpha
  • Source name/version: Postgres 0.4.30
  • Destination name/version: GCS 0.2.9
  • Step: Sync
  • Description:

I want to assign a toleration to the airbyte worker jobs with this value

        - key: k8s.company.com/node-pool
          value: data
          operator: Equal

However, if I don’t specify an effect this toleration is not added to the job.
like JOB_KUBE_TOLERATIONS: key=k8s.bodyguard.ai/node-pool,operator=Equal,value=data

And if I add an effect like
JOB_KUBE_TOLERATIONS: key=k8s.bodyguard.ai/node-pool,operator=Equal,effect=NoExecute,value=data

I got this error

1 node(s) had taint {k8s.bodyguard.ai/node-pool: data}, that the pod didn't tolerate

Is it possible to make this effect property optional ?

Thanks

2

Hey, you need to assign first the tolerations to the node so that while we are creating the pods it selects based on that. You can find it more here Taints and Tolerations | Kubernetes

It will also be helpful if you can help us with the usecase on how you are using TOLERATIONS.

3

Ok will have a look and will describe the use case if it’s successful but in a few word my airbyte pods are set to a particular node pool and I want that the sync worker job to be executed in this node pool and not in the default one

4

@harshith it works well with the good taint on the node, and the variables JOB_KUBE_NODE_SELECTORS and JOB_KUBE_TOLERATIONS properly set !

5

@lucienfregosi Hello! Are you using the helm chart or the Kustomize deployment ? Im currently trying to use this configs for a similar usecase but im having some trouble. Can you show me how you configured this part of your chart as an example ? :slight_smile: