Setting custom environment variables for Airbyte Jobs in Kubernetes environment

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

In a Kubernetes environment, setting custom environment variables for Airbyte Jobs can be achieved by adjusting the CONTAINER_ORCHESTRATOR_ENABLED variable. This change affects how Airbyte behaves in the Kubernetes environment.


Question

Hello everyone,
I am testing Airbyte 0.63.11 using Helm in a Kubernetes environment. In my environment where Airbyte is installed, it is necessary to go through an HTTP Proxy to communicate with the internet.
I successfully set the HTTP(S)_PROXY environment variables for the Airbyte worker, but I found that these environment variables are not inherited by the Pods executing the Jobs generated from the worker, causing communication timeouts.
Upon investigation, I found a post suggesting that setting “CONTAINER_ORCHESTRATOR_ENABLED” to false would allow environment variables set with JOB_DEFAULT_ENV_PREFIX to be inherited. I tried this, and it indeed worked as expected.
So, here are my questions:

  1. What is the CONTAINER ORCHESTRATOR?
  2. How does Airbyte change its behavior when the workers.env_vars.CONTAINER_ORCHESTRATOR_ENABLED environment variable is set to false using Helm? I’d like to understand the potential risks and operational overhead associated with this change.
  3. Are there any other methods to set custom environment variables for the Job execution Pods generated by the worker, besides setting “CONTAINER_ORCHESTRATOR_ENABLED” to false?

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["kubernetes-environment", "custom-environment-variables", "container-orchestrator", "airbyte-jobs", "helm"]