Summary
When creating a new source in Airbyte on a private Kubernetes cluster, there is an issue with pulling images from Docker Hub. The user is seeking guidance on how to configure Airbyte to use a different image registry for connectors, specifically modifying the registry URL. They are also looking for best practices and example values.yaml files for this configuration.
Question
I have recently installed Airbyte in our private Kubernetes cluster and have been conducting a proof of concept. During this process, I encountered an issue when trying to create a new source.
By default, the creation of a new source triggers a Kubernetes Job pod, which attempts to pull the required image from Docker Hub. However, pulling images directly from Docker Hub is not permitted in our private cluster.
I am looking for guidance on how to configure Airbyte to use a different image registry for these connectors. We have a proxy cache feature that can pull images from Docker Hub, but I need to modify the registry URL accordingly.
Any best practices, or example values.yaml files for configuring Airbyte in a private Kubernetes environment, would be greatly appreciated.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["airbyte", "private-kubernetes-cluster", "image-registry", "docker-hub", "values.yaml", "connector"]
For Airbyte components (server, webapp, worker and so on), you need to update values for image: in values.yaml .
If you have container orchestrator enabled you might need to update extraEnv section for worker
- name: CONTAINER_ORCHESTRATOR_IMAGE
value: "private-registry/container-orchestrator:..."```
For Airbyte source & destination connectors, situation is more tricky. You need to connect to Airbyte's PostgreSQL database first. Then in `actor_definition_version` table you need to update `docker_repository` for every connector and its version that you want to use.
Keep in mind, that advice above related to connectors touches internals and it’s not guaranteed to work in the future.
You can also add source/destination connectors via user interface with Add a new Docker connector and provide Docker repository name , Docker image tag values that point to Docker image in your cache.
For (server, webapp, worker and so on), I already got it running by updating the values.yaml. For sources and destinations registry, I will try your solution.
Can you also let me know if there is any custom solution to integrate with SSO, any example is helpful
You can check these:
https://www.keycloak.org/
https://github.com/oauth2-proxy/oauth2-proxy
However I won’t provide any help in configuring any of them.
I noticed that keycloak appears in values.yaml in Airbyte’s helm charts, but I haven’t tried to enable and use it.
Few years ago I used oauth2-proxy in Kubernetes and it was relatively easy to configure.
<@U05JENRCF7C>, Thank you very much for your suggestions. I was able to successfully modify the database, and the pods are now pulling images from the correct registries.
However, I have encountered a new issue while configuring the connector. Specifically, the pod that is spun up does not seem to have the capability to configure the “security context” within the Helm chart. Have you come across this issue before? If so, do you know of any potential workarounds?
Additionally, when I check the Airbyte server logs during connector configuration, I see an error indicating a failure to connect to “http://connectors.airbyte.com|connectors.airbyte.com.” Is an active connection to this endpoint mandatory? In our infrastructure, internet connectivity is blocked.
at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]```
You can check my answer here: https://airbytehq.slack.com/archives/C021JANJ6TY/p1723624623025359?thread_ts=1723621083.040129&cid=C021JANJ6TY
Spawning jobs is done dynamically, so it might be necessary to build custom version of Airbyte until “security context” for jobs is implemented
I can see containerSecurityContext or podSecurityContext for more “static” Airbyte core components in values.yaml in Helm charts but not in jobs section.
I’d recommend reporting new issue about that here: https://github.com/airbytehq/airbyte/issues/new/choose
As I checked https://connectors.airbyte.com/, it is “connectors registry” and it stores Airbyte CI versions as well
https://github.com/search?q=org%3Aairbytehq+connectors.airbyte.com&type=code
I think it’s not mandatory, but you might see some errors in UI as in logs.
You could mock <http://connectors.airbyte.com|connectors.airbyte.com> or configure proxy to see less noise in logs.