Automating Airbyte abctl Deployment with Proxy Service

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is seeking a workaround for deploying a proxy service required for Airbyte abctl Kubernetes deployment, facing a dependency issue with the abctl.kubeconfig file needed for installation.


Question

Hello,
I have asked this https://airbytehq.slack.com/archives/C01AHCD885S/p1732033031914199|question in your AI channel and the answer has lead me to reach out to you here.
For a bit more context on the above question, and related to our resolved issue https://airbytehq.slack.com/archives/C021JANJ6TY/p1731599582195379|here, we are migrating to airbyte abctl from docker compose. We have successfully managed to do this manually which is great! However, now we want to automate the infrastructure deployment and we have a bit of a pickle:
• Our abctl k8s deployment is dependent on a pod service pair that contains a proxy that allows airbyte to connect to our external database (Cloud SQL proxy btw).
• We want to deploy this proxy service pair on the same namespace and with the same default configuraiton as the other airbyte pods. For this, we need the abctl.kubeconfig file in order to deploy the proxy like so: kubectl apply -f cloud-sql-proxy.yaml -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig
• To have the abctl.kubeconfig file, we need to launch the abctl local install command. But for this installation to work we need the proxy, so you can see the chicken and the egg problem.
Do you see a quick workaround? I was thinking of downloading the abctl.kubeconfig file beforehand but 1. Your AI told me not to :smile: 2. I dont know where I can access it.
Thanks for you help.
FYI - <@U05FB419LMR>


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['abctl', 'kubernetes', 'cloud-sql-proxy', 'deployment', 'infrastructure']
2

An idea, abctl local install has option --chart=STRING where you can pass path to a chart. What you can do is to create helm chart that has Airbyte helm chart as a dependency and the rest of the code defines your proxy service
https://helm.sh/docs/chart_template_guide/subcharts_and_globals/

For production deployment I would recommend regular Kubernetes instead of abctl

3

An idea, abctl local install has option --chart=STRING where you can pass path to a chart. What you can do is to create helm chart that has Airbyte helm chart as a dependency and the rest of the code defines your proxy service
https://helm.sh/docs/chart_template_guide/subcharts_and_globals/

For production deployment I would recommend regular Kubernetes instead of abctl

4

<@U05JENRCF7C>, thank you for the tip. In order to pass the Airbyte helm chart as a dependency, I need to have it somewhere in the local environment. How can I download it? Is it available in a git repository based on the version of airbyte we are deploying? Thanks

5

https://github.com/airbytehq/airbyte-platform/blob/0be62b72122162cc24075d2894f42c7926cadb5b/charts/airbyte/Chart.yaml#L52-L55|Chart.yaml#L52-L55
You should be able to use a remote reference like this:

  - name: airbyte
    repository: <https://airbytehq.github.io/helm-charts/>
    version: 1.2.0```
or download full chart from <https://github.com/airbytehq/helm-charts>
6

The rest of an implementation is yours. Good luck :wink: