Help with upgrading Airbyte in Kubernetes

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is seeking assistance with upgrading an old Airbyte version running in Docker to support a new proof of concept involving terraform, dbt, and airflow. The installation was done using abctl in a Kubernetes cluster with an external database.


Question

Hi <@U01MMSDJGC9>,
I really need help to find why and how to resolve.

Airbyte is part of my project this year.
A lot of functionalities make sense in our ecosystem.

I have installed via abctl AND in a kubernetes cluster with external database.

And still stuck to this step…

(An old air byte version0.57 is runing here in docker. But I need to upgrade to push a POC, terraform dbt airflow)

https://airbytehq.slack.com/archives/C01AHCD885S/p1733484641047759


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['airbyte', 'upgrade', 'kubernetes', 'docker', 'abctl', 'terraform', 'dbt', 'airflow']
2

here is my simple abctl deployment

I’m trying to install Airbyte 1.2.0 on a Linux VM using abctl.

The installation appears successful, and Airbyte is running without any error logs from the pods. The PostgreSQL database seems properly set up, with all the expected schemas and tables created. The only record I’ve seen added so far is the one for the workspace.

global:
  env_vars:
    HTTP_PROXY: "<http://10.0.0.0:10>"
    HTTPS_PROXY: "<http://10.0.0.0:10>"
    NO_PROXY: "10.0.0.0/8,localhost,127.0.0.1,.<http://domain.fr|domain.fr>,.cluster.local,.local,.svc,airbyte-*,airbyte-db-0,airbyte-minio-0,airbyte-abctl-airbyte-connector-builder-server-svc,airbyte-abctl-airbyte-server-svc,airbyte-abctl-airbyte-webapp-svc,airbyte-abctl-temporal,airbyte-abctl-workload-api-server-svc,airbyte-db-svc,airbyte-minio-svc,airbyte-abctl-worker-svc,airbyte-temporal,airbyte-workload-api-server-svc"
    http_proxy: "<http://10.0.0.0:10>"
    https_proxy: "<http://10.0.0.0:10>"
    no_proxy: "10.0.0.0/8,localhost,127.0.0.1,.<http://domain.fr|domain.fr>,.cluster.local,.local,.svc,airbyte-*,airbyte-db-0,airbyte-minio-0,airbyte-abctl-airbyte-connector-builder-server-svc,airbyte-abctl-airbyte-server-svc,airbyte-abctl-airbyte-webapp-svc,airbyte-abctl-temporal,airbyte-abctl-workload-api-server-svc,airbyte-db-svc,airbyte-minio-svc,airbyte-abctl-worker-svc,airbyte-temporal,airbyte-workload-api-server-svc"
    JAVA_TOOL_OPTIONS: "-Dhttp.proxyHost=10.0.0.0 -Dhttp.proxyPort=10 -Dhttps.proxyHost=10.0.0.0 -Dhttps.proxyPort=10 -Dhttp.nonProxyHosts=10.*|localhost|127.0.0.1|*.internal|*.cluster.local|*.local|*.svc|airbyte-*|airbyte-db-0|airbyte-minio-0|airbyte-abctl-airbyte-connector-builder-server-svc|airbyte-abctl-airbyte-server-svc|airbyte-abctl-airbyte-webapp-svc|airbyte-abctl-temporal|airbyte-abctl-workload-api-server-svc|airbyte-db-svc|airbyte-minio-svc|airbyte-worker-svc|airbyte-temporal|airbyte-workload-api-server-svc""
    
webapp:
  image:
    repository: registry/airbyte-webapp-no-ipv6
    tag: 1.2.0```
```abctl local install -v --values=./values.yaml --host=<http://domain.fr|domain.fr> --port=8080 --no-browser```
```FROM airbyte/webapp:1.2.0

USER root

# Ensure the airbyte user exists
RUN adduser -D -s /bin/bash airbyte

# Add the script to remove IPv6
COPY remove_ipv6.sh /tmp/remove_ipv6.sh
RUN chmod +x /tmp/remove_ipv6.sh &amp;&amp; \
    /tmp/remove_ipv6.sh &amp;&amp; \
    echo "Nginx configuration after IPv6 removal:" &amp;&amp; \
    cat /etc/nginx/templates/default.conf.template &amp;&amp; \
    rm /tmp/remove_ipv6.sh

USER airbyte```
3

<@U01MMSDJGC9>
does it make sense?

Raphael Meignant [6:01 PM]
<@U04SE163WC8> behind proxy abctl deployment, Should My controlplane container have the same noproxy variables as passed to pods environment variables?
If yes this means that I should pass env variables to the host sincecontrolplane container pick host env variables?

http://kapa.ai|kapa.ai [6:01 PM]
Let me check the knowledge base for you again. One moment! :books:

[6:02 PM]
Based on the knowledge sources provided, I don’t have specific information about the proxy configuration for abctl deployments or the control plane container’s environment variables. However, I can provide some general insights based on the available information about Airbyte’s deployment and proxy configurations.

When deploying Airbyte behind a proxy, it’s generally a good practice to ensure that all components, including the control plane container, have consistent proxy configurations. This typically includes setting the http_proxy, https_proxy, and no_proxy environment variables.

From the information available in the <https://github.com/airbytehq/airbyte/issues/16595|Github issue about proxy variables>, we can see that proxy-related environment variables are important for Airbyte’s functionality, especially when dealing with external connections.

Regarding your specific questions:

  1. Should the control plane container have the same no_proxy variables as passed to pods environment variables?
  • While there’s no explicit mention of the control plane container in the provided sources, it would be logical to maintain consistency across all components of the Airbyte deployment. So yes, it would be advisable to have the same no_proxy variables for the control plane container as for the other pods.
  1. If yes, does this mean that I should pass env variables to the host since the control plane container picks up host env variables?
  • The control plane container’s behavior in terms of inheriting environment variables from the host is not explicitly mentioned in the provided sources. However, if the control plane container does indeed inherit environment variables from the host, then yes, you would need to set these variables on the host system.

To ensure proper configuration, you might want to:

  1. Set the necessary proxy-related environment variables (http_proxy, https_proxy, no_proxy) on the host system.
  2. Verify that these variables are being correctly passed to or inherited by the control plane container.
  3. Ensure that the same variables are also set for other Airbyte pods as needed.

Given the limited specific information about abctl deployments and proxy configurations in the provided sources, I recommend consulting the official Airbyte documentation or reaching out to Airbyte support for more detailed guidance on configuring proxy settings for abctl deployments. They would be able to provide more accurate and up-to-date information tailored to your specific deployment scenario.

4

<@U01MMSDJGC9> are you able to provide exhaustive list of components and addresses to register as no-proxy please?

no_proxy=

AND

JAVA_TOOL_OPTION= -Dhttp.nonProxyHosts=