Summary
User is experiencing failures in
workload-api-serverandworkload-launchercomponents after upgrading Airbyte on Kubernetes, related to Vault access issues. The error indicates a certificate problem when attempting to request a new Vault token. User seeks assistance in configuring Vault for environment variable injection without using it for secret management.
Question
Hello team 
, we’re using Airbyte deployed on K8s. This week, during an attempt to upgrade from version 0.59.1 to 1.2.0, the workload-api-server and workload-launcher components started failing. The issue seems to be related to Vault access, with the following error:
time="2024-11-28T14:56:09Z" level=error msg="failed to request new Vault token" app=vault-env err="Put \"<https://vault:8200/v1/auth/kubernetes/login>\": x509: certificate signed by unknown authority"
For more context, we’re injecting Vault into our namespace because it stores secrets, such as database credentials. However, we are not using Vault as a secret manager for Airbyte. Despite this, the pods under the two replicasets (workload-api-server and workload-launcher) seem to include references to Vault in their configurations:
- name: VAULT_ADDR
value: <https://vault:8200>
- name: VAULT_SKIP_VERIFY
value: 'false'
- name: VAULT_AUTH_METHOD
value: jwt
- name: VAULT_PATH
value: kubernetes
- name: VAULT_ROLE
value: airbyte
- name: VAULT_IGNORE_MISSING_SECRETS
value: 'false'
- name: VAULT_ENV_PASSTHROUGH
- name: VAULT_JSON_LOG
value: 'false'
- name: VAULT_CLIENT_TIMEOUT
value: 10s
- name: VAULT_LOG_LEVEL
value: info
...```
From the documentation, `SECRET_PERSISTENCE` is set to `None` by default. We attempted to override this in the `values.yaml` file under `global.env_vars.SECRET_PERSISTENCE`, and the change did take effect but changed nothing. Additionally, we tried overriding the Vault-related variables using the `extraEnv` section in the same `values.yaml` file. While these variables were included in the pod configurations, they were overridden by the default values (mentioned before the default values).
Currently, our goal is to use Vault only for environment variable injection, not for secret management, while we assess the best approach moving forward. Can someone assist us in correctly configuring this setup? Thank you
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1732806524251999) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
['airbyte', 'kubernetes', 'vault', 'upgrade', 'workload-api-server', 'workload-launcher', 'certificate-error']
</sub>