Trouble deploying Airbyte with external database using Helm

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

User is facing issues deploying Airbyte with an external database set using Helm. The generated manifests do not take the custom host, user, or database specified in the values.yaml file.


Question

Hello. Hopefully someone here can help with my issue.

I opened the following issue on airbyte’s GH https://github.com/airbytehq/airbyte/issues/36238

Long story short, I cannot deploy airbyte with an external database set successfully with helm.

Here is my values.yaml:

  database:
    secretName: "secrets-airbyte-airbyte-webapp-svc-prod"
    secretValue:  "airbyte-db-password"

postgresql:
  enabled: false

externalDatabase:
  host: "<http://my-custom-host.com|my-custom-host.com>"
  user: "postgres"
  database: "airbyte"
  password: ""
  existingSecret: "secrets-airbyte-airbyte-webapp-svc-prod"
  existingSecretPasswordKey: "airbyte-db-password"
  port: 5432
  jdbcUrl: ""```
No matter what I try I cannot get the generated manifests to take my host, user or database. Any advice?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1710786208807189) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["airbyte", "external-database", "helm", "deployment", "manifests"]
</sub>
2

Hi <@U06QR4RLN56>, sorry about the newbie question, but what command did you run to install airbyte?

I’ve copied your values and chart files, and updated them with my own details. When I run helm install airbyte airbyte/airbyte --values values.yaml --version 0.56.16, the installation was successful… but when I checked kubectl describe configmaps airbyte-airbyte-env | grep DATABASE none of the values had been overwritten:

DATABASE_PORT:
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:
DATABASE_URL:
DATABASE_DB:
RUN_DATABASE_MIGRATION_ON_STARTUP:
CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:
DATABASE_HOST:```
3

Probably they moved to a secret now

4

Thanks Marcos, and sorry for my newbie mistake :see_no_evil: for anyone who might have this problem in the future, in my case, the airbyte-db-0 pod gets recreated after upgrading the helm chart, but I could confirm the RDS db was being used in the monitoring tab of the AWS RDS Console.

One other problem I had was I could only find documentation stating the database user needs “<Configuring the Airbyte Database | Airbyte Documentation read and write permissions to the database>”. In our case at least, the airbyte-temporal failed with:

...
sql schema version compatibility check failed: pq: database "temporal" does not exist```
Once I removed the tables and types created by Airbyte and gave the additional permission with `alter role &lt;username&gt; createdb;`, the upgrade worked.

<@U02TQLBLDU4>, I'm not sure if I've misunderstood the documentation, or it's maybe worth updating..? (I'm happy to do it, just not sure if I missed something else obvious :see_no_evil:)