Trouble configuring Airbyte to use existing Postgres database in K8s cluster

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

User is facing issues configuring Airbyte to use an existing Postgres database in a K8s cluster. The Helm charts documentation is out of date, and the temporal pod is complaining about not being able to connect to a temporal database. User created the temporal database but is now encountering a schema version compatibility error.


Question

Hi. I’m trying to use the latest Helm charts (0.61.0) to get Airbyte running in a K8s cluster. Other than the documentation of the chart’s values apparently being severely out of date, I am struggling with making Airbyte use my existing Postgres database. I’ve managed to get the bootstrapper to initialize the database, but now the temporal pod is complaining about not being able to connect to a temporal database. I’m not seeing any evidence in the Helm charts that this is even configurable?

I eventually caved and created the extra temporal database it was asking for, but now I’m getting the following error:

sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "REDACTED", user "REDACTED", database "temporal", no encryption
I’ve verified that the user it is trying to connect as has access to that database. Any hints? Thanks.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["airbyte", "postgres-database", "k8s-cluster", "helm-charts", "temporal-pod", "schema-version", "compatibility-error"]
2

I had troubles with existing databases too

My solution:

  1. Use airbyte/temporal instead of temporalio/auto-setup (temporalio/auto-setup has several problems for me: hostname without key -i in entrypoint.sh so i get ipv6 address instead of ipv4; i think old auto-setup doesn’t allow ot skip database creation)
  2. Set env variable SKIP_DB_CREATE=true
  3. I needed to disable TLS verification (i hope to fix it in the future)
    My current values for temporal are:
  enabled: true
  replicaCount: 1

  image:
    repository: airbyte/temporal
    pullPolicy: IfNotPresent
    tag: "${airbyteVersion}"

  service:
    type: ClusterIP
    port: 7233

  podAnnotations: {}
  podLabels: {}
  containerSecurityContext: {}
  extraInitContainers: []

  livenessProbe:
    enabled: false

  readinessProbe:
    enabled: false

  nodeSelector: ${nodeSelector}
  tolerations: ${tolerations}

  affinity: {}

  extraEnv:
    - name: DB_PORT
      value: "5432" # FIXME: pgbouncer doesn't work with old driver in temporal auto-setup
    - name: DBNAME
      value: "${temporalDbName}"
    - name: VISIBILITY_DBNAME
      value: "${temporalVisibilityDbName}"
    - name: SKIP_DB_CREATE
      value: "true"
    - name: SQL_TLS  # for migrations
      value: "true"
    - name: SQL_TLS_ENABLED  # for server
      value: "true"
#    - name: SQL_CONNECT_ATTRIBUTES  # for migration to work with transaction pooling <https://community.temporal.io/t/postgres-pq-pgbouncer-and-prepared-statements/5644>
#      value: "binary_parameters=yes"
    - name: SQL_TLS_DISABLE_HOST_VERIFICATION  # FIXME
      value: "true"
    - name: SQL_HOST_VERIFICATION  # FIXME
      value: "false"```