Kubernetes: temporal: Cannot start with empty DBs already created

Summary - Unable to start temporal on K8s with pre-created temporal and temporal_visibility empty DBs in postgres. User is not expected to have createDB permissions but has Grant All on the DBs.

I am trying to run Airbyte on my local docker desktop with an external Postgres instance located on the host machine. The airbyte-bootloader completes successfully and i can see tables in the airbyte DB. Other deployments like connector-builder-server, cron, pod-sweeper and worker are also up.
However, temporal is unable to start and hence, server and webapp are also down.

In accordance with my production scenario, i have already created the temporal and temporal_visibility DB as in production I will not get createDB permissions and DBs will be created in advance.
airbyteuser has been given GRANT ALL on these DBs.

The following variables and their values are showing up in temporal container logs as correctly configured.

  • DB=postgresql
  • DBNAME=temporal
  • VISIBILITY_DBNAME=temporal_visibility
  • DB_PORT=5432
  • POSTGRES_SEEDS=host.docker.internal
  • POSTGRES_USER=airbyteuser

However, this is where the script is failing.

  • temporal-sql-tool --plugin postgres --ep host.docker.internal -u airbyteuser -p 5432 create --db temporal
    2023-05-03T10:32:24.233Z ERROR Unable to create SQL database. {“error”: “pq: permission denied to create database”, “logging-call-at”: “handler.go:97”}

I have also tried to change the DBNAME and VISIBILITY_DBNAME to custom values e.g. airbyte_temporal. That does not solve the problem.

Am i missing a step when configuring the external DB, or do i need to do more than just create the DB and grant permissions?
From a production perspective, does this mean that DB creation and schema setup must be done separately and configure to run the airbyte helm install with SKIP_SCHEMA_SETUP=true?

Also, Airbyte seems to be using temporal 1.13 which is a much older version and the auto-setup image is showing CVEs against it. Is there an upgrade to the latest temporal version planned? What is the highest version of temporal that Airbyte can work with?

1 Like

A few more experiments later this is what i am experiencing :-

It does seem like the Temporal auto-setup requires a user/role with CreateDB permissions. There doesn’t seem to be any way around this (tried with v1.13.0, v1.15.0, v1.20.2)
Even with a user that has CreateDB permission (e.g. postgres user), Temporal v1.13.0 complains (DB already exists) if the temporal and visibility DBs have been pre-created.
Temporal latest (v1.20.2) seems to run into compatibility issues with Airbyte - i did not capture the exact issues.
Temporal 1.15.0 seems to be able to handle pre-created DBs (albeit still requires a user with CreateDB permissions) and does not seem to have compatibility issues with Airbyte.

Is it advisable to run Airbyte with Temporal 1.15.0 in Production?
Are there steps for how to correctly perform Temporal version upgrades in a production instance?

1 Like

I get same issue with you