[Kubernetes] Temporal deployment failling to connect with External Azure Postgres DB

Platform, Deploy & Infra Issues
, , , ,
1
  • Is this your first time deploying Airbyte?: No
  • Deployment: Kubernetes
  • Airbyte Version: 0.40.14
  • Description: I was trying to configure an external Azure Postgres DB to Airbyte using k8s, but temporal deployment somehow does not have access to the database. All .env and .secret variables are set, I also allowed all hosts in the Azure Postgres DB firewall. Airbyte-bootloader was able to connect and create main tables, but temporal deployment is having issues creating/accessing temporal and temporal_visibility databases (I have already created them manually, but it didn’t change the log output).
.env
DATABASE_HOST=XXXX.postgres.database.azure.com
DATABASE_PORT=5432
DATABASE_DB=airbyte
DATABASE_URL=jdbc:postgresql://<HOST>:<PORT>/<DB>?user=<USER>&password=<PASSWORD> (according to the other params)
CONFIG_DATABASE_USER=
CONFIG_DATABASE_PASSWORD=
CONFIG_DATABASE_URL=

.secret
DATABASE_USER=airbyte
DATABASE_PASSWORD=*********

airbyte-temporal.txt (4.2 KB)

2

Hey maybe have a look at this issue https://github.com/temporalio/temporal/issues/896 here. Incase that doesn’t help feel free to create a github issue so that team can look into it and help

3

Temporal incompatibility with PostgreSQL was fixed on version 1.2.1, we are running on version 1.7.0 here, so it shouldn’t be a problem. Don’t know if temporal.yaml or docker-compose.yaml is loading all necessary params for Azure PostgreSQL, maybe some other parameter needs to be included (but didn’t find any ref to that on both Airbyte and Azure PostgreSQL documentation).

4

Azure PostgreSQL require a SSL certificate that wasn’t available for temporal deployment, now that we have configured it, it works.

5

I have the same problem.

I tried to solve it by setting the temporal env variables SQL_TLS, SQL_TLS_ENABLED and SQL_TLS_DISABLE_HOST_VERIFICATION to true, but the only result is that airbyte-server waits until it dies for the temporal namespace default to become available.

What was the proper solution?

2023-02-20 10:40:57 e[33mWARNe[m i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):235 - Waiting for namespace default to be initialized in temporal...
2023-02-20 10:40:59 e[32mINFOe[m i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=9, target=airbyte-temporal:7233}}
2023-02-20 10:40:59 e[33mWARNe[m i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):245 - Ignoring exception while trying to request Temporal namespace:
io.grpc.StatusRuntimeException: NOT_FOUND: Namespace default does not exist.
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.51.1.jar:1.51.1]
6

Never mind, it seems that the temporal database setup was incomplete, I think it was able to create but not configure the temporal database before I added all the settings. Then it didnt re-apply the auto-setup when the database was already in place

7

@Erik can you please share how did you fix this problem I ran into the same as you on AKS v1.26 and Azure prostgres Flex Server v13.11 . the server crash with error : 2023-05-28 16:25:45 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):259 - Waiting for namespace default to be initialized in temporal…
2023-05-28 16:25:47 INFO i.t.s.WorkflowServiceStubsImpl():86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=5, target=airbyte-1685050831-temporal:7233}}
2023-05-28 16:25:47 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):269 - Ignoring exception while trying to request Temporal namespace: