Airbyte-Server connection with Airbyte-Temporal seems to fail with Docker-Compose Up (Centos 7)

Auzun · December 14, 2022, 12:04pm

Is this your first time deploying Airbyte?: Yes
OS Version / Instance: CentOS 7
Memory / Disk: you can use something like 32Gb / 500Gb
Deployment: Docker-Compose
Airbyte Version: 0.39.38-alpha
Source name/version: n/a
Destination name/version: n/a
Step: The issue is happening during deployment with docker-compose up
Description: I’ve been facing some issues trying to deploy Airbyte using docker-compose command.

I pulled all the files from github and after that I ran docker-compose up without any changes to Airbyte files.
All of the containers are up and running state but I cannot not reach to the web ui from the web console.
It says “Cannot reach server. The server may still be starting up” and gives 502 Bad gateway Error when try to access /api/v1/workspace/list path.

After checking the airbyte-server and airbyte-worker container logs, I noticed that they have the same warning logs about airbyte-temporal server.
Container log shows to us airbyte-server is waiting for temporal server and waiting for namespace default to be initialized in temporal.
Then retrying failure and DEADLINE_EXCEEF exception. However it stucks in that point and cannot connect or start synchronizing.

I’ve tried 0.40.23 and 0.40.24 versions, docker-compose up and down commands and disable firewalld service. I always face the same situation.

Here some of the airbyte-server logs:

2022-12-14 09:17:41 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONFIG_DATABASE_USER: 'docker'
2022-12-14 09:17:41 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****'
2022-12-14 09:17:41 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:postgresql://db:5432/airbyte'
2022-12-14 09:17:41 e[32mINFOe[m c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2022-12-14 09:17:41 e[32mINFOe[m c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2022-12-14 09:17:41 e[32mINFOe[m c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting...
2022-12-14 09:17:41 e[32mINFOe[m c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed.
2022-12-14 09:17:42 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-12-14 09:17:42 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'
2022-12-14 09:17:42 e[32mINFOe[m i.a.s.ServerApp(getServer):163 - Checking databases..
2022-12-14 09:17:42 e[32mINFOe[m i.a.s.ServerApp(assertDatabasesReady):139 - Checking configs database flyway migration version..
2022-12-14 09:17:42 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONFIGS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-12-14 09:17:42 e[33mWARNe[m i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2022-12-14 09:17:42 e[32mINFOe[m i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection...
2022-12-14 09:17:42 e[32mINFOe[m i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
2022-12-14 09:17:43 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:43 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://db:5432/airbyte (PostgreSQL 13.7)
2022-12-14 09:17:43 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):46 - Current database migration version 0.
2022-12-14 09:17:43 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):47 - Minimum Flyway version required 0.35.15.001.
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):62 - Verified that database has been migrated to the required minimum version 60000.
2022-12-14 09:17:49 e[32mINFOe[m i.a.s.ServerApp(assertDatabasesReady):145 - Checking jobs database flyway migration version..
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable JOBS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-12-14 09:17:49 e[33mWARNe[m i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte jobs database connection...
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://db:5432/airbyte (PostgreSQL 13.7)
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):46 - Current database migration version 0.35.62.001.
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):47 - Minimum Flyway version required 0.29.15.001.
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):62 - Verified that database has been migrated to the required minimum version 60000.
2022-12-14 09:17:49 e[32mINFOe[m i.a.s.ServerApp(getServer):166 - Creating Staged Resource folder...
2022-12-14 09:17:49 e[32mINFOe[m i.a.s.ServerApp(getServer):169 - Creating config repository...
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-12-14 09:17:49 e[32mINFOe[m i.a.s.ServerApp(getServer):185 - Creating jobs persistence...
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable TEMPORAL_HISTORY_RETENTION_IN_DAYS: '30'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '5'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS: '30'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS: '600'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable TEMPORAL_CLOUD_ENABLED: 'false'
2022-12-14 09:17:49 e[32mINFOe[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for temporal server...
2022-12-14 09:17:49 e[33mWARNe[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):243 - Waiting for namespace default to be initialized in temporal...
2022-12-14 09:17:56 e[33mWARNe[m i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.988404657s. 
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.44.1.jar:1.44.1]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:286) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:279) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:186) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.lambda$createTemporalService$0(TemporalUtils.java:72) ~[io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.getTemporalClientWhenConnected(TemporalUtils.java:248) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:68) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:83) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:87) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.server.ServerApp.getServer(ServerApp.java:199) [io.airbyte-airbyte-server-0.39.38-alpha.jar:?]
	at io.airbyte.server.ServerApp.main(ServerApp.java:288) [io.airbyte-airbyte-server-0.39.38-alpha.jar:?]
2022-12-14 09:18:01 e[33mWARNe[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):253 - Ignoring exception while trying to request Temporal namespace:
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998956601s. 
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.44.1.jar:1.44.1]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:286) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:279) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:186) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.lambda$createTemporalService$0(TemporalUtils.java:72) ~[io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.getTemporalClientWhenConnected(TemporalUtils.java:248) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:68) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:83) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:87) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.server.ServerApp.getServer(ServerApp.java:199) [io.airbyte-airbyte-server-0.39.38-alpha.jar:?]
	at io.airbyte.server.ServerApp.main(ServerApp.java:288) [io.airbyte-airbyte-server-0.39.38-alpha.jar:?]

airbyte-server.log (23.7 KB)
airbyte-temporal.log (82.8 KB)

Could you help me understand what I am doing wrong?
Thank you in advance!

marcosmarxm · December 14, 2022, 3:59pm

Can you try different OS?

Auzun · December 15, 2022, 7:17am

Hello,

First of all, I appreciate for your answer.

However, I have a project related to the development of an existing system and airbyte is one of
the part of it. That’s why, I cannot change the OS and must find a solution for problem.

Have you encountered similar errors before and is there a different solution you can suggest?

marcosmarxm · December 15, 2022, 7:43pm

I found it, there is a guide to deploy in CentOS https://github.com/airbytehq/airbyte/blob/master/docs/archive/faq/deploying-on-other-os.md make sure you follow it. It is for CentOS 8, hope works for 7 too.

Topic		Replies	Views
Error installing/deploying airbyte \| trying to create an HTTP API Source Platform, Deploy & Infra Issues getting-started , connectors , deploy	9	1198	July 14, 2022
Airbyte-temporal started failed after execute docker-compose up Platform, Deploy & Infra Issues deploy	2	1214	February 13, 2023
Error deploying Airbyte with an external Postgres database (AWS RDS) Platform, Deploy & Infra Issues deployment , deploy , temporal	2	1637	September 7, 2022
Issue with docker-compose up when installing Airbyte locally on Mac Platform Questions docker , platform , question , airbyte-temporal , docker-compose	2	37	May 14, 2024
Trouble connecting Airbyte to PostgreSQL instance on remote machine Platform Questions platform , airbyte , question , airbyte-temporal , docker-compose	1	45	May 14, 2024

Airbyte-Server connection with Airbyte-Temporal seems to fail with Docker-Compose Up (Centos 7)

Related topics