Airbyte-Server connection with Airbyte-Temporal seems to fail with Docker-Compose Up (Centos 7)

Platform, Deploy & Infra Issues
,
1

  • Is this your first time deploying Airbyte?: Yes

  • OS Version / Instance: CentOS 7

  • Memory / Disk: you can use something like 32Gb / 500Gb

  • Deployment: Docker-Compose

  • Airbyte Version: 0.39.38-alpha

  • Source name/version: n/a

  • Destination name/version: n/a

  • Step: The issue is happening during deployment with docker-compose up

  • Description: I’ve been facing some issues trying to deploy Airbyte using docker-compose command.

    I pulled all the files from github and after that I ran docker-compose up without any changes to Airbyte files.
    All of the containers are up and running state but I cannot not reach to the web ui from the web console.
    It says “Cannot reach server. The server may still be starting up” and gives 502 Bad gateway Error when try to access /api/v1/workspace/list path.

    After checking the airbyte-server and airbyte-worker container logs, I noticed that they have the same warning logs about airbyte-temporal server.
    Container log shows to us airbyte-server is waiting for temporal server and waiting for namespace default to be initialized in temporal.
    Then retrying failure and DEADLINE_EXCEEF exception. However it stucks in that point and cannot connect or start synchronizing.

    I’ve tried 0.40.23 and 0.40.24 versions, docker-compose up and down commands and disable firewalld service. I always face the same situation.

Here some of the airbyte-server logs:

2022-12-14 09:17:41 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONFIG_DATABASE_USER: 'docker'
2022-12-14 09:17:41 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****'
2022-12-14 09:17:41 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:postgresql://db:5432/airbyte'
2022-12-14 09:17:41 e[32mINFOe[m c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2022-12-14 09:17:41 e[32mINFOe[m c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2022-12-14 09:17:41 e[32mINFOe[m c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting...
2022-12-14 09:17:41 e[32mINFOe[m c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed.
2022-12-14 09:17:42 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-12-14 09:17:42 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'
2022-12-14 09:17:42 e[32mINFOe[m i.a.s.ServerApp(getServer):163 - Checking databases..
2022-12-14 09:17:42 e[32mINFOe[m i.a.s.ServerApp(assertDatabasesReady):139 - Checking configs database flyway migration version..
2022-12-14 09:17:42 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONFIGS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-12-14 09:17:42 e[33mWARNe[m i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2022-12-14 09:17:42 e[32mINFOe[m i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection...
2022-12-14 09:17:42 e[32mINFOe[m i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
2022-12-14 09:17:43 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:43 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://db:5432/airbyte (PostgreSQL 13.7)
2022-12-14 09:17:43 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):46 - Current database migration version 0.
2022-12-14 09:17:43 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):47 - Minimum Flyway version required 0.35.15.001.
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):62 - Verified that database has been migrated to the required minimum version 60000.
2022-12-14 09:17:49 e[32mINFOe[m i.a.s.ServerApp(assertDatabasesReady):145 - Checking jobs database flyway migration version..
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable JOBS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-12-14 09:17:49 e[33mWARNe[m i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte jobs database connection...
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://db:5432/airbyte (PostgreSQL 13.7)
2022-12-14 09:17:49 e[32mINFOe[m o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):46 - Current database migration version 0.35.62.001.
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):47 - Minimum Flyway version required 0.29.15.001.
2022-12-14 09:17:49 e[32mINFOe[m i.a.d.c.DatabaseMigrationCheck(check):62 - Verified that database has been migrated to the required minimum version 60000.
2022-12-14 09:17:49 e[32mINFOe[m i.a.s.ServerApp(getServer):166 - Creating Staged Resource folder...
2022-12-14 09:17:49 e[32mINFOe[m i.a.s.ServerApp(getServer):169 - Creating config repository...
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-12-14 09:17:49 e[32mINFOe[m i.a.s.ServerApp(getServer):185 - Creating jobs persistence...
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable TEMPORAL_HISTORY_RETENTION_IN_DAYS: '30'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '5'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS: '30'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS: '600'
2022-12-14 09:17:49 e[32mINFOe[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable TEMPORAL_CLOUD_ENABLED: 'false'
2022-12-14 09:17:49 e[32mINFOe[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for temporal server...
2022-12-14 09:17:49 e[33mWARNe[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):243 - Waiting for namespace default to be initialized in temporal...
2022-12-14 09:17:56 e[33mWARNe[m i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.988404657s. 
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.44.1.jar:1.44.1]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:286) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:279) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:186) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.lambda$createTemporalService$0(TemporalUtils.java:72) ~[io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.getTemporalClientWhenConnected(TemporalUtils.java:248) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:68) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:83) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:87) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.server.ServerApp.getServer(ServerApp.java:199) [io.airbyte-airbyte-server-0.39.38-alpha.jar:?]
	at io.airbyte.server.ServerApp.main(ServerApp.java:288) [io.airbyte-airbyte-server-0.39.38-alpha.jar:?]
2022-12-14 09:18:01 e[33mWARNe[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):253 - Ignoring exception while trying to request Temporal namespace:
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998956601s. 
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.44.1.jar:1.44.1]
	at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.44.1.jar:1.44.1]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:286) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:279) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:186) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.lambda$createTemporalService$0(TemporalUtils.java:72) ~[io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.getTemporalClientWhenConnected(TemporalUtils.java:248) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:68) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:83) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:87) [io.airbyte-airbyte-workers-0.39.38-alpha.jar:?]
	at io.airbyte.server.ServerApp.getServer(ServerApp.java:199) [io.airbyte-airbyte-server-0.39.38-alpha.jar:?]
	at io.airbyte.server.ServerApp.main(ServerApp.java:288) [io.airbyte-airbyte-server-0.39.38-alpha.jar:?]

airbyte-server.log (23.7 KB)
airbyte-temporal.log (82.8 KB)

Could you help me understand what I am doing wrong?
Thank you in advance!

2

Can you try different OS?

3

Hello,

First of all, I appreciate for your answer.

However, I have a project related to the development of an existing system and airbyte is one of
the part of it. That’s why, I cannot change the OS and must find a solution for problem.

Have you encountered similar errors before and is there a different solution you can suggest?

4

I found it, there is a guide to deploy in CentOS https://github.com/airbytehq/airbyte/blob/master/docs/archive/faq/deploying-on-other-os.md make sure you follow it. It is for CentOS 8, hope works for 7 too.