- Is this your first time deploying Airbyte?: Yes
- OS Version / Instance: Ubuntu 20.04.4 LTS
- Memory / Disk: 4Gb / 75 Gb
- Deployment: Docker deployment
- Airbyte Version: 0.39.42-alpha
Description: I’ve been facing some issues trying to deploy Airbyte using an external Postgres database located at Amazon RDS. Here the steps I followed:
-
git clone
-
cd /airbyte
-
Modify .env parameters:
DATABASE_USER=airbyte
DATABASE_PASSWORD=PASSWORD
DATABASE_HOST=HOST_NAME.eu-west-1.rds.amazonaws.com
DATABASE_PORT=5432
DATABASE_DB=airbyte
DATABASE_URL=jdbc:postgresql://HOST_NAME.eu-west-1.rds.amazonaws.com:5432/airbyte
CONFIG_DATABASE_USER=
CONFIG_DATABASE_PASSWORD=
CONFIG_DATABASE_URL= -
Remove db service config from docker-compose.yaml.
-
docker-compose up
After that, airbyte-temporal container starts giving some log errors about not being able to commit some transactions against the database (RDS). Furthermore, if I try to set a new source or upgrade its version, airbyte-server and airbyte-worker containers also start to give some errors with DEADLINE_EXCEEDED exception.
Here some of the logs:
airbyte-server | 2022-08-09 14:19:56 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure
airbyte-server | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context deadline exceeded
airbyte-server | at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-server | at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-server | at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-server | at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2642) ~[temporal-serviceclient-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.client.WorkflowClientLongPollHelper.lambda$getInstanceCloseEvent$0(WorkflowClientLongPollHelper.java:143) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) ~[temporal-serviceclient-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.client.WorkflowClientLongPollHelper.getInstanceCloseEvent(WorkflowClientLongPollHelper.java:131) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.client.WorkflowClientLongPollHelper.getWorkflowExecutionResult(WorkflowClientLongPollHelper.java:72) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.client.RootWorkflowClientInvoker.getResult(RootWorkflowClientInvoker.java:93) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:243) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:225) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) ~[temporal-sdk-1.8.1.jar:?]
airbyte-server | at jdk.proxy2.$Proxy85.run(Unknown Source) ~[?:?]
airbyte-server | at io.airbyte.workers.temporal.TemporalClient.lambda$submitCheckConnection$1(TemporalClient.java:113) ~[io.airbyte-airbyte-workers-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:466) ~[io.airbyte-airbyte-workers-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.workers.temporal.TemporalClient.submitCheckConnection(TemporalClient.java:112) ~[io.airbyte-airbyte-workers-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.scheduler.client.DefaultSynchronousSchedulerClient.lambda$createSourceCheckConnectionJob$0(DefaultSynchronousSchedulerClient.java:74) ~[io.airbyte.airbyte-scheduler-client-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.scheduler.client.DefaultSynchronousSchedulerClient.execute(DefaultSynchronousSchedulerClient.java:153) ~[io.airbyte.airbyte-scheduler-client-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.scheduler.client.DefaultSynchronousSchedulerClient.createSourceCheckConnectionJob(DefaultSynchronousSchedulerClient.java:70) ~[io.airbyte.airbyte-scheduler-client-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceCreate(SchedulerHandler.java:150) ~[io.airbyte-airbyte-server-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.server.apis.ConfigurationApi.lambda$executeSourceCheckConnection$77(ConfigurationApi.java:742) ~[io.airbyte-airbyte-server-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:873) ~[io.airbyte-airbyte-server-0.39.42-alpha.jar:?]
airbyte-server | at io.airbyte.server.apis.ConfigurationApi.executeSourceCheckConnection(ConfigurationApi.java:742) ~[io.airbyte-airbyte-server-0.39.42-alpha.jar:?]
airbyte-server | at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
airbyte-server | at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
airbyte-server | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
airbyte-server | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
airbyte-server | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
airbyte-server | at java.lang.Thread.run(Thread.java:1589) [?:?]
airbyte-worker | 2022-08-09 14:19:59 WARN i.t.i.a.HeartbeatContextImpl(doHeartBeatLocked):145 - Heartbeat failed
airbyte-worker | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.999875139s. [closed=[], open=[[remote_addr=airbyte-temporal/10.176.7.132:7233]]]
airbyte-worker | at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-worker | at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-worker | at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-worker | at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.recordActivityTaskHeartbeat(WorkflowServiceGrpc.java:2716) ~[temporal-serviceclient-1.8.1.jar:?]
airbyte-worker | at io.temporal.internal.client.ActivityClientHelper.sendHeartbeatRequest(ActivityClientHelper.java:61) ~[temporal-sdk-1.8.1.jar:?]
airbyte-worker | at io.temporal.internal.activity.HeartbeatContextImpl.sendHeartbeatRequest(HeartbeatContextImpl.java:180) ~[temporal-sdk-1.8.1.jar:?]
airbyte-worker | at io.temporal.internal.activity.HeartbeatContextImpl.doHeartBeatLocked(HeartbeatContextImpl.java:139) ~[temporal-sdk-1.8.1.jar:?]
airbyte-worker | at io.temporal.internal.activity.HeartbeatContextImpl.lambda$scheduleNextHeartbeatLocked$0(HeartbeatContextImpl.java:162) ~[temporal-sdk-1.8.1.jar:?]
airbyte-worker | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) [?:?]
airbyte-worker | at java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
airbyte-worker | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
airbyte-worker | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
airbyte-worker | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
airbyte-worker | at java.lang.Thread.run(Thread.java:1589) [?:?]
airbyte-server | 2022-08-09 14:20:01 INFO i.a.s.RequestLogger(filter):112 - REQ 10.176.7.134 GET 200 /api/v1/health
airbyte-webapp | 172.28.20.82 - - [09/Aug/2022:14:20:01 +0000] "GET /api/v1/health HTTP/1.1" 200 18 "http://iappairbyte01:8000/workspaces/8c9bfed5-d78c-4966-961e-6067e3cac1cb/source/new-source" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" "-"
airbyte-worker | 2022-08-09 14:20:10 WARN i.t.i.a.HeartbeatContextImpl(doHeartBeatLocked):145 - Heartbeat failed
airbyte-worker | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.997847497s. [closed=[], open=[[buffered_nanos=23218991, remote_addr=airbyte-temporal/10.176.7.132:7233]]]
airbyte-worker | at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-worker | at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-worker | at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.44.1.jar:1.44.1]
airbyte-worker | at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.recordActivityTaskHeartbeat(WorkflowServiceGrpc.java:2716) ~[temporal-serviceclient-1.8.1.jar:?]
airbyte-worker | at io.temporal.internal.client.ActivityClientHelper.sendHeartbeatRequest(ActivityClientHelper.java:61) ~[temporal-sdk-1.8.1.jar:?]
airbyte-worker | at io.temporal.internal.activity.HeartbeatContextImpl.sendHeartbeatRequest(HeartbeatContextImpl.java:180) ~[temporal-sdk-1.8.1.jar:?]
airbyte-worker | at io.temporal.internal.activity.HeartbeatContextImpl.doHeartBeatLocked(HeartbeatContextImpl.java:139) ~[temporal-sdk-1.8.1.jar:?]
airbyte-worker | at io.temporal.internal.activity.HeartbeatContextImpl.lambda$scheduleNextHeartbeatLocked$0(HeartbeatContextImpl.java:162) ~[temporal-sdk-1.8.1.jar:?]
airbyte-worker | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) [?:?]
airbyte-worker | at java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
airbyte-worker | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
airbyte-worker | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
airbyte-worker | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
airbyte-worker | at java.lang.Thread.run(Thread.java:1589) [?:?]
airbyte-temporal | {"level":"error","ts":"2022-08-09T14:20:10.770Z","msg":"transaction rollback error","error":"sql: transaction has already been committed or rolled back","logging-call-at":"common.go:75","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/common/persistence/sql.(*SqlStore).txExecute\n\t/temporal/common/persistence/sql/common.go:75\ngo.temporal.io/server/common/persistence/sql.(*sqlTaskManager).UpdateTaskQueue\n\t/temporal/common/persistence/sql/task.go:191\ngo.temporal.io/server/common/persistence.(*taskManagerImpl).UpdateTaskQueue\n\t/temporal/common/persistence/task_manager.go:174\ngo.temporal.io/server/common/persistence.(*taskRateLimitedPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceRateLimitedClients.go:499\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:716\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:252\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:168\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/temporal/internal/goro/group.go:57"}
airbyte-temporal | {"level":"error","ts":"2022-08-09T14:20:10.770Z","msg":"Operation failed with internal error.","error":"UpdateTaskQueue: read tcp 10.176.7.132:45546->10.172.82.28:5432: read: connection timed out","metric-scope":39,"logging-call-at":"persistenceMetricClients.go:1229","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/common/persistence.(*metricEmitter).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:1229\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:720\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:252\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:168\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/temporal/internal/goro/group.go:57"}
airbyte-temporal | {"level":"error","ts":"2022-08-09T14:20:10.770Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"GET_SPEC","wf-task-queue-type":"Activity","store-operation":"update-task-queue","error":"UpdateTaskQueue: read tcp 10.176.7.132:45546->10.172.82.28:5432: read: connection timed out","logging-call-at":"taskReader.go:171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:171\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/temporal/internal/goro/group.go:57"}
airbyte-temporal | {"level":"error","ts":"2022-08-09T14:20:10.774Z","msg":"transaction rollback error","error":"sql: transaction has already been committed or rolled back","logging-call-at":"common.go:75","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/common/persistence/sql.(*SqlStore).txExecute\n\t/temporal/common/persistence/sql/common.go:75\ngo.temporal.io/server/common/persistence/sql.(*sqlTaskManager).UpdateTaskQueue\n\t/temporal/common/persistence/sql/task.go:191\ngo.temporal.io/server/common/persistence.(*taskManagerImpl).UpdateTaskQueue\n\t/temporal/common/persistence/task_manager.go:174\ngo.temporal.io/server/common/persistence.(*taskRateLimitedPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceRateLimitedClients.go:499\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:716\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:252\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:168\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/temporal/internal/goro/group.go:57"}
airbyte-temporal | {"level":"error","ts":"2022-08-09T14:20:10.774Z","msg":"Operation failed with internal error.","error":"UpdateTaskQueue: read tcp 10.176.7.132:45544->10.172.82.28:5432: read: connection timed out","metric-scope":39,"logging-call-at":"persistenceMetricClients.go:1229","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/common/persistence.(*metricEmitter).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:1229\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:720\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:252\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:168\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/temporal/internal/goro/group.go:57"}
airbyte-temporal | {"level":"error","ts":"2022-08-09T14:20:10.774Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"fe82ec994426:41c3fc47-d826-4b11-87e5-ecd60cf14a60","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"UpdateTaskQueue: read tcp 10.176.7.132:45544->10.172.82.28:5432: read: connection timed out","logging-call-at":"taskReader.go:171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:171\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/temporal/internal/goro/group.go:57"}
I’ve checked that airbyte, temporal and temporal_visibility databases have been successfully created inside the database (RDS) and that airbyte user has the required grants over these databases.
I’ve also checked that I have connectivity from each of the 4 containers (airbyte-server, airbyte-worker, airbyte-webapp and airbyte-temporal) to the database (RDS).
Could you help me understand what I am doing wrong?
Thank you in advance!