Issue with creating source or destination using external Postgres DB

Summary

Error ‘context deadline exceeded’ is encountered while trying to create a source or destination using an external Postgres database.


Question

Hey guys!
Has anyone seen this issue before?

airbyte-temporal | {"level":"error","ts":"2024-02-26T15:36:20.389Z","msg":"service failures","operation":"RecordWorkflowTaskStarted","wf-namespace":"default","error":"GetWorkflowExecution: failed. Error: context deadline exceeded","logging-call-at":"telemetry.go:341","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error|go.temporal.io/server/common/log.(*zapLogger).Error>\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:341\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:174\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/metrics.NewServerMetricsTrailerPropagatorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:113\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.42.0/interceptor.go:344\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:145\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1186\ngo.temporal.io/server/api/historyservice/v1._HistoryService_RecordWorkflowTaskStarted_Handler\n\t/home/builder/temporal/api/historyservice/v1/service.pb.go:1231\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1376\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1753\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:998"}
I’m not able to create any source or destination (only happens while using external postgres db)



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["issue", "creating-source", "creating-destination", "external-postgres-db", "context-deadline-exceeded"]

Not my part of the codebase, but looks like high latency in temporal when it tries to start a job. If that only happens with external Pg, must be high latency of that database?

Hey <@U069EMNRPA4>
Sorry to bother you. Would you be so kind to update me on the status of this issue? I would really appreciate it

Sorry! We have an “everyone in the office” week and it’s a bit chaotic.

Hi <@U069EMNRPA4> is there anything we can do to work around this? I’m also facing these temporal issues and fails to start sync pods, even the UI takes ages to load. Running Helm 0.61.0 on K8

<@U069EMNRPA4> to give you more context we know for sure that the issue happens when an external PG database is configured

That seems to be pretty popular huh, let me link this to our platform team and see what they know

Yeah. If there’s any way to override the timeout params via the temporal args in the helm chart would be great. Communication seems to be the issue from the logs

Besides that, of course it should not take minutes to create a source

Thank you <@U069EMNRPA4> ! We would really appreciate your assistance on this matter.

thanks a lot <@U069EMNRPA4> :+1:

Hey <@U069EMNRPA4>! Are there any updates on this matter?
Also, <@U05CXP45USF> , how is it going on your end? Maybe you’ve found a solution?

Hey, no I didn’t. Still using the inbuilt postgres :confused:

latency is just way too long as I have 2500 connections and the UI doesnt work at all.