Issue with creating source or destination using external Postgres DB

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Error ‘context deadline exceeded’ is encountered while trying to create a source or destination using an external Postgres database.


Question

Hey guys!
Has anyone seen this issue before?

airbyte-temporal | {"level":"error","ts":"2024-02-26T15:36:20.389Z","msg":"service failures","operation":"RecordWorkflowTaskStarted","wf-namespace":"default","error":"GetWorkflowExecution: failed. Error: context deadline exceeded","logging-call-at":"telemetry.go:341","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error|go.temporal.io/server/common/log.(*zapLogger).Error>\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:341\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:174\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/metrics.NewServerMetricsTrailerPropagatorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:113\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.42.0/interceptor.go:344\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:145\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1186\ngo.temporal.io/server/api/historyservice/v1._HistoryService_RecordWorkflowTaskStarted_Handler\n\t/home/builder/temporal/api/historyservice/v1/service.pb.go:1231\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1376\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1753\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:998"}
I’m not able to create any source or destination (only happens while using external postgres db)


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["issue", "creating-source", "creating-destination", "external-postgres-db", "context-deadline-exceeded"]
2

Not my part of the codebase, but looks like high latency in temporal when it tries to start a job. If that only happens with external Pg, must be high latency of that database?

3

Hey <@U069EMNRPA4>
Sorry to bother you. Would you be so kind to update me on the status of this issue? I would really appreciate it

4

Sorry! We have an “everyone in the office” week and it’s a bit chaotic.

5

Hi <@U069EMNRPA4> is there anything we can do to work around this? I’m also facing these temporal issues and fails to start sync pods, even the UI takes ages to load. Running Helm 0.61.0 on K8

6

<@U069EMNRPA4> to give you more context we know for sure that the issue happens when an external PG database is configured

7

That seems to be pretty popular huh, let me link this to our platform team and see what they know

8

Yeah. If there’s any way to override the timeout params via the temporal args in the helm chart would be great. Communication seems to be the issue from the logs

9

Besides that, of course it should not take minutes to create a source

10

Thank you <@U069EMNRPA4> ! We would really appreciate your assistance on this matter.

11

thanks a lot <@U069EMNRPA4> :+1:

12

Hey <@U069EMNRPA4>! Are there any updates on this matter?
Also, <@U05CXP45USF> , how is it going on your end? Maybe you’ve found a solution?

13

Hey, no I didn’t. Still using the inbuilt postgres :confused:

14

latency is just way too long as I have 2500 connections and the UI doesnt work at all.

15

Hey guys! Did you manage to connect an external postgres?

16

We are running Airbyte with an external postgres server for the internal database

17

What’s your airbyte version?

18

0.57.2 but we have been using external postgres for many versions

19

How did you set it up? I encounter lots of ‘temporal’ issues as you can see in the thread above

20

Where is the db hosted btw?