Error with custom CloudSQL Postgres instance configuration causing airbyte-temporal errors and 502 response

Summary

After changing the database config to a custom CloudSQL Postgres instance in Airbyte, encountering airbyte-temporal errors and eventually a 502 response when trying to create a test e2e source.


Question

Hi there!

I’ve pulled the latest airbyte version from master branch and started the server. It all was fine with the default configuration however once I changed the database config to a custom CloudSQL Postgres instance I still managed to get the server started but once I tried creating a test e2e source I got a lot of airbyte-temporal errors in the console and eventually 502 response from the server. I really could use some help. Thank you in advance!

airbyte-temporal | {"level":"error","ts":"2024-02-11T02:47:02.758Z","msg":"Fail to process task","shard-id":2,"address":"172.23.0.3:7234","component":"transfer-queue-processor","wf-namespace-id":"42808cc7-10a3-42e1-a1fa-0144403f5fcb","wf-id":"697b9a5a-f599-43e4-9243-8f55c5a68cc1","wf-run-id":"c0f0d2cc-d028-4d55-91d9-b8fc0f75c755","queue-task-id":13631497,"queue-task-visibility-timestamp":"2024-02-11T02:46:57.982Z","queue-task-type":"TransferWorkflowTask","queue-task":{"NamespaceID":"42808cc7-10a3-42e1-a1fa-0144403f5fcb","WorkflowID":"697b9a5a-f599-43e4-9243-8f55c5a68cc1","RunID":"c0f0d2cc-d028-4d55-91d9-b8fc0f75c755","VisibilityTimestamp":"2024-02-11T02:46:57.98236996Z","TaskID":13631497,"TaskQueue":"CHECK_CONNECTION","ScheduledEventID":15,"Version":0},"wf-history-event-id":15,"error":"shard status unknown","lifecycle":"ProcessingFailed","logging-call-at":"lazy_logger.go:68","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error|go.temporal.io/server/common/log.(*zapLogger).Error>\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:347\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["custom-cloudsql-postgres-instance", "airbyte-temporal-errors", "502-response", "database-configuration"]

Can you check your CloudSQL if it was able to create the temporal and temporal_visibility databases. The worker and server have any errors?

It was.
temporal , temporal_visibility and airbyte databased were created.
I decided to launch a server with airbyte v0.50.7 from scratch and the issue disappeared. However, I still get occasional errors from airbyte-temporal and the syncs are really slow. It takes 3 minutes for a sync job to be seen as started int the UI after I click Sync

So basically the issue Im getting persists through different versions at different level of impact on app performance

It doesn’t look right, my suggestion is to open a Github ticket and describe exactly you done to run Airbyte (what changes in the default configs and how you’re trying to access the CloudSQL)

I appreciate your help! Will do