Summary
Airbyte platform experiencing delays in connection runtimes due to queuing jobs internally. Seeking recommendations to manage and optimize the job queue for improved performance.
Question
Hi everyone! ![]()
We’re currently using Airbyte as our main data collection tool, managing over 200+ workspaces, each with approximately 10 connections. Airbyte is deployed on a n2-standard-48 machine (48 vCPUs, 192 GB RAM) in GCP using abctl.
Recently, we’ve encountered a challenge: a connection that typically completes in just 5 minutes is now taking 2-3 hours. It seems like Airbyte is queuing jobs internally, which is causing significant delays.
Has anyone experienced a similar issue? Any recommendations on how to better manage or optimize the job queue to improve connection runtimes would be greatly appreciated! ![]()
Thank you in advance for your insights! ![]()
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.