Optimizing Job Queue in Airbyte for Faster Connection Runtimes

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Airbyte platform experiencing delays in connection runtimes due to queuing jobs internally. Seeking recommendations to manage and optimize the job queue for improved performance.


Question

Hi everyone! :wave:
We’re currently using Airbyte as our main data collection tool, managing over 200+ workspaces, each with approximately 10 connections. Airbyte is deployed on a n2-standard-48 machine (48 vCPUs, 192 GB RAM) in GCP using abctl.
Recently, we’ve encountered a challenge: a connection that typically completes in just 5 minutes is now taking 2-3 hours. It seems like Airbyte is queuing jobs internally, which is causing significant delays.
Has anyone experienced a similar issue? Any recommendations on how to better manage or optimize the job queue to improve connection runtimes would be greatly appreciated! :rocket:
Thank you in advance for your insights! :raised_hands:


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte", "job-queue", "connection-runtimes", "optimization", "queuing-jobs"]