Connection Performance Issue in Airbyte

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User reports a significant increase in connection runtime from 5 minutes to 2-3 hours due to internal job queuing in Airbyte deployed on GCP. Seeking recommendations for managing or optimizing the job queue.


Question

Hi everyone! :wave:
We’re currently using Airbyte as our main data collection tool, managing over 200+ workspaces, each with approximately 10 connections. Airbyte is deployed on a n2-standard-48 machine (48 vCPUs, 192 GB RAM) in GCP using abctl.
Recently, we’ve encountered a challenge: a connection that typically completes in just 5 minutes is now taking 2-3 hours. It seems like Airbyte is queuing jobs internally, which is causing significant delays.
Has anyone experienced a similar issue? Any recommendations on how to better manage or optimize the job queue to improve connection runtimes would be greatly appreciated! :rocket:
Thank you in advance for your insights! :raised_hands:


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['airbyte', 'gcp', 'abctl', 'job-queue', 'connection-runtime']
2

Hello Adil this is an interesting question. Did you check this https://docs.airbyte.com/operator-guides/scaling-airbyte|documentation or change anything related to worker section in the values.yaml?

  enabled: true
  # -- Number of worker replicas
  replicaCount: 1```
If you increase the number of workers it allows to run more job in parallel.