Increasing Airbyte Workers in Self-Managed Environment

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

User is seeking guidance on how to increase the number of Airbyte workers in a self-managed open-source Airbyte server running on AWS EC2.


Question

Hi, We are running a self-managed open-source Airbyte server on our AWS EC2 instance. Could you please let us know how to increase the number of Airbyte workers in this environment?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['airbyte', 'self-managed', 'aws', 'ec2', 'workers']
2

You need to change the worker section in the values.yaml?

  enabled: true
  # -- Number of worker replicas
  replicaCount: 1```
3

I am not seeing any values.yaml file in my Airbyte directory. However, I do see the following files:

airbyte.log configs docker-compose.debug.yaml docker-compose.yaml flags.yml run-ab-platform.sh run-ab-platform.sh.1 temporal
If there are any other relevant files or configurations, could you kindly guide me on that?

4

This is a deprecated and no longer supported method https://github.com/airbytehq/airbyte/discussions/40599
I suggest migrating to abctl

5

Could you please assist me with the migration process?

6

Check docs and video how to do that
https://docs.airbyte.com/deploying-airbyte/migrating-from-docker-compose
https://www.youtube.com/watch?v=xwPi5_gV1ys

7
  1. Yes
  2. No, there is a discussion about it https://github.com/airbytehq/airbyte/discussions/41084