Increasing Max Parallel Sync Count in Airbyte Kubernetes Deployment

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is attempting to increase the maximum parallel sync count in an Airbyte Kubernetes deployment using Helm chart 0.64.205. They are configuring the TEMPORAL_WORKER_PORTS in values.yaml but are facing issues due to hardcoded values in the templates. They seek clarification on updating hardcoded values, port usage for worker replicas, and discrepancies in the number of ports defined in different configuration files.


Question

Hello! i’m trying to understand hot to increase max parallel sync count in Airbyte k8 deployment (airbyte helm chart 0.64.205).

I tried to set TEMPORAL_WORKER_PORTS via values.yaml like

  settings:
    worker:
      replicaCount: 2
      extraEnv:
        - name: MAX_SPEC_WORKERS
          value: "32"
        - name: MAX_CHECK_WORKERS
          value: "16"
        - name: MAX_DISCOVER_WORKERS
          value: "16"
        - name: MAX_NOTIFY_WORKERS
          value: "16"
        - name: TEMPORAL_WORKER_PORTS
          value: 9001,9002,9003,9004,9005,9006,9007,9008,..., 9120```
But providing the "TEMPORAL_WORKER_PORTS" is not working.

I see that TEMPORAL_WORKER_PORTS hardcoded here airbyte/0.64.205/templates/env-configmap.yaml
```  TEMPORAL_WORKER_PORTS: 9001,9002,9003,9004,9005,9006,9007,9008,9009,9010,9011,9012,9013,9014,9015,9016,9017,9018,9019,9020,9021,9022,9023,9024,9025,9026,9027,9028,9029,9030,9031,9032,9033,9034,9035,9036,9037,9038,9039,9040```
And ports hardcoded here airbyte/0.64.205/charts/worker/templates/deployment.yaml
```        - containerPort: 9001 # start temporal worker port pool
        - containerPort: 9002
        - containerPort: 9003
        ...
        - containerPort: 9030```
So my questions is next
1. Do i need to update these hardcoded values?
2. Is it ok that i will have the same ports for a few workers replica pods?
3. Why in deployment.yaml only 30 ports, but in env-configmap.yaml - 40 ports?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1718403936361329) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['max-parallel-sync', 'airbyte-k8s', 'helm-chart', 'temporal-worker-ports', 'values.yaml']
</sub>