How to increase max parallel sync count in Airbyte k8 deployment

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is trying to increase the max parallel sync count in Airbyte Kubernetes deployment using the Airbyte Helm chart 0.64.205. They are facing issues setting the TEMPORAL_WORKER_PORTS via values.yaml. The user noticed that TEMPORAL_WORKER_PORTS is hardcoded in env-configmap.yaml and deployment.yaml files. The questions are about updating the hardcoded values, having the same ports for multiple worker replica pods, and the discrepancy in the number of ports between the two files.


Question

Hello! i’m trying to understand hot to increase max parallel sync count in Airbyte k8 deployment (airbyte helm chart 0.64.205).

I tried to set TEMPORAL_WORKER_PORTS via values.yaml like

  settings:
    worker:
      replicaCount: 2
      extraEnv:
        - name: MAX_SPEC_WORKERS
          value: "32"
        - name: MAX_CHECK_WORKERS
          value: "16"
        - name: MAX_DISCOVER_WORKERS
          value: "16"
        - name: MAX_NOTIFY_WORKERS
          value: "16"
        - name: TEMPORAL_WORKER_PORTS
          value: 9001,9002,9003,9004,9005,9006,9007,9008,..., 9120```
But providing the "TEMPORAL_WORKER_PORTS" is not working.

I see that TEMPORAL_WORKER_PORTS hardcoded here airbyte/0.64.205/templates/env-configmap.yaml
```  TEMPORAL_WORKER_PORTS: 9001,9002,9003,9004,9005,9006,9007,9008,9009,9010,9011,9012,9013,9014,9015,9016,9017,9018,9019,9020,9021,9022,9023,9024,9025,9026,9027,9028,9029,9030,9031,9032,9033,9034,9035,9036,9037,9038,9039,9040```
And ports hardcoded here airbyte/0.64.205/charts/worker/templates/deployment.yaml
```        - containerPort: 9001 # start temporal worker port pool
        - containerPort: 9002
        - containerPort: 9003
        ...
        - containerPort: 9030```
So my questions is next
1. Do i need to update these hardcoded values?
2. Is it ok that i will have the same ports for a few workers replica pods?
3. Why in deployment.yaml only 30 ports, but in env-configmap.yaml - 40 ports?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1718403936361329) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["airbyte", "kubernetes-deployment", "helm-chart", "temporal-worker-ports", "replica-count", "env-configmap", "deployment.yaml", "container-ports"]
</sub>