Summary
The user is trying to increase the number of active threads for better writing performance in Airbyte. They have tried various configurations but the active threads remain stuck at 2. They are seeking guidance on what configuration they might be missing.
Question
How to increase active threads?
Hi team, I have been trying in all ways to increase the number of active threads (i.e. The number of processes which send batches to the writer) with no success. This is the trace I am aiming to increase:
2024-08-19 17:52:56 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 2
Regardless of the resources and environment variables, the amount of active threads is stuck in only 2.
I am trying to understand which configuration I am missing to increase the number of active threads and improve the writing performance.
Has someone any guidance or hint?
Below some of the configuration I have made this far. Happy to share more details:
The job resources have the following:
requests:
cpu: 5
memory: 1Gi
limits:
cpu: 8
memory: 8Gi
I currently have 2 workers replica with the following resources:
replicaCount: 2
(...)
limits:
cpu: 5000m
memory: 5Gi
cpu: 3000m
memory: 2Gi
For the wokers pod and the temporal pod I added the following environmental variables:
extraEnv:
- name: TEMPORAL_CLI_SHOW_STACKS
value: "TEMPORAL_CLI_SHOW_STACKS=1"
- name: SQL_MAX_CONNS
value: "40"
- name: MICRONAUT_SERVER_MAX_REQUEST_SIZE
value: "52328800"
- name: worker_replica_count
value: "12"
- name: max_sync_workers
value: "400"
- name: max_spec_workers
value: "100"
- name: max_check_workers
value: "100"
- name: max_discover_workers
value: "100"
- name: SQL_MAX_CONNS
value: "40"
- name: SUBMITTER_NUM_THREADS
value: "200"
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.