Based on the info in the docs, there are a couple of options for adjusting job parallelism.
SUBMITTER_NUM_THREADS- hard upper limit on the total number of concurrent jobs that can run across the entire system.
MAX_*_WORKERS- upper limit on the number of concurrent jobs that a single airbyte worker can run.
TEMPORAL_WORKER_PORTS- my understanding on this is still a little hazy, but it sounds like this should be equal to the sum of the
MAX_*_WORKERSvariables or else jobs will hang? Is this correct?
- replicas of
airbyte-worker- increases the number of airbyte workers running jobs. each airbyte worker can run up to
Could you give a concrete example of how to adjust these values to be able to run 50/100/1000 concurrent syncs?
Is there any more info on the relationship between
MAX_*_WORKERS and how these values interact?