Understanding workers, jobs and concurrency

Q&A
1

As per the documentation - all interactions with connectors are run as jobs performed by a Worker and workers can be spec worker, check connection worker, discovery worker or sync worker.

Question 1 - Can a worker perform only a single job at a time or even more than one ?

Also for configuring parallelization, we have different parameters - MAX_*_WORKERS

  • As per this link, these are Maximum number of * workers allowed to run in parallel
  • As per this second link, these are maximum number of * workers each Airbyte Worker container can support

Question 2 - Are these two definitions different ? What is this Airbyte Worker ? Is this worker different from the other four types of workers - spec/check/discover/sync ?

We have another configuration parameter SUBMITTER_NUM_THREADS - the maximum number of concurrent jobs the Scheduler schedules.

Question 3 - How does this last configuration parameter relate to the MAX_*_WORKERS ?

2

Hey,

Worker → Which is responsible for creating sync/check/discover/spec pods → you can check about these here https://docs.airbyte.com/understanding-airbyte/jobs

Thus the MAX_*_WORKERS control these pod creations

3

Hey Harshith, thanks for the reply. But I got confused by reading the particular documentation link that you have mentioned. By the first As per the documentation, I meant that particular page (couldn’t add more than two links in the question - so had to remove it, because I already had other links)

Anyway, can you please specifically answer the three questions that I have asked ?

4
  1. A Worker can perform more jobs and that is dependent on the PORTS open.
  2. Airbyte worker is the worker container and spec/check/discover/sync are the jobs the worker can schedule
  3. MAX_*_WORKER controls the jobs that are created by the worker
5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.