As per the documentation - all interactions with connectors are run as jobs performed by a Worker and workers can be spec worker, check connection worker, discovery worker or sync worker.
Question 1 - Can a worker perform only a single job at a time or even more than one ?
Also for configuring parallelization, we have different parameters -
- As per this link, these are Maximum number of * workers allowed to run in parallel
- As per this second link, these are maximum number of * workers each Airbyte Worker container can support
Question 2 - Are these two definitions different ? What is this Airbyte Worker ? Is this worker different from the other four types of workers - spec/check/discover/sync ?
We have another configuration parameter
SUBMITTER_NUM_THREADS - the maximum number of concurrent jobs the Scheduler schedules.
Question 3 - How does this last configuration parameter relate to the
Worker → Which is responsible for creating sync/check/discover/spec pods → you can check about these here https://docs.airbyte.com/understanding-airbyte/jobs
Thus the MAX_*_WORKERS control these pod creations
Hey Harshith, thanks for the reply. But I got confused by reading the particular documentation link that you have mentioned. By the first
As per the documentation, I meant that particular page (couldn’t add more than two links in the question - so had to remove it, because I already had other links)
Anyway, can you please specifically answer the three questions that I have asked ?
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.