Orchestrator container gets same resource limit/request values as read/write containers

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: CentOS Linux 7 (Core)
  • Memory / Disk: 16Gb / 500GB
  • Deployment: Kubernetes
  • Airbyte Version: 40.13
  • Source name/version: -
  • Destination name/version: -
  • Step: sync

I have noticed that when I set the JOB_MAIN_CONTAINER_MEMORY_LIMIT that it also applies a memory limit to the Orchestrator container/pod for the sync jobs.

  • Is this supposed to happen?
  • Is this behaviour different in newer versions of Airbyte (>40.13)?
  • Does the orchestrator actually need roughly the same amount of memory as the read / write pods?
  • Is it possible to specify memory limits/requests for read / write pods without specifying them for the orchestrator (or vice versa)?

Furthermore the ‘check’ pods do not seem to be affected by the JOB_MAIN_CONTAINER_MEMORY_LIMIT setting.

  • Does this apply to ‘spec’ and ‘discover’ pods as well?
  • Is it possible to have more granular control over the resources of the pods for these specific types of jobs?
2

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

3

Hey @BravoDeltaBD, I’m waiting on more info on this from the team, hope to get back to you shortly!

4

@natalyjazzviolin Thanks for your reply. Has the team been able to take a look at this yet?

In the meantime I have found answers to some of my own questions by more carefully reading the docs: Configuring Connector Resources | Airbyte Documentation.

Is it possible to specify memory limits/requests for read pods or write pods individually (without affecting the other or orchestrator pods)?

Yes, this is possible. You can configure connector specific resource requirements for any source or destination connector, which only influences instances of that specific connector (instances being specific sources and destinations).

Unlike ‘connection specific resource requirements’, this resource setting only applies to pods/containers of that specific connector, instead of overwriting the orchestrator pod and both of the connector pods for that connection.

Be careful to note that this affects all connections using a source or destination instance of that connector. At our company we now create multiple connectors of the same image and version but with different names and resource requirements. For example, we now have a CustomCsvSourceConnector-1GiB and a CustomCsvSourceConnector-5GiB, each with their own respective resource requirements. (We use custom connectors, but I assume the same principle can be applied to predefined connectors as well)

Is it possible to have granular control over the resource requirements along the axes of Connector-type and the sync/spec/check/discover job-types?

Yes (though I have not tested this yet). As can be seen in the documentation on connector-specific resource requirements, a different ‘jobType’ can be specified in the resource_requirements string, so it seems that it should be possible to assign a custom set of resource requirements per job-type per connector.

My (new/reframed) questions:

  • Are the orchestrator pods supposed to get their resource requirements from the JOB_MAIN_CONTAINER_* environment variables? If so, why?

  • Has this behaviour changed in versions >40.13 (version of our current Airbyte deployment)?

  • Is there any documented specification of how the value for the resource_requirements field can be validly configured for the actor_definition table and the connection table?

1 Like
5

Thank you for the in-depth update, Boris!

To answer your questions:

  1. Yes, the orchestrator pods get their resource requirements from the JOB_MAIN_CONTAINER_* environment variables. These environment variables are used to configure the resource requirements for the job containers, which include the orchestrator pods. They help manage the minimum and maximum CPU and memory usage for the job containers, ensuring that the resources are allocated efficiently and within the desired limits.Here are the relevant environment variables:1. JOB_MAIN_CONTAINER_CPU_REQUEST - Define the job container’s minimum CPU usage.

  2. JOB_MAIN_CONTAINER_CPU_LIMIT - Define the job container’s maximum CPU usage.

  3. JOB_MAIN_CONTAINER_MEMORY_REQUEST - Define the job container’s minimum RAM usage.

  4. JOB_MAIN_CONTAINER_MEMORY_LIMIT - Define the job container’s maximum RAM usage.These variables are applicable to both Docker and Kubernetes deployments, and their units follow either Docker or Kubernetes, depending on the deployment (source).Setting these environment variables helps ensure that the orchestrator pods and other job containers have the appropriate resources allocated to them, preventing issues such as out-of-memory exceptions or inefficient resource usage.

  5. I don’t believe there have been changes since v0.40.13, that being said we always encourage users to upgrade to the latest version.

  6. Yes, there are documented specifications for the resource_requirements field for both the actor_definition and connection tables.For the actor_definition table, the resource_requirements field is a JSON blob with the schema defined in ActorDefinitionResourceRequirements.yaml. This field sets a default resource requirement for any connector of this type, overriding the default set for all connector definitions. It can be further overridden by a connection-specific resource requirement (source).For the connection table, the resource_requirements field is a JSON blob with the schema defined in ResourceRequirements.yaml. This field sets a default resource requirement for the connection, overriding the default set for all connector definitions and the default set for the connector definitions (source).

Let me know if this helps!