Setting Resources for Discovery Jobs

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Configuring resources for discovery jobs in Airbyte platform and troubleshooting when the configuration does not work.


Question

hi

how I can set the resources for discovery jobs? I configured jobs resources (this configuration set this env JOB_MAIN_CONTAINER_CPU_REQUEST), but it doesnt work. I checked that the env var is set on all airbyte pods.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["resources", "discovery-jobs", "configuration", "airbyte-platform", "env-var"]
2

setting via environment variable was only relevant to the Docker Compose deployment method, which is now deprecated as of 1.0. If you’re using abctl or helm-based deployments, you would set this in your values.yaml using something like:

  jobs:
    resources:
      requests:
        cpu: 1000m
        memory: 1Gi
      limits:
        cpu: 2000m
        memory: 3Gi```
. . . which will apply to all job types. If you want to apply to connector-specific job types, you can look at <https://docs.airbyte.com/operator-guides/configuring-connector-resources#configuring-connector-specific-requirements|this section of the docs> on how to set those values within the internal Airbyte database. Just note that it pre-dates the changes to configuration, so doesn't reflect that the environment vars are set by the config, not the other way around.

One caveat though: there was a <https://github.com/airbytehq/airbyte/issues/38178|GitHub issue> related these connector/connection-specific configs not getting applied as of ~0.63, so I'm not sure if that has been resolved as of 1.0. If it works when you try it, please do make sure to update that ticket and let everyone know!
3

yes! the config that I changed is global.jobs.resources.requests (you can’t see global because is above), but nothing :confused: i didn’t tried connector-specific config, but i will try

4

I finally set it up like this and it worked :slightly_smiling_face:

  env_vars:
    DISCOVER_JOB_MAIN_CONTAINER_CPU_REQUEST: 4000m
    DISCOVER_JOB_MAIN_CONTAINER_MEMORY_REQUEST: 8Gi```
5

<@U07RLEXQF3Q> Thank you so much for sharing this! It really helps while I am now migrating to abctl :smile:

• Would you be able to share what is the difference between 4000m and 4Gi?
• Also what was previous JOB_MAIN_CONTAINER_CPU_REQUEST is now prefixed with DISCOVER ?

6

hi <@U0606G3NKU4>!
• That’s is CPU value and is measured in just numbers or in “millis”, so 4000m is equal to 4
• That I saw in docs JOB_MAIN_CONTAINER_CPU_REQUEST its a global variable for any kind of job (but discover jobs are not taking that env - maybe its a bug?), and DISCOVER_ are specific for discover jobs and that envs works fine.

7

<@U07RLEXQF3Q> i had previously configured the job_main_container CPU & memory request and limit parameters in the .env file… But in the new abctl setup I’m facing difficulty finding any docs showing how to setup those values in the values.yaml that is equivalent to the all jobs setting