Summary
The user is trying to configure memory limits for the normalization job in Airbyte Helm chart using values.yaml file.
Question
Hello Team,
Currently I’m trying to run a sync job for 1million Csv to postgres through airbyte . When i try to run this sync job with docker compose with following values in .env JOB_MAIN_CONTAINER_MEMORY_REQUEST=90Gi
JOB_MAIN_CONTAINER_MEMORY_LIMIT=100Gi
NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT=100Gi
NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST=90Gi im able to complete the sync job. Is there any way to configure NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT=100Gi
NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST=90Gi this values through values.yaml in airbyte helm chart
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.