After Upgrading Airbyte to v0.34.4-alpha get error

airbyte-server logs throw error :
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.977610470s.
Untitled.txt (21.4 KB)

Hi Ayyoup,
Could you please have a look to this topic and increase your current resources to run Airbyte?
Please share your current memory / CPU and disk sizing too.
You will find some suggestions here on how to size your Airbyte instance.

Hi i run Airbyte in 15.5GB RAM
107GB of disk space

Could you please share your full sync logs and upgrade to a 32GB RAM instance?

already attach my logs on txt above. Then maybe i’ll try to hard reset airbyte first then import configuration

i try run in this resource
airb free -h
total used free shared buff/cache available
Mem: 3,7G 2,4G 145M 53M 1,2G 1,0G
Swap: 3,0G 48M 3,0G

and up, but this is staging server not in production

I think you have a clear lack of memory as it is swapping.
Please give more memory to your instance and jobs. You can check our recommendations here: https://docs.airbyte.com/operator-guides/scaling-airbyte/
You can give more memory to your jobs by changing the value this environment variable: JOB_MAIN_CONTAINER_MEMORY_REQUEST

what is JOB_MAIN_CONTAINER_MEMORY_REQUEST ? any documentation to explain this variables ?

From Configuring Airbyte:

  1. JOB_MAIN_CONTAINER_MEMORY_LIMIT - Define the job container’s maximum RAM usage. Units follow either Docker or Kubernetes, depending on the deployment. Defaults to none.
  2. JOB_MAIN_CONTAINER_MEMORY_REQUEST - Define the job container’s minimum RAM usage. Units follow either Docker or Kubernetes, depending on the deployment. Defaults to none
    From Scaling Airbyte

Our Java connectors currently follow Java’s default behaviour with container memory and will only use up to 1/4 of the host’s allocated memory. e.g. On a Docker agent with 8GBs of RAM configured, a Java connector limits itself to 2Gbs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods. You may want to customize this by setting JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT environment variables to custom values.

Let me know if you need more information.

1 Like