Gradual increase in memory usage

Platform, Deploy & Infra Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu - 20.04
  • Memory / Disk: 64GB/ 200GB
  • Deployment: Docker
  • Airbyte Version: 39.28

When we start Airbyte, the memory usage of airbyte-worker is around 1.5GB. Memory usage increase as time passes. Currently, it uses around 4GB. Is this expected? How to handle it efficiently?
PS: I know that we can limit the resources of each of the containers but wanted to understand what’s causing the memory increase to go up in the first place.

2

Did you create more connectors & connections or keep the same as the begining?

3

@marcosmarxm we have the same number of connectors and connections. The memory usage went by 300mb since the time I asked the question. So it’s now almost 4.3GB. Memory usage won’t come down even if there is a long after jobs stopped running.

And airbyte-server memory usage is also slowly increasing but at a slower pace, 5-10MB per day.

4

Do you mind sharing the output of docker stats ?

5

image
image1261×77 12 KB

Here

6

Sorry the delay here Prudhvi, did the memory increase again or become steady now?

7

It is increasing, it crossed 9GB as of now. Current screenshot

image
image1244×73 11.9 KB

8

Seems potentially related to Resource leak in airbyte-worker

9

Pudhvi and Adam I request the engineer team to check for what is the expected resources needed and if maybe a bug exists.

10

@marcosmarxm just wanted to check if there are any updates on this.

And here is current resource usage. It keeps on increasing:

image
image951×78 9.74 KB

11

The team is investigating a possible issue with memory leaky in the logging system. Until a solution is merged the workaround is to restart airbyte-server and airbyte-worker containers. I created the issue to track the issue: https://github.com/airbytehq/airbyte/issues/16652

1 Like
12

Hello Prudhvi Raj, it’s been a while without an update from us. Are you still having problems or did you find a solution?