When we start Airbyte, the memory usage of airbyte-worker is around 1.5GB. Memory usage increase as time passes. Currently, it uses around 4GB. Is this expected? How to handle it efficiently?
PS: I know that we can limit the resources of each of the containers but wanted to understand what’s causing the memory increase to go up in the first place.
@marcosmarxm we have the same number of connectors and connections. The memory usage went by 300mb since the time I asked the question. So it’s now almost 4.3GB. Memory usage won’t come down even if there is a long after jobs stopped running.
And airbyte-server memory usage is also slowly increasing but at a slower pace, 5-10MB per day.
The team is investigating a possible issue with memory leaky in the logging system. Until a solution is merged the workaround is to restart airbyte-server and airbyte-worker containers. I created the issue to track the issue: https://github.com/airbytehq/airbyte/issues/16652