RAM usage keeps increasing for every sync

I am monitoring CPU and Memory usage for airbyte syncs from postgres to bigquery using grafana. I have deployed airbyte in GCP VM using docker. I am observing that for every subsequent sync Memory usage keeps on increasing. Even idle memory usage also keeps on increasing after every sync. If this continues we might get OOM errors. You can see the screenshot below

Hi @SP271,
Could you please drill a bit into the memory usage and check which container gets its memory increased sync after sync?
Are you running a single sync? If I read the chart correctly it looks like you are running an every two-hour sync which is taking ~1H to run, am I right?

@alafanechere I am running the sync for every 3 hours which takes approx 1-1.5 hours to complete. I monitored the ram usages of different containers. What I found was the ram is increasing slightly for every container except airbyte-webapp. Also it increased a lot for airbyte-worker. You can compare it. I am attaching the screenshots after 8-10 syncs.

I think it could be interesting to let Airbyte run an extended period and check if any garbage collection process cleans up the idle RAM. I’m going to ping our infra team to understand what could explain this memory increase.

@SP271 it’s also worth mentioning that the worker is constantly performing some other tasks that are not only sync jobs. I’d encourage you to try defining resource constraints on the worker container in the docker-compose.yml file

@alafanechere If I will define resource constraints on worker and scheduler can they create any problem because of these constraints?

You might get Java OutOfMemory errors if the constraint are too strict, but you will avoid a global memory error and will only have to restart the worker or the scheduler