I am monitoring CPU and Memory usage for airbyte syncs from postgres to bigquery using grafana. I have deployed airbyte in GCP VM using docker. I am observing that for every subsequent sync Memory usage keeps on increasing. Even idle memory usage also keeps on increasing after every sync. If this continues we might get OOM errors. You can see the screenshot below
Hi @SP271,
Could you please drill a bit into the memory usage and check which container gets its memory increased sync after sync?
Are you running a single sync? If I read the chart correctly it looks like you are running an every two-hour sync which is taking ~1H to run, am I right?
@alafanechere I am running the sync for every 3 hours which takes approx 1-1.5 hours to complete. I monitored the ram usages of different containers. What I found was the ram is increasing slightly for every container except airbyte-webapp. Also it increased a lot for airbyte-worker. You can compare it. I am attaching the screenshots after 8-10 syncs.
I think it could be interesting to let Airbyte run an extended period and check if any garbage collection process cleans up the idle RAM. I’m going to ping our infra team to understand what could explain this memory increase.
@SP271 it’s also worth mentioning that the worker is constantly performing some other tasks that are not only sync jobs. I’d encourage you to try defining resource constraints on the worker container in the docker-compose.yml file
@alafanechere If I will define resource constraints on worker and scheduler can they create any problem because of these constraints?
You might get Java OutOfMemory errors if the constraint are too strict, but you will avoid a global memory error and will only have to restart the worker or the scheduler
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.