- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu 20.04.3 LTS
- Memory / Disk: 64 GB RAM
- Deployment: Docker
- Airbyte Version: 0.35.15-alpha
- Description: Airbyte-server is currently using 48 GB RAM on a 64 GB server. It starts with 2 GB of ram and gradually the usage is increasing. This is causing a serious problems as it is not leaving enough resources for the sync jobs.
Please check the attachments:
Hey may I know how many connections do you have?
We have 300 connections but at most 15 jobs run parallely. When the screen shot was taken only 1 job was running. Hardly 10 jobs ran in 4-5 hrs gap before the screen shot was taken. We were hoping that GC would kickin to clean up and the usage would go down but anything did not happen
Oops can you restart the server and check if it’s still consuming that much of RAM?
Yes we did restart the server and docker as well. Airbyte server takes up approx 1GB of ram in the beginning but as soon as any job starts running it shoots up again. We just now checked for 2 jobs it was using 50gb of ram. After a while the Ram usage is coming down(after docker restart) but isn’t 50gb of ram for two short lived jobs(5-10mins) is insane.
We are not able to handle 4-5 parallel jobs on 64GB - 16core machine.
Got it. Could you create an issue in github so that team can look into this and get back. Also did you try the kubernetes deployment?
We cannot use Kubernetes because we heavily rely on custom DBT scripts. Unfortunately
airbyte kubernetes version does not support it.
Im curious to know if there is any plan to support it in near future.
I will raise an issue and will tag it here. Can you please take with priority because
our production system is down.
I have raised an issue: