Summary
The user is facing a memory leak issue after upgrading the Postgres and BigQuery connectors in Airbyte on a GCP VM. They had to downgrade to resolve the high RAM utilization problem. They are looking for a way to identify stable and reliable versions of the connectors.
Question
Hi, we have open source airbyte hosted on gcp vm with 64gb ram.
We recently upgraded airbyte to version to 0.50.39 on open source, which came with updated connectors as well. I’m primarily using postgres connector for source and bigquery connector for destination. I upgraded connectors:
• Postgres connector from 0.4.16 -> 3.3.0
• BigQuery connector from 1.1.6 -> 2.3.25
Docker containers started consuming too much RAM and RAM utilisation peaked resulting in VM unresponsiveness. Eventually I downgraded connectors to old version which I was using before. This resolved the issue and now I have the upgraded airbyte with postgres connector on version 0.4.16 and bigquery connector on version 1.1.6.
I am guessing there is memory leak in latest version of these connectors but I would like to upgrade connectors as the ones I have currently are really old. Is there any way of finding out which versions of the connectors are stable or reliable.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.