Airbyte Internal Database connections are not closed

Platform, Deploy & Infra Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu 22.04
  • Memory / Disk: 32GB / 200GB
  • Deployment: Docker
  • Airbyte Version: 0.35.14

We have hosted our airbyte databse on RDS postgres. Connections to the database are not being closed and we are reaching max connections. This happened earlier so we changed max_connections settings from default value of Postgres to 500. How can we solve this problem? We suspect this is due to connections are not being closed properly from the server. We have to restart the database to solve the issue.

Logs:
“2022-07-20 06:07:52 \u001b[32mINFO\u001b[m i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):74 - Database is not ready yet. Please wait a moment, it might still be initializing…”,

      "2022-07-20 06:07:57 \u001b[33mWARN\u001b[m i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):65 - Waiting for database to become available...",

      "2022-07-20 06:07:57 \u001b[32mINFO\u001b[m i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...",

Screenshots from RDS:

Screenshot from 2022-07-20 10-26-13
Screenshot from 2022-07-20 10-26-13905×430 23.8 KB

Screenshot from 2022-07-20 10-26-41
Screenshot from 2022-07-20 10-26-41944×443 37 KB

Screenshot from 2022-07-20 10-27-03
Screenshot from 2022-07-20 10-27-03971×430 39.8 KB

Please we are facing this issue repeatedly and currently our production system is down

Thank you

2

Hello @prudhvi85, can you see in the logs if it’s the source or destination connectors that are not exiting correctly? Could you please update to the latest version of Airbyte to see if that might fix the issue?

3

@natalyjazzviolin My understanding is that the source or destination workers doesn’t connect to Airbyte’s internal database directly as the server will fetch all the details required and spin up these workers. Please correct me if I am wrong.

In case the workers are connecting directly, does it mean that the job which fails will not close the database connection?

4

I need some more details to help you debug this: were you able to update Airbyte to the latest version? What connectors and versions of connectors are you using?

5

@natalyjazzviolin This issue is resolved after updating to airbyte. Thank you for your time!!

6

So glad to hear that! Closing the thread.