Unable to connect to external database

Platform, Deploy & Infra Issues
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 4Gb / 25 Gb
  • Deployment: Docker
  • Airbyte Version: 0.35.65-alpha
  • Source name/version: airbyte/source-mysql:0.5.6
  • Destination name/version: airbyte/destination-bigquery:0.3.12
  • Step: normal sync
  • ingestion type : CDC - Incremental Append
  • Description:

My final objective is to separate my Airbyte database on a GCP SQL instance.

As a first step i’m trying to generate a local postgres DB and connect my Airbyte instance to it.
To help me i followed the documentation.

Followin documentation, it seems that my Airbyte instance is not able to connect to database.

Capture d’écran 2022-04-21 à 11.11.30
Capture d’écran 2022-04-21 à 11.11.301268×586 134 KB 


2022-04-21 09:13:44 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):75 - Database is not ready yet. Please wait a moment, it might still be initializing...

2022-04-21 09:13:49 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...

My .env file is like that :

### DATABASE ###
# Airbyte Internal Job Database, see https://docs.airbyte.io/operator-guides/configuring-airbyte-db
DATABASE_USER=postgres
DATABASE_PASSWORD=password
DATABASE_HOST=host.docker.internal # refers to localhost of host
DATABASE_PORT=3000
DATABASE_DB=postgres
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT/${DATABASE_DB} (do not include the username or password here)
DATABASE_URL=jdbc:postgresql://host.docker.internal:3000/postgres?ssl=true&sslmode=require
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001

I although tried to implement another docker postgres database, with the same issue.

Those are my current containers :

Capture d’écran 2022-04-21 à 11.11.03
Capture d’écran 2022-04-21 à 11.11.03406×680 87 KB

I didn’t found on slack ou on this site any solution.
Can you help me ?

2

Hello @ftapin,
There is a similar issue I tried to troubleshoot that might be interesting if you try to connect to a Cloud SQL and on which I suggest some debugging steps.

As Airbyte containers are started in host network mode so you should be able to use localhost instead of host.docker.internal.

Did you activate SSL on your docker postgres DB? I see this postgres?ssl=true&sslmode=require in your DATABASE_URL, your airbyte-db container might have SSL turned off.

4

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.