DBT Normalization can not connect

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Debian 11
  • Memory / Disk: you can use something like 2Gb / 50 Gb
  • Deployment: Are you using Docker or Kubernetes deployment? Docker
  • Airbyte Version: What version are you using now? * 0.39.25-alpha
  • Source name/version: Oracle DB 0.3.17
  • Destination name/version: Postgres 0.3.20
  • Step: The issue is happening during sync, creating the connection or a new source? DBT
  • Description: DBT can not connect during normalization. If I remove the normalization option the JSON data is correctly saved on the destination.

I understand that could be an issue with the ssh connection but not sure how to solve it. I haven’t a limit of connection on the destination DB Server.

2022-06-25 15:20:12 e[42mnormalizatione[0m >   Generating airbyte_ctes/kmk/vouchers_preventadetalle_ab3.sql from VOUCHERS_PREVENTADETALLE
2022-06-25 15:20:12 e[42mnormalizatione[0m >   Generating airbyte_tables/kmk/vouchers_preventadetalle.sql from VOUCHERS_PREVENTADETALLE
2022-06-25 15:20:13 e[42mnormalizatione[0m > Detected tunnel method SSH_PASSWORD_AUTH for normalization
2022-06-25 15:20:13 e[42mnormalizatione[0m > Running: sshpass -e ssh -f -N -M -o StrictHostKeyChecking=no -S {control socket} -l anibal -L 50001:localhost:5432 chi.com
2022-06-25 15:20:13 e[42mnormalizatione[0m > Warning: Permanently added 'chi.com,11.11.57.51' (ECDSA) to the list of known hosts.
2022-06-25 15:20:13 e[42mnormalizatione[0m > ssh tunnel opened
2022-06-25 15:20:20 e[42mnormalizatione[0m >            [--event-buffer-size EVENT_BUFFER_SIZE]
2022-06-25 15:20:20 e[42mnormalizatione[0m >   --event-buffer-size EVENT_BUFFER_SIZE
2022-06-25 15:20:20 e[42mnormalizatione[0m > 
2022-06-25 15:20:20 e[42mnormalizatione[0m > DBT >=1.0.0 detected; using 10K event buffer size
2022-06-25 15:20:20 e[42mnormalizatione[0m > 
2022-06-25 15:20:26 e[42mnormalizatione[0m > 15:20:26  Running with dbt=1.0.0
2022-06-25 15:20:26 e[42mnormalizatione[0m > 15:20:26  Partial parse save file not found. Starting full parse.
2022-06-25 15:20:33 e[42mnormalizatione[0m > 15:20:33  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
2022-06-25 15:20:33 e[42mnormalizatione[0m > There are 2 unused configuration paths:
2022-06-25 15:20:33 e[42mnormalizatione[0m > - models.airbyte_utils.generated.airbyte_views
2022-06-25 15:20:33 e[42mnormalizatione[0m > - models.airbyte_utils.generated.airbyte_incremental
2022-06-25 15:20:33 e[42mnormalizatione[0m > 
2022-06-25 15:20:33 e[42mnormalizatione[0m > 15:20:33  Found 136 models, 0 tests, 0 snapshots, 0 analyses, 526 macros, 0 operations, 0 seed files, 34 sources, 0 exposures, 0 metrics
2022-06-25 15:20:33 e[42mnormalizatione[0m > 15:20:33  
2022-06-25 15:20:33 e[42mnormalizatione[0m > 15:20:33  Encountered an error:
2022-06-25 15:20:33 e[42mnormalizatione[0m > Database Error
2022-06-25 15:20:33 e[42mnormalizatione[0m >   connection to server at "localhost" (::1), port 50001 failed: Connection refused
2022-06-25 15:20:33 e[42mnormalizatione[0m >   	Is the server running on that host and accepting TCP/IP connections?
2022-06-25 15:20:33 e[42mnormalizatione[0m >   connection to server at "localhost" (127.0.0.1), port 50001 failed: Connection refused
2022-06-25 15:20:33 e[42mnormalizatione[0m >   	Is the server running on that host and accepting TCP/IP connections?
2022-06-25 15:20:33 e[42mnormalizatione[0m >   
2022-06-25 15:20:33 e[42mnormalizatione[0m > 
2022-06-25 15:20:33 e[42mnormalizatione[0m > Diagnosing dbt debug to check if destination is available for dbt and well configured (1):
2022-06-25 15:20:33 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m > 15:20:39  Running with dbt=1.0.0
2022-06-25 15:20:39 e[42mnormalizatione[0m > dbt version: 1.0.0
2022-06-25 15:20:39 e[42mnormalizatione[0m > python version: 3.9.9
2022-06-25 15:20:39 e[42mnormalizatione[0m > python path: /usr/local/bin/python
2022-06-25 15:20:39 e[42mnormalizatione[0m > os info: Linux-5.10.0-13-amd64-x86_64-with-glibc2.31
2022-06-25 15:20:39 e[42mnormalizatione[0m > Using profiles.yml file at /data/8/1/normalize/profiles.yml
2022-06-25 15:20:39 e[42mnormalizatione[0m > Using dbt_project.yml file at /data/8/1/normalize/dbt_project.yml
2022-06-25 15:20:39 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m > Configuration:
2022-06-25 15:20:39 e[42mnormalizatione[0m >   profiles.yml file [OK found and valid]
2022-06-25 15:20:39 e[42mnormalizatione[0m >   dbt_project.yml file [OK found and valid]
2022-06-25 15:20:39 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m > Required dependencies:
2022-06-25 15:20:39 e[42mnormalizatione[0m >  - git [OK found]
2022-06-25 15:20:39 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m > Connection:
2022-06-25 15:20:39 e[42mnormalizatione[0m >   host: localhost
2022-06-25 15:20:39 e[42mnormalizatione[0m >   port: 50001
2022-06-25 15:20:39 e[42mnormalizatione[0m >   user: anibal
2022-06-25 15:20:39 e[42mnormalizatione[0m >   database: kmk_mig
2022-06-25 15:20:39 e[42mnormalizatione[0m >   schema: kmk
2022-06-25 15:20:39 e[42mnormalizatione[0m >   search_path: None
2022-06-25 15:20:39 e[42mnormalizatione[0m >   keepalives_idle: 0
2022-06-25 15:20:39 e[42mnormalizatione[0m >   sslmode: None
2022-06-25 15:20:39 e[42mnormalizatione[0m >   Connection test: [ERROR]
2022-06-25 15:20:39 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m > 1 check failed:
2022-06-25 15:20:39 e[42mnormalizatione[0m > dbt was unable to connect to the specified database.
2022-06-25 15:20:39 e[42mnormalizatione[0m > The database returned the following error:
2022-06-25 15:20:39 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m >   >Database Error
2022-06-25 15:20:39 e[42mnormalizatione[0m >   connection to server at "localhost" (::1), port 50001 failed: Connection refused
2022-06-25 15:20:39 e[42mnormalizatione[0m >   	Is the server running on that host and accepting TCP/IP connections?
2022-06-25 15:20:39 e[42mnormalizatione[0m >   connection to server at "localhost" (127.0.0.1), port 50001 failed: Connection refused
2022-06-25 15:20:39 e[42mnormalizatione[0m >   	Is the server running on that host and accepting TCP/IP connections?
2022-06-25 15:20:39 e[42mnormalizatione[0m >   
2022-06-25 15:20:39 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m > Check your database credentials and try again. For more information, visit:
2022-06-25 15:20:39 e[42mnormalizatione[0m > https://docs.getdbt.com/docs/configure-your-profile
2022-06-25 15:20:39 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m > 
2022-06-25 15:20:39 e[42mnormalizatione[0m > Control socket connect(/tmp/sshsocket-20680): Connection refused
2022-06-25 15:20:39 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):161 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:63) ~[io.airbyte-airbyte-workers-0.39.25-alpha.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:21) ~[io.airbyte-airbyte-workers-0.39.25-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.25-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:60) ~[io.airbyte-airbyte-workers-0.39.25-alpha.jar:?]
	... 3 more
	Suppressed: io.airbyte.workers.exception.WorkerException: Normalization process wasn't successful
2

Can you check this issue? https://github.com/airbytehq/airbyte/issues/13758 Maybe is the cause of your problem using a custom port from 22

3

Thanks for your answer @marcosmarxm.
No, that isn’t my case because the destination server is using the 22 port. A probe of that is the destination connection is working is the airbyte_ tables were created using the same connection. The issue is happening when DBT tries to connect after that.

4

Strange because normalization+ssh for Postgres was implemented here: https://github.com/airbytehq/airbyte/issues/5294
I was able to create (using integration account) Hubspot to Postgres through a SSH Tunnel

Are you trying localhost as the host for the database?

5

Not sure I’m following you. The DB is in a remote host connected by ssh
But now you mention it I could try adding the destination DB on the same host where Airbyte is running.
Either way, considering the error message it looks like some issue with multiple connections. I already checked out my server can receive multiple connections from the Airbyte host.
Not sure how to troubleshoot this issue.

6

As I thought, I’ve moved the DB to the same server running Airbyte and the normalization worked well.
I’ll probably keep this configuration.

7

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.