Normalization process wasn't keeps failing

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Centos
  • Memory / Disk: you can use something like 4Gb / 1 Tb
  • Deployment: Docker deployment?
  • Airbyte Version: 0.38.4-alpha
  • Source name/version: MySQL
  • Destination name/version: PostgreSQL
  • Step: The issue is happening during sync, creating the connection or a new source?
  • Description: Normalization process wasn’t successful.

When normalization is selected the following error occurs. If Raw Data (JSON) is selected the sync completes successfully.

	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:103) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:229) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	... 13 more
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:61) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:19) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	... 1 more
Caused by: io.airbyte.workers.WorkerException: Normalization Failed.
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:58) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:19) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
	... 1 more
	Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful
		at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:162) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:46) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
		at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:19) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.38.4-alpha.jar:?]
		at java.lang.Thread.run(Thread.java:833) [?:?]
2

Waiting complete logs be posted.

4

Looks dbt is not able to resolve the host name you had inserted:

2022-05-25 04:29:13 e[42mnormalizatione[0m > host: 10.144.33.57:5432,10.144.34.57:5432/panda?target_session_attrs=read-write2022-05-25 04:29:13 e[42mnormalizatione[0m > port: 54322022-05-25 04:29:13 e[42mnormalizatione[0m > user: service_panda2022-05-25 04:29:13 e[42mnormalizatione[0m > database: panda2022-05-25 04:29:13 e[42mnormalizatione[0m > schema: data_import2022-05-25 04:29:13 e[42mnormalizatione[0m > search_path: None2022-05-25 04:29:13 e[42mnormalizatione[0m > keepalives_idle: 02022-05-25 04:29:13 e[42mnormalizatione[0m > sslmode: None2022-05-25 04:29:13 e[42mnormalizatione[0m > Connection test: [ERROR]2022-05-25 04:29:13 e[42mnormalizatione[0m > 2022-05-25 04:29:13 e[42mnormalizatione[0m > 1 check failed:2022-05-25 04:29:13 e[42mnormalizatione[0m > dbt was unable to connect to the specified database.2022-05-25 04:29:13 e[42mnormalizatione[0m > The database returned the following error:2022-05-25 04:29:13 e[42mnormalizatione[0m > 2022-05-25 04:29:13 e[42mnormalizatione[0m > >Database Error2022-05-25 04:29:13 e[42mnormalizatione[0m > could not translate host name "10.144.33.57:5432" to address: Name or service not known2022-05-25 04:29:13 e[42mnormalizatione[0m > could not translate host name "10.144.34.57:5432/panda?target_session_attrs=read-write" to address: Name or service not known2022-05-25 04:29:13 e[42mnormalizatione[0m > 2022-05-25 04:29:13 e[42mnormalizatione[0m >

Could you target only one endpoint of your cluster?

5

I will make that change and try again.

It is strange though, as i can see the namespace and an empty table created on the destination.

6

The normalization has now successfully completed.

Is there a cleanup that is supposed to run as part of the normalization to remove the temp table ?

7

No, there is an issue to clean tmp tables: https://github.com/airbytehq/airbyte/issues/7011 but today is not possible.
You can create some routines to clean them for you.

8

Thanks. The issue i am seeing is the _tmp tables are being cleaned up, but the _aribyte_raw tables remain.

9

Airbyte raw contains the complete data. _tmp tables for unsuccessful syncs aren’t deleted because of this the issue there.

10

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.