Normalization fails from MySQL, MS SQL Server to Clickhouse during sync

Connector Questions & Issues
,
1

0f6e13d0_a726_4b19_b1bc_6dfb1b912a0d_logs_273_txt.txt (248.2 KB)

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 32Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: 0.40.14
  • Source name/version: Microsoft SQL Server (MSSQL) (0.4.22), MySQL (1.0.3)
  • Destination name/version: Clickhouse ALPHA (0.2.0)
  • Step: The issue is happening during sync. Data normalization fails
  • Description: Airbyte is able to populate Raw Data(JSON) into Clickhouse however if normalized tabular data option is selected in transformation tab the sync fails. Log is uploaded…
2

This might help.

2022-12-01 00:35:04 normalization > dbt was unable to connect to the specified database.

2022-12-01 00:35:04 normalization > The database returned the following error:

2022-12-01 00:35:04 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -

2022-12-01 00:35:04 normalization > >Database Error

2022-12-01 00:35:04 normalization > Code: 102. Unexpected packet from server 20.40.54.26:18123 (expected Hello or Exception, got Unknown packet)

2022-12-01 00:35:04 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -

2022-12-01 00:35:04 normalization > Check your database credentials and try again. For more information, visit:

2022-12-01 00:35:04 normalization > Connection profiles | dbt Developer Hub

2022-12-01 00:35:04 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -

2022-12-01 00:35:04 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -

2022-12-01 00:35:04 INFO i.a.w.g.DefaultNormalizationWorker(run):82 - Normalization executed in 12 seconds.

2022-12-01 00:35:04 ERROR i.a.w.g.DefaultNormalizationWorker(run):90 - Normalization Failed.

2022-12-01 00:35:04 INFO i.a.w.g.DefaultNormalizationWorker(run):95 - Normalization summary: io.airbyte.config.NormalizationSummary@350137ce[startTime=1669854892160,endTime=1669854904696,failures=[io.airbyte.config.FailureReason@184be30d[failureOrigin=normalization,failureType=system_error,internalMessage=Code: 102. Unexpected packet from server 20.40.54.26:18123 (expected Hello or Exception, got Unknown packet),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@64e0ec44[additionalProperties={attemptNumber=2, jobId=273, from_trace_message=true}],stacktrace=AirbyteDbtError:

Encountered an error:

Database Error

Code: 102. Unexpected packet from server 20.40.54.26:18123 (expected Hello or Exception, got Unknown packet),retryable=,timestamp=1669854904695], io.airbyte.config.FailureReason@78588d7b[failureOrigin=normalization,failureType=system_error,internalMessage=Code: 102. Unexpected packet from server 20.40.54.26:18123 (expected Hello or Exception, got Unknown packet),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@44db7af9[additionalProperties={attemptNumber=2, jobId=273, from_trace_message=true}],stacktrace=AirbyteDbtError:

Encountered an error:

Database Error

Code: 102. Unexpected packet from server 20.40.54.26:18123 (expected Hello or Exception, got Unknown packet),retryable=,timestamp=1669854904695]]]

2022-12-01 00:35:04 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling…

3

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

4

Hello! Thank you so much for your patience - we’re a small team and catching up with some backlog messages. Let me look into this for you and I hope to have some ideas for you Monday!

5

Hello team. Kindly look into my matter

6

Hi @ryujjin, my sincerest apologies for making you wait so long. I’ve been looking into this further and noticed that this error has come up before:
https://discuss.airbyte.io/t/moving-data-from-bigquary-to-clickhouse/1089

I checked and the Clickhouse normalization image is current, so we can eliminate that possibility. I do see that the Clickhouse destination version that is next updates dbt, so I wonder if that might fix your issue! Could you please update and let me know?

Thank you!