Summary
Encountering ‘Normalization failed during the dbt run’ error when adding a column ‘tel’ and syncing MySQL to Clickhouse. Error message indicates missing columns in the query.
Question
Hello, I want to synchronize MySQL → Clickhouse
When I sync it for the first time, there are no issues,
but when I add a column “tel” and sync it again, I encounter the error “Normalization failed during the dbt run.”
Source : MySQL
Destination : Clickhouse
Detect and propagate schema changes : Detect changes and manually approve
Sync mode : Incremental | Append + Deduped
2024-01-30 10:33:08 normalization > 10:33:08 Unable to do partial parsing because saved manifest not found. Starting full parse.
2024-01-30 10:33:09 normalization > 10:33:09 [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
2024-01-30 10:33:09 normalization > There are 1 unused configuration paths:
2024-01-30 10:33:09 normalization > - models.airbyte_utils.generated.airbyte_tables
2024-01-30 10:33:09 normalization > 10:33:09 Found 5 models, 0 tests, 0 snapshots, 0 analyses, 753 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
2024-01-30 10:33:09 normalization > 10:33:09
2024-01-30 10:33:09 normalization > 10:33:09 Concurrency: 1 threads (target='prod')
2024-01-30 10:33:09 normalization > 10:33:09
2024-01-30 10:33:09 normalization > 10:33:09 1 of 3 START sql view model _airbyte_sink.mixed_to_row_stg ............................................................. [RUN]
2024-01-30 10:33:09 normalization > 10:33:09 1 of 3 OK created sql view model _airbyte_sink.mixed_to_row_stg ........................................................ [OK in 0.11s]
2024-01-30 10:33:09 normalization > 10:33:09 2 of 3 START sql incremental model sink.mixed_to_row_scd ............................................................... [RUN]
2024-01-30 10:33:09 normalization > 10:33:09 2 of 3 ERROR creating sql incremental model sink.mixed_to_row_scd ...................................................... [ERROR in 0.19s]
2024-01-30 10:33:09 normalization > 10:33:09 3 of 3 SKIP relation sink.mixed_to_row ................................................................................. [SKIP]
2024-01-30 10:33:09 normalization > 10:33:09
2024-01-30 10:33:09 normalization > 10:33:09 Finished running 1 view model, 2 incremental models in 0 hours 0 minutes and 0.49 seconds (0.49s).
2024-01-30 10:33:09 normalization > 10:33:09
2024-01-30 10:33:09 normalization > 10:33:09 Completed with 1 error and 0 warnings:
2024-01-30 10:33:09 normalization > 10:33:09
2024-01-30 10:33:09 normalization > 10:33:09 Database Error in model mixed_to_row_scd (models/generated/airbyte_incremental/scd/sink/mixed_to_row_scd.sql)
2024-01-30 10:33:09 normalization > 10:33:09 :HTTPDriver for <http://localhost:8123> returned response code 404)
2024-01-30 10:33:09 normalization > 10:33:09 Code: 47. DB::Exception: Missing columns: 'inc_data.tel' while processing query: 'SELECT _airbyte_mixed_to_row_hashid, id, seq, inc_data.tel AS tel, name, phone, _ab_cdc_cursor, _ab_cdc_log_pos, _ab_cdc_log_file, _ab_cdc_deleted_at, _ab_cdc
2024-01-30 10:33:09 normalization > 10:33:09
2024-01-30 10:33:09 normalization > 10:33:09 Done. PASS=1 WARN=0 ERROR=1 SKIP=1 TOTAL=3
2024-01-30 10:33:11 platform > Normalization summary: io.airbyte.config.NormalizationSummary@11b8779e[startTime=1706610783786,endTime=1706610791926,failures=[io.airbyte.config.FailureReason@5155e2de[failureOrigin=normalization,failureType=system_error,internalMessage=[0m10:33:09.836037 [error] [MainThread]: :HTTPDriver for <http://localhost:8123> returned response code 404),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@f5142a6[additionalProperties={attemptNumber=4, jobId=86, from_trace_message=true}],stacktrace=AirbyteDbtError:
10:33:09.819988 [error] [Thread-1 (]: 2 of 3 ERROR creating sql incremental model sink.mixed_to_row_scd ...................................................... [ERROR in 0.19s]
10:33:09.835681 [error] [MainThread]: Database Error in model mixed_to_row_scd (models/generated/airbyte_incremental/scd/sink/mixed_to_row_scd.sql)
10:33:09.836037 [error] [MainThread]: :HTTPDriver for <http://localhost:8123> returned response code 404)
10:33:09.836307 [error] [MainThread]: Code: 47. DB::Exception: Missing columns: 'inc_data.tel' while processing query: 'SELECT _airbyte_mixed_to_row_hashid, id, seq, inc_data.tel AS tel, name, phone, _ab_cdc_cursor, _ab_cdc_log_pos, _ab_cdc_log_file, _ab_cdc_deleted_at, _ab_cdc,retryable=<null>,timestamp=1706610791924,additionalProperties={}], io.airbyte.config.FailureReason@72cfb951[failureOrigin=normalization,failureType=system_error,internalMessage=[0m10:33:09.836037 [error] [MainThread]: :HTTPDriver for <http://localhost:8123> returned response code 404),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@20192411[additionalProperties={attemptNumber=4, jobId=86, from_trace_message=true}],stacktrace=AirbyteDbtError:
10:33:09.819988 [error] [Thread-1 (]: 2 of 3 ERROR creating sql incremental model sink.mixed_to_row_scd ...................................................... [ERROR in 0.19s]
10:33:09.835681 [error] [MainThread]: Database Error in model mixed_to_row_scd (models/generated/airbyte_incremental/scd/sink/mixed_to_row_scd.sql)
10:33:09.836037 [error] [MainThread]: :HTTPDriver for <http://localhost:8123> returned response code 404)
10:33:09.836307 [error] [MainThread]: Code: 47. DB::Exception: Missing columns: 'inc_data.tel' while processing query: 'SELECT _airbyte_mixed_to_row_hashid, id, seq, inc_data.tel AS tel, name, phone, _ab_cdc_cursor, _ab_cdc_log_pos, _ab_cdc_log_file, _ab_cdc_deleted_at, _ab_cdc,retryable=<null>,timestamp=1706610791924,additionalProperties={}]],additionalProperties={}]```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1706659637082779) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["mysql", "clickhouse", "sync", "schema-changes", "incremental", "dbt-run", "error", "missing-columns"]
</sub>