Hi all !
-
Is this your first time deploying Airbyte?: No
-
OS Version / Instance:GCP compute engine
-
Memory / Disk: 4vcpu / 16Gb Mem / 200Gb disk
-
Deployment: Docker
-
Airbyte Version: 0.40.4
-
Source name/version: postgresql 1.0.7
-
Destination name/version: BigQuery 1.1.16
-
Step
- I had a postgresql <> BQ connection wokring w/ CDC pgoutput with 3 tables
- A source table schema changed in pgsql (new col added) table bar
- So I refreshed the source in the Airbyte connection, it well discover the new col.
- I decided to make a reset as recommended
- it fails on reset
- Description:
the reset fails on the normalization step w/ the following err :
2022-09-07 11:36:29 - Additional Failure Information: Cannot replace a table with a different partitioning spec. Instead, DROP the table, and then recreate it. New partitioning spec is interval(type:day,field:_airbyte_emitted_at) clustering(_airbyte_emitted_at) and existing spec is interval(type:day,field:_airbyte_emitted_at) clustering(_airbyte_unique_key,_airbyte_emitted_at)
What is strange is that the errors occurs on the tables which did not changes
2022-09-07 11:36:29 normalization > Found 12 models, 0 tests, 0 snapshots, 0 analyses, 602 macros, 0 operations, 0 seed files, 3 sources, 0 exposures, 0 metrics
2022-09-07 11:36:29 normalization > Concurrency: 8 threads (target='prod')
2022-09-07 11:36:29 normalization > 1 of 3 START table model ab_poc_cdc5.bar................................................................................ [RUN]
2022-09-07 11:36:29 normalization > 2 of 3 START incremental model ab_poc_cdc5.far.......................................................................... [RUN]
2022-09-07 11:36:29 normalization > 3 of 3 START incremental model ab_poc_cdc5.gar.......................................................................... [RUN]
2022-09-07 11:36:29 normalization > 1 of 3 OK created table model ab_poc_cdc5.bar........................................................................... [CREATE TABLE (0.0 rows, 0 processed) in 2.25s]
2022-09-07 11:36:29 normalization > 2 of 3 ERROR creating incremental model ab_poc_cdc5.far................................................................. [ERROR in 6.37s]
2022-09-07 11:36:29 normalization > 3 of 3 ERROR creating incremental model ab_poc_cdc5.gar................................................................. [ERROR in 7.02s]
2022-09-07 11:36:29 normalization > Finished running 1 table model, 2 incremental models in 7.98s.
2022-09-07 11:36:29 normalization > Completed with 2 errors and 0 warnings:
2022-09-07 11:36:29 normalization > Database Error in model far (models/generated/airbyte_incremental/ab_poc_cdc5/far.sql)
2022-09-07 11:36:29 normalization > Cannot replace a table with a different partitioning spec. Instead, DROP the table, and then recreate it. New partitioning spec is interval(type:day,field:_airbyte_emitted_at) clustering(_airbyte_emitted_at) and existing spec is interval(type:day,field:_airbyte_emitted_at) clustering(_airbyte_unique_key,_airbyte_emitted_at)
2022-09-07 11:36:29 normalization > Database Error in model gar (models/generated/airbyte_incremental/ab_poc_cdc5/gar.sql)
2022-09-07 11:36:29 normalization > Cannot replace a table with a different partitioning spec. Instead, DROP the table, and then recreate it. New partitioning spec is interval(type:day,field:_airbyte_emitted_at) clustering(_airbyte_emitted_at) and existing spec is interval(type:day,field:_airbyte_emitted_at) clustering(_airbyte_unique_key,_airbyte_emitted_at)
2022-09-07 11:36:29 normalization > Done. PASS=1 WARN=0 ERROR=2 SKIP=0 TOTAL=3
how can I perform a source schema change w/ postgres CDC data collection connection ?
Thanx