Bigquery Cannot replace a table with a different partitioning spec

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: GKE 1.22.8-gke.202
  • Memory / Disk: 30GB
  • Deployment: Kubernetes
  • Airbyte Version: What version are you using now?
  • Source name/version: Postgres 1.0.2
  • Destination name/version: Bigquery 1.1.14
  • Step: Issue happening on subsequent syncs
  • Description: Getting the following error on subsequent syncs

Cannot replace a table with a different partitioning spec. Instead, DROP the table, and then recreate it. New partitioning spec is interval(type:day,field:_airbyte_emitted_at) clustering(_airbyte_unique_key,_airbyte_emitted_at) and existing spec is interval(type:day,field:_airbyte_emitted_at) clustering(_airbyte_emitted_at)

Checking the table in Bigquery yields
Screen Shot 2022-08-13 at 10.58.01 PM

2

Can you give more context about what you’re trying to achieve or your use case @kempspo ?

3

Same issue here. My source is MySQL. Every time I add a table to a BigQuery connection I need to perform a stream reset, which is fine, the ‘reset streams’ warning reminds of that. But I get this error after the the initial copy, so I wait 24 hours for the initial read for CDC, only for it to fail at the end because of a dbt error. Then I have to drop all tables and restart. Very frustrating.

Should a stream reset not also alter the partitioning spec?