Summary
When syncing data from S3 to BigQuery using Airbyte Cloud, encountering a warning about a duplicate column name ‘_airbyte_meta’ in the CREATE TABLE query. The issue seems to be related to having duplicate columns with different data types.
Question
Hi, I’m using Airbyte Cloud to sync data from S3 to BigQuery and have encountered the following error:
Warning from destination: com.google.cloud.bigquery.BigQueryException: Duplicate column name _airbyte_meta in CREATE TABLE at [7:1]
I checked the BigQuery history and found that the query has duplicate columns with different data types. Has anyone else encountered this problem too?
CREATE TABLE `playground-project`.`poc_0919`.`mongodb_from_airbyte_test` ( _airbyte_raw_id STRING NOT NULL, _airbyte_extracted_at TIMESTAMP NOT NULL, _airbyte_meta JSON NOT NULL, _airbyte_generation_id INTEGER, `_airbyte_data` JSON, `_airbyte_meta` JSON, `_airbyte_raw_id` STRING, `_ab_source_file_url` STRING, `_airbyte_extracted_at` INT64, `_airbyte_generation_id` INT64, `_ab_source_file_last_modified` STRING ) PARTITION BY (DATE_TRUNC(_airbyte_extracted_at, DAY)) CLUSTER BY `_airbyte_extracted_at`;
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.