- Is this your first time deploying Airbyte?: Yes
- OS Version / Instance: Ubuntu (Linux airbyte 5.4.0-1086-azure #91~18.04.1-Ubuntu SMP Thu Jun 23 20:33:05 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux)
- Memory / Disk: you can use something like 32GB/100GB
- Deployment: Docker
- Airbyte Version: 0.39.28-alpha
- Source name/version: BigQuery 0.1.8
- Destination name/version: Snowflake 0.4.30
- Step: normalization
I have a BigQuery source that has a few tables with columns defined as arrays. For the columns defined as ARRAY, the data makes it to the raw table just fine, but the datatype for the array column is FLOAT.
The sync ends up failing during normalization with errors like:
2022-07-19 22:52:44 normalization > 22:52:39.115256 [error] [MainThread]: Database Error in model VSCHEDULEOPENSHIFT (models/generated/airbyte_tables/DIMENSIONS/VSCHEDULEOPENSHIFT.sql) 2022-07-19 22:52:44 normalization > 22:52:39.115606 [error] [MainThread]: 100038 (22018): Numeric value '' is not recognized
Based on the normalization docs, I would have expected a second table to be created, but I don’t see that happening.
Right now, my only workaround to not have a failure is to have two separately configured connections between BigQuery and Snowflake - one to handle the set of tables that don’t have any array columns with normalization, and one to handle the others only as raw data, and then create views in Snowflake to handle the normalization for end users.
Am I missing something?