Normalization fails after upgrading Airbyte

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: EC2 t2.xlarge
  • Memory / Disk: 16Gb
  • Deployment: Docker
  • Airbyte Version: 0.40.0-alpha
  • Source name/version: Postgres 1.0.3
  • Destination name/version: BigQuery 1.1.14
  • Step: Normalization

I was originally using the following versions:

  • Airbyte Version: 0.39.4
  • Source name/version: Postgres 0.4.31
  • Destination name/version: BigQuery 1.1.11

I then upgraded connectors only, so I was in following versions:

  • Airbyte Version: 0.39.4
  • Source name/version: Postgres 1.0.3
  • Destination name/version: BigQuery 1.1.14

My Postgres → BigQuery connection was running smoothly at this point

Then I upgraded Airbyte itself to version 0.40.0 and my connection stopped working at the normalization step with the following kind of errors:

2022-08-22 15:33:43 e[42mnormalizatione[0m > e[33mDatabase Error in model buyback_refund_scd (models/generated/airbyte_incremental/scd/prod/buyback_refund_scd.sql)e[0m
2022-08-22 15:33:43 e[42mnormalizatione[0m >   Column 4 in UNION ALL has incompatible types: DATETIME, TIMESTAMP at [91:5]

I tried to Reset the connection thinking that it might be due to an inconsistency due to the upgrade but it was not sufficient, I kept getting the same error.

So it seems to be linked to the new Airbyte version.

Full logs attached (I truncated the middle part because of the size limit, but I don’t think there was anything of interest there)

logs-776 (2).txt (721.8 KB)

2

Hey is it the same even if you create a new connection?

3

Hi,

Got the same issue after upgrading all Airbyte + destination + postgresql source to the latest version.

How can we handle it w/out loosing any historical data ?

4

Hey right now it’s hard to identify a common way to handle this usecase. We generally tend to handle this case by case. So yeah if you have anything more specific we can help there.