Unsigned Bigint column getting converted to integer when syncing MySQL to GCP BigQuery

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Unsigned bigint column from MySQL is being converted to integer instead of numeric when syncing to GCP BigQuery in Airbyte.


Question

Hello, I am facing strange behavior when syncing MySQL to GCP BigQuery, our table has an “unsigned bigint” column which is getting converted into integer when reaching BQ. The correct translation would be numeric so im not sure why is it casting it into integer. We are trying to migrate into Airbyte and this is kinda of holding us back since we have multiple services that connect to BQ and expect the numeric datatype.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["mysql-connector", "gcp-bigquery", "airbyte-platform", "unsigned-bigint", "numeric-datatype"]
2

I guess you’ll have to do custom transformation using dbt project
Current afaik there is no way to change the casting which airbyte does itself if custom transformation is disabled.

3

We are running on k8s, I think dbt transform is not supported over this platform :'c

4

Dbt does run on k8s. I haven’t tried it though.
My poc works on docker and the dbt core and dbt big query is being used in a container.

Also airbyte internally uses dbt for its basic transformation so if airbyte is available in k8s so dont think running a custom dbt on k8s will be any issue.

5

https://github.com/airbytehq/airbyte/issues/5091|https://github.com/airbytehq/airbyte/issues/5091

This issues goes over why it doesn’t work. :disappointed: