Summary
When setting up a new connection in Airbyte after switching from Fivetran, the sync process is removing columns created by Fivetran and changing some column names. The user is looking for the best way to maintain the original column names and data types from Fivetran.
Question
Hello! I am currently switching from fivetran to Airbyte. I am first copying all the tables from fivetran to keep the original data types and columns. However when I set up a new connection in Airbyte the sync completely gets rid of all the columns created by fivetran and and also changes the names of some columns as well. What is the best way to set up a new connection in Airbyte while keeping the column names and data types the same as they were originally created by fivetran?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["preserving-column-names", "data-types", "new-connection", "fivetran", "airbyte"]
We moved from Fivetran to Airbyte and we found that you cannot achieve this. It is easier and safer to start over.
I supposed you could in theory put “fake” metadata in your destination so it looks like it was managed by airbyte all along and setup a state table pointing to the latest records. How badly do you want it?
That might work but that’s really hacky and I feel like it could lead to undesired results down the line
(Do so at your own risk lol)
Thank you so much for both your input! We need the data very badly since so many of our dashboards and downstream applications depend on the column names and data types that were implemented by fivetran
Are you talking about the fivetran metadata columns or columns from your source schema?
Both fivetran metadata columns and source columns
Yea that’s what I was thinking…Using transformations/views to convert the airbyte data to look like it was in fivetran…Sounds like it might be computationally expensive though
It doesn’t make much sense for the most part…But I am trying to avoid updating hundreds of queries that have the fivetran metadata columns/ data types/ names. The queries will fail if the columns don’t exists anymore of if the column names have changed or if column data types have changed
use views as an interim step to avoid disruption, but better to bite the bullet and move your models to use airbyte
But if Airbyte and Fivetran have metadata columns that map 1 to 1 then you can use a view that abstracts them to a single column (like airbyte_deleted and fivetran_deleted become fivetran_deleted in the view)
That’s what I would do in the short term to avoid disruption
don’t have to do a transformation, just create the views in your destination directly
Yea that’s what I was thinking, but explained it poorly before
We are in a similar situation and thinking about views longterm
Syncing MySQL and PostgreSQL to Snowflake for Sigma and DBT and John Snow Labs
Yea I knew this was going to be a massive undertaking…I am glad there is a community of others going through the same experience
I really appreciate the input from both of you 