Summary
Guidance on migrating data and handling schema differences between Airbyte versions
Question
Hi Team,
I have been trying to migrate from Airbyte 0.50.6 to Airbyte 1.1.0 (OSS) . Any suggestions on how best to do it, and what steps to follow. I have both Airbytes setup at the moment. The following steps are what I have achieved already:
1- Created Sources by fetching source definition from API and converting it to Terraform config and apply
2- Created destinations like above
3- Created connections like above
However, I am seeing few majore differences in actual tables i.e.
1- Airbyte 1 is not creating _scd tables
2- The schema of tables generated by Airbyte 1 is different from Airbyte 0.5.
So how do I go about the migration, as I want to incrementally load the data since the point it was synced by older airbyte, but these schema differences mean I cannot run the sync on existing synced tables. Any suggestions on this will be highly appreciated. Thanks
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.