Migrating data from Docker Compose to Kubernetes instance in Airbyte

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Exploring the most up-to-date approach for migrating data from a Docker Compose deployment to a Kubernetes instance in Airbyte.


Question

Hi airbyters, since docker compose deployment is deprecated I deployed a new airbyte instance to kubernetes, but i’m having some trouble understanding how to migrate the data, what is the most up to date approach?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["migrating-data", "docker-compose", "kubernetes", "data-migration", "airbyte-platform"]
2

I’d recommend configuring external database

  1. prepare fresh new PostgreSQL database
  2. dump database from docker compose deployment
  3. import dump to the new PostgreSQL database
  4. configure new deployment to use external database https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#external-airbyte-database
3

of course, you can use database (statefulset) that exists in helm charts; whatever suits you

I find external database easier to manage and that gives more flexibility in replacing deployments

4

thank you <@U05JENRCF7C> :raised_hands: will do that. Also on my previous deployment *.json files were created for the connections sources and destinations eg. source_config.json , destination_config.json . Don’t these need to be migrated too somehow?

they are stored under the airbyte_workspace docker volume

5

I don’t think so. As far as I know configurations for source/destination connectors are stored in database, in table: actor, and configurations for connections are stored in database, in table: connection.

6

than you so much <@U05JENRCF7C>!