How to import/export Airbyte to a new instance (docker to docker deploy)

The Pull Request remove import / export handler by cgardens · Pull Request #16403 · airbytehq/airbyte · GitHub removed the import/export handler. This was a security measure Airbyte took to prevent users to leak their credentials.
In this tutorial I’ll show to you how to move (let’s say…) a dev version to another instance.

Some considerations:

  • This tutorials use the default airbyte-db credentials, pay attention is you have changed them;
  • This tutorial works for a docker to docker migration, I didn’t test or validate on kubernetes or docker to external-db (I’ll try in the future);
  • Try to use the same Airbyte platform version in both instances.

Generate the backup data

run docker ps and you must see Airbyte running. You must be able to access your instance locally or using a ssh tunnel.

IMAGE                                       PORTS
airbyte/proxy:0.40.26             >8000-8001/tcp, 80/tcp,>8003/tcp
airbyte/worker:0.40.26            >9000/tcp
airbyte/server:0.40.26                      8000/tcp,>8001/tcp
airbyte/webapp:0.40.26            >80/tcp
airbyte/temporal:0.40.26                    6933-6935/tcp, 6939/tcp, 7233-7235/tcp, 7239/tcp
airbyte/db:0.40.26                          5432/tcp

Let’s stop all services and only start the database to prevent any new data/update in it.

  1. docker-compose down to stop all services
  2. docker-compose up -d db to only start the database

you can check with docker ps and should see only the database container running.

Let’s create the backup database file.
docker exec airbyte-db pg_dump -U docker airbyte > airbyte_backup.sql

It will generate a local SQL file. You can check the file size and compress if it is too big.

docker-compose down to stop the database again.

Recreate our instance

Copy/transfer the SQL file to your new server.

Go to your new server and start Airbyte using docker-compose up -d after the service is running let’s stop to rebuild the database with docker-compose down.

Start only the database docker-compose up -d db
Run the following commands:

  1. docker exec airbyte-db psql -U docker -c 'drop database airbyte'; to drop the new database
  2. docker exec airbyte-db psql -U docker -c 'create database airbyte with owner docker;'
  3. cat airbyte.sql| docker exec -i airbyte-db psql -U docker -d airbyte to regenerate your database.
  4. docker-compose up -d to restart Airbyte with your backup.
  5. (need improvement) You must trigger all connections again to create the temporal workflow for them again. You can use the Airbyte API to do it.

Some useful links:

It works perfectly on GCE!

1 Like

I am having trouble in the end when trying to do docker-compose up -d. I am facing airbyte-bootloader errors. Anything you can help me with ?

@7vikpeculiar can you describe what steps did you follow and version are you using?

My airbyte version was 0.35.12-alpha and I am trying to upgrade it to the latest version. I followed all the above steps, and the db migration was successful.

Then at Step 4, of recreation

docker-compose up -d to restart Airbyte with your backup.`

The boot-loader is having trouble starting up.

@marcosmarxm any idea how to solve this ?

would you expect this to work with older versions too? I tried with doing this from version 0.30.18 to the latest but docker-compose up fails with 255 error

@schlattk I think this method isn’t supposed to be used for migrating to different version of airbyte. The schema structures seem to be different, which is why I was having trouble migrating from 0.35.12 to latest.

@7vikpeculiar yes came to the same conclusion - thanks

1 Like

Sorry to not answer before @schlattk the method is to transfer an instance to another with the SAME version.