Best way to migrate connectors across airbyte instances?

Q&A
,
#1

hi all, what’s the best practice for migrating connectors across airbyte instances? we currently have two instances: staging and prod. we have a bunch of connectors running on staging and just set up our prod instance, but we don’t know how to migrate. some options i can think of:

  1. somehow promoting our staging instance to prod — maybe we can copy all the data from our staging env to our prod env? is there a backup/restore option?
  2. creating new airbyte connectors in prod that point to the same places as staging connectors, and then switching over. what would happen in this case?
  3. setting up brand new connectors with new destinations in prod — rerunning historical syncs or performing manual backfills

would be great to hear some POVs on these options

#2

Hey you can do these

  1. You can dump data from the database and reload them back in the prod database
  2. If the destination is same you can create new connections in the prod instance and set the state in the database
  3. This is something as you said you can create all new and pull from all history
#3

When you say “dump from the database” do you mean the airbyte “internal” database?
I would guess it’s something you could automate more easily with an external db right? (would be tricky to do from an ec2 docker compose installation since the db image would move around I would guess)?

So for example you could have a dev external db and a prod external db. Then your CICD pipeline involves triggering a sync of these - dumping from dev to prod.
Could you do this via airbyte itself - having airbyte sync the two? That sounds like it might be circular in logic and prone to problems though.

#4

Hey yeah you can directly take dump from postgres is what I meant

1 Like