Summary
The user is facing issues migrating their Airbyte instance to an external database after updating the data from a backup. The Airbyte instance running version 0.50.33 is not functioning properly with the external database that already has populated data.
Question
I’m migrating my local airbyte-db to an external one, but to avoid losing data I dumped the postgres database and updated the data of my new external database with this backup. My old airbyte instance was running version 0.50.33. The problem is the instance can’t work properly when I start it with an external db with alredy populated data.
Can someone please help me here, we need to migrate to an external db asap but I can’t make it work! 
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["migrate", "airbyte-db", "external-database", "postgres", "data-backup", "version-0.50.33"]
Did you move just the DB, or are you swapping to external and upgrading Airbyte at the same time?
If it’s the latter, I’d recommend splitting these into two operations to minimize issues
Well, the problem was because the service running in cloud (using kubernetes) is in version 0.50.33 and mine (locally) was a newer version. But I already downgraded to the same version (0.50.33) and it keeps me throwing errors like:
2024-07-26 12:37:42 2024-07-26 15:37:42 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Error instantiating bean of type [io.airbyte.api.client.AirbyteApiClient]
When I connect to a clean, brand new db, one with no data, it works perfectly. But then I pg_restored into this db and it’s not working anymore. My goal is just to avoid losing our customers data
Are you restoring the db before the first time Airbyte connects to it?
No, I first connected to it with a clean db, as its shown in the documentation. Then I restored the db and it stopped working
Also some containers keep restarting for no reason aparently
I would also pay for someone to help me with this migration, do you guys provide this service?
did you back up both the Temporal DB and the Airbyte DB?
These are the tables in the database I restored
so the DB named temporal on the same host is used for scheduling and state management, so you’ll want to export that one as well.
that may be the issue you’re running into
I’m not sure if Airbyte will recreate that state or not
And how do I configure this temporal db in my new instance? It doesn’t work even with a completely clean db anymore.
I followed the instructions in the documentation but it only shows how to point to the default db and config db.
it creates it on the same host on the first run (if it doesn’t already exist). it also handles upgrading it between versions and such
So from here I should backup airbyte (already did) and temporal right?
And how do I configure my new instance to use it? Or it does automatically by just updating those db env variables?
# Airbyte Internal Job Database, see <https://docs.airbyte.io/operator-guides/configuring-airbyte-db>
DATABASE_USER=mathbarbosa
DATABASE_PASSWORD=****
DATABASE_HOST=host.docker.internal
DATABASE_PORT=5432
DATABASE_DB=airbyte
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB} (do not include the username or password here)
DATABASE_URL=jdbc:<postgresql://host.docker.internal:5432/airbyte>
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.40.26.001```
correct, you can back both up and then restore them to the new external
The existing DB vars you have will connect to it, and it’ll create the Temporal database with the appropriate permissions the first time it connects to the Airbyte DB. you’ll see this in your logs as it checks the version of the schema, creates them if needed, and upgrades them if needed as well.
I’ve always brought over temporal_visibility as well, although I don’t know that it’s actually needed.