Exploring best practices for backing up Airbyte sources, destinations, and connections and storing them in source control.
What is the best practice for getting your airbyte source/destination/connections backed up or in some sort of source control? I asked AI and searched the docs but had no luck.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
I personally also commit my Builder/Low-code YAML files to source control for easier history/change management. But that’s all obviously outside the normal Airbyte workflow (it does store the various versions and drafts in the database as well, specifically in the declarative_manifest table, so they’re safe in your database backups and you’re able to get them if needed, but have to convert the relevant YAML from how they store it in a JSON blob)
Thanks for the reply!
Further expanding on this topic which I’m also highly invested in, you can actually configure any source and destination using the Terraform provider, although any custom one needs to be referenced by its definition ID, which is accessible via the configuration API (https://github.com/airbytehq/airbyte/issues/32744)
I’ve been using it for built in sources and destinations, but in theory it’s possible for custom connectors too.
<@U066U84SC0P> Thanks for clarifying. The only challenge here is that when importing (e.g. into a new instance), the IDs change so it gets a little messy if you’re trying to move things using TF right now. Feels like there’s more development needed in that space. But there are definitely some options there, just not quite end-to-end yet.
(I’m hoping to see it get to a point where you could export and import your full config as part of your TF setup to be run alongside your infrastructure TF to facilitate migrations and rebuilds that are fully TF-managed.)