I’d like to start a conversation about developing a Terraform provider to manage sources, destinations, and connections as code.
Our use case:
We would like to be able to manage and update our data sync operations as code. Right now we have some complicated connections where we need to select specific tables out of hundreds on a 10 TB postgresql databsse, we need to purposely avoid sync jobs on some tables, and have many different types of connections depending on the table. Using the UI to configure this leads to a lot of risk. If we accidentally enable one of the giant tables, we could create huge processes that don’t help us.
How we would prefer to work:
For other systems, we use declarative infrastructure as code and are aligned on using Terraform for those. Being able to define connections using a Terraform provider that communicates with our Airbyte installation would allow us to manage our processes as code, providing lots of safety, and add consistency with other usage.
This is the kind of project that we’d be happy to contribute to.
Happy to hear thoughts, ideas, etc.
We just released this week our CLI that will help you perform this kind of operation.
We considered working on a Terraform provider but realized that it will require leveraging the
DynamicPseudoType that was not yet available on the terraform SDK. We need this type to be able to dynamically define the schema of each Airbyte connector configuration that we retrieve for our API, otherwise, we’d have to hardcode all the connector’s configuration schema in the provider which is not something we want to do.
If you have any alternative approach in mind to overcome this problem we’d be really glad to hear it.
+1 to this. We’re thinking of switching over to Airbyte and having a terraform provider would strengthen the case quite a bit.
Thanks for your feedback, we are planning on implementing this feature. We hope that you’ll give us a chance when we release it!
I just wanted to add a strong +1 on this idea.
Having a terraform provider would drastically help us manage connectors at scale in my company, save a lot of time, avoid repetitive manual operations, and ensure we have a similar configuration in different environments.
Thanks by advance to the community for all the work on this topic
Hello Airbyte Team,
my data engineering team and I would be also very interested in having a terraform provider for Airbyte to manage connectors more robustly. Is there any update on this topic or an open issue to track the development?
Moreover, I came across this terraform provider for airbyte, has anyone tested it?