Terraform Provider for Airbyte Connections

Good Afternoon,

I’d like to start a conversation about developing a Terraform provider to manage sources, destinations, and connections as code.

Our use case:

We would like to be able to manage and update our data sync operations as code. Right now we have some complicated connections where we need to select specific tables out of hundreds on a 10 TB postgresql databsse, we need to purposely avoid sync jobs on some tables, and have many different types of connections depending on the table. Using the UI to configure this leads to a lot of risk. If we accidentally enable one of the giant tables, we could create huge processes that don’t help us.

How we would prefer to work:

For other systems, we use declarative infrastructure as code and are aligned on using Terraform for those. Being able to define connections using a Terraform provider that communicates with our Airbyte installation would allow us to manage our processes as code, providing lots of safety, and add consistency with other usage.

This is the kind of project that we’d be happy to contribute to.

Happy to hear thoughts, ideas, etc.

Hi @JeremyDOwens,
We just released this week our CLI that will help you perform this kind of operation.

We considered working on a Terraform provider but realized that it will require leveraging the
DynamicPseudoType that was not yet available on the terraform SDK. We need this type to be able to dynamically define the schema of each Airbyte connector configuration that we retrieve for our API, otherwise, we’d have to hardcode all the connector’s configuration schema in the provider which is not something we want to do.

If you have any alternative approach in mind to overcome this problem we’d be really glad to hear it.

1 Like