Summary
The user wants to use Terraform to automatically generate a connector for QuickBooks, which is a community-supported source in Airbyte. They have provided a sample source configuration and want guidance on how to write this automatically using Terraform.
Question
Hi there, I had an issue related to a source which is supported by community. I wanted to connect quick books which is available as an community only supported source. Now I want to use terraform to generate that connector. I create the source using the UI of my OSS build and retrieve it using api for demonstration purposes: (Quickbook is only supported in OSS and hence a connector has not been generated )
Do help me understand how I can use terraform to write this automatically…
"sourceId": "2b953b45-341f-445d-8e15-28d115a8500c",
"name": "QuickBooks",
"sourceType": "quickbooks",
"workspaceId": "4c882d54-a6a7-459a-b617-aa6fff9a46f3",
"configuration": {
"sandbox": true,
"start_date": "2024-06-01T00:00:00Z",
"credentials": {
"realm_id": "**********",
"auth_type": "oauth2.0",
"client_id": "ABRSwH6l9FdVhteCHqm5ais6dKsJGa4UKCjIPci6NNc18i9nsO",
"access_token": "**********",
"client_secret": "**********",
"refresh_token": "**********",
"token_expiry_date": "2024-08-30T00:00:00Z"
}
}
}```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1718754995038699) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["quickbooks", "community-supported-source", "terraform", "connector-generation", "source-configuration"]
</sub>
Yes, but the entire validation flow dosnt work with that. As we are automating the process using LLMs. Any way we can try to build our own provider alongside, or a custom validator?
If you want custom resource for QuickBooks, you might consider generating your own version of terraform provider with speakeasy
Yes, but this yaml doesn’t include the spec for quickbooks, any way to generate it automatically?
in my opinion, it would be easier to use terraform validation and create a small module - a wrapper for airbyte_source_custom than creating your own version of terraform provider
keep in mind that any customizations will require future maintenance and updates
Firstly, thanks for your help in such a concise and articulate manner. Since I am new to terraform, I am not sure if I understand correctly what you mean by the difference between wrapper and terraform provider. I will try to explore just that, do point me to any resources if any.
By provider here I refer to https://github.com/airbytehq/terraform-provider-airbyte|terraform-provider-airbyte
By a wrapper here I mean small <https://developer.hashicorp.com/terraform/language/modules|terraform module> that will have mostly airbyte_source_custom resource, and variable will have some validations
If you are new to some technologies, keep it simple as much as possible; you can improve it later; introduce some validations and so on
So I tried building some wrappers or at least getting a sense of this verification before I hit terraform apply. So, honestly I couldn’t figure it out. Many of the validations that I built required to run a python script. and almost in every type of way, I had to hit terraform apply inorder for the verification to take place. Now, what i intend to do differently is being able to verify the script using terraform validate itself. Any idea how to do that?
Everytime based on the syntax the schema gets approved even if its not valid. One of the things I did was to use a null resource and run a python script inside it. <@U05JENRCF7C>
using null resource makes things more complicated
you can use built-in validation for variable
https://developer.hashicorp.com/terraform/language/values/variables#custom-validation-rules
and type constraint type = object(...)
https://developer.hashicorp.com/terraform/language/expressions/type-constraints
I haven’t tested it, but for your case, you could have something like this:
type = object({
sandbox = bool
start_date = string
credentials = object({
realm_id = string
auth_type = string
client_id = string
access_token = string
client_secret = string
refresh_token = string
token_expiry_date = string
})
})
description = "Configuration for the application including credentials and settings."
validation {
condition = contains(["oauth2.0"], var.configuration.credentials.auth_type)
error_message = "auth_type must be 'oauth2.0'."
}
}```
you are right, but I am not looking for such a hardcoded validation, I want it to be dynamic based on the spec.json file.
Hi <@U078H8TAU6S>! Slightly unrelated question - are you using Kubernetes do deploy Airbyte? How were you able to expose the API?
I’d like to use Terraform as well but I’m having troubles with setting up the api.
Really appreciate your help on this!
Hey there, we are not using K8s, just plain and simple container deployment for now.
For a single docker container if you are not able to access the API let me know.
Any idea how I can do that? <@U05JENRCF7C>
Thanks for the response anyways! Did you have to expose it to other port or were you able to access it on /api endpoint?