Granular Configuration of Streams in Airbyte Terraform Provider

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is inquiring about the ability to granularly configure streams in the Airbyte Terraform provider for syncing data from Postgres to BigQuery.


Question

Hi, Im trying to work with airbyte terraform provider to sync fom Postgres to BigQuery. Is there a way to granularly configure streams in the terraform file itself. From what I understand, this is not possible atm. I’d still ilke to confirm here!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["granular-configuration", "streams", "airbyte-terraform-provider", "postgres", "bigquery"]
2

How granularly? Please be precise what do you want to configure.

3

I just want to be able to do what I could do through the UI. Basically select some tables to sync from a schema, configure what type of sync id want and possibly also the columns id like to sync, and which one to use as primary key and cursor?

4

according to https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs/resources/source_postgres|this (schemas can only be array of string?)
but according to https://github.com/airbytehq/airbyte/issues/36756|this (I can see someone defining a whole JSON object for this in their question)
I am wondering where to find the documentation for the latter?

5

I checked configuration and I have something like this for BigQuery -> S3 connection

    streams = [
      {
        name         = "mobile_daily"
        sync_mode    = "incremental_append"
        cursor_field = ["modified_at"]
      },
      {
        name         = "mobile_monthly"
        sync_mode    = "incremental_append"
        cursor_field = ["modified_at"]
      },
    ]
  }```
so you will be able to cover most of your need
6

https://reference.airbyte.com/reference/createconnection
worth checking, because it looks like API supports selecting fields, so maybe terraform provider is able to do it as well

7

selected_fields is also in docs
https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs/resources/connection

8

awesome thanks! this helps
didnt realise this confugurations has to be part of that airbyte_connection resource