Enabling and Disabling Source Fields in Connections

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

This user is looking for a way to easily enable and disable source fields in connections to avoid manual toggling for over 600 fields.


Question

hello, how can I easily enable and disable source fields to be synced in any connection ?
I have more than 600 fields in my source object and it is painful to manually toggle/untoggle


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["enable-disable", "source-fields", "sync", "connection", "toggle"]
2

you might take a look at Airbyte’s terraform provider
https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs

3

Thanks <@U05JENRCF7C> but the finest granularity is stream, not field of stream. Or I am wrong ?

4

Hey <@U05FB419LMR> you can toggle all fields on or off with this switch, does that work for you?

5

hey <@U044KRQLNJZ>, I would like to be able to cherry pick around 150 fields out of 600 (it is not as simple as all or nothing :confused: )
if it was programmatically possible it would have been much easier to set up and manage over time :pray:

6

Sure, that makes sense. I thought maybe you just needed to enable a handful of fields. Unfortunately we don’t have a great way to do that via UI or terraform right now AFAIK

7

would toggling fields using a regex filter have helped? or only programmatic access?

8

we currently have a list of fields that we collect using a custom job, the goal is to use Airbyte to perform the data collect
-> then we have to be able to programmatically identify fields to be synced
NB: we can’t leave all fields enabled otherwise the data collect would last more than 6 hours for 600 fields / 900k HS Contacts :disappointed:
I suppose a such feature in the Terraform provider would be the best solution :fire: :pray:

9

Hi guys, if I may ask a question pls how are you dynamically passing the list of streams (table names) when using Airbyte TF provider? I need to disable views/functions. Are you combining something like Python with terraform?

This seems to be my biggest sticking point in switching to this from doing it via the rest api.

<@U05JENRCF7C>

10

<@U05CXP45USF> in my project I use static list of streams (tables). How do you generate that dynamic list?
For example, you could use extra provider that provides data for use by Airbyte resources
https://registry.terraform.io/providers/hashicorp/local/latest/docs/data-sources/file
https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http
https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/external

11

<@U044KRQLNJZ> do you think an ETA can be shared regarding a such feature ? :pray:

12

<@U05FB419LMR> we don’t have an ETA, but there is an issue you can follow here: https://github.com/airbytehq/airbyte/issues/35043 Column selection is currently not supported through the API, which prevents the terraform provider from letting you select specific fields

13

Hi <@U044KRQLNJZ>, I have the below error message while reaching the issue :confused:

14

Sorry about that, we moved some issues around recently and that ended up in a different repository. This is not on our roadmap for the first half of 2024.

15

how can I follow this topic ? :pray:

16

<@U05FB419LMR> the issue should now be visible again and you can track progress here: https://github.com/airbytehq/airbyte/issues/35043