Issue setting up Airbyte cloud with Terraform for connection and selecting fields

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

When setting up a connection and selecting fields in Airbyte cloud using Terraform, encountering a HTTP/2.0 400 Bad Request response. Looking for assistance in resolving this issue.


Question

Hi All,
I am setting up Airbyte cloud with Terraform and just run into an issue when trying to setup a connection and selecting fields. Not sure what I am doing wrong. Has anyone run into the a similar issue?

  data_residency                       =
  destination_id                       = 
  name                                 = "Zendesk"
  non_breaking_schema_updates_behavior = "propagate_columns"
  prefix                               = "zendesk_"
  source_id                            = 
  for_each =  { for stream in local.zendesk_streams.streams : stream.name => stream }
  configurations = {
    streams = [{
      name = each.value.name
      cursor_field = each.value.cursor_field
      selected_fields = [
        {
          field_path = each.value.selected_fields
        }
      ]
      sync_mode = each.value.sync_mode
    }]
  }
}```
The template renders correctly, but I get a `HTTP/2.0 400 Bad Request` response.

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1730300649403449) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["airbyte-cloud", "terraform", "connection", "selecting-fields", "http-400-bad-request"]
</sub>
2

What happen if you remove the for_each loop? Can you try to create one connector without it?

3

I figured out the issue. In the selected fields, I cannot pass more than one field in the list. It has to be one attribute per field.
I also removed the for_each.

4

<@U01MMSDJGC9> now it complains about the cursor_field not being included in the selected_fields. But it I am including it, I the terraform plan output, I can see it is included. Is this a problems someone else has had?

5

<@U07TYLTKZ70> can you paste current terraform code?

6 
  data_residency                       = "eu"
  destination_id                       = 
  name                                 = "Zendesk"
  non_breaking_schema_updates_behavior = "propagate_columns"
  prefix                               = "zendesk_"
  source_id                            = airbyte_source_zendesk_support.zendesk_support.source_id
  configurations = {
    streams = [
      {
        name = local.zendesk.tickets.name
        cursor_field = local.zendesk.tickets.cursor_field
        selected_fields = [
          for field in local.zendesk.tickets.selected_fields : {
            field_path=[field]
          }
        ]
        sync_mode = local.zendesk.tickets.sync_mode
      }
    ]
  }
  schedule = {
    cron_expression = "30 0 * * * ?"
    schedule_type = "cron"
  }
}```
<@U05JENRCF7C>
7

and what is the value of local.zendesk.tickets?

8 
    "tickets": {
        "name": "tickets",
        "cursor_field":["id"],
        "selected_fields": [
            "allow_attachments",
            "allow_channelback",
             "id"
        ],
        "sync_mode":"incremental_append"
    }
}```
this is the locak.zendesk json <@U05JENRCF7C>
9

when terreform planning it renders correctly

10

this might the case of source-defined cursor and you need to skip setting cursor_field or provide the same value as source-defined cursor for that stream
https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#source-defined-cursor

you can find that this is happening in the code
https://github.com/airbytehq/airbyte-platform/blob/595a4c945b8ccaebbe08896affbe264678431386/airbyte-server/src/main/kotlin/io/airbyte/server/apis/publicapi/helpers/AirbyteCatalogHelper.kt#L211|AirbyteCatalogHelper.kt#L211
https://github.com/airbytehq/airbyte-platform/blob/595a4c945b8ccaebbe08896affbe264678431386/airbyte-server/src/main/kotlin/io/airbyte/server/apis/publicapi/helpers/AirbyteCatalogHelper.kt#L388-L389|AirbyteCatalogHelper.kt#L388-L389

https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs/resources/connection#cursor_field
> This field is REQUIRED if sync_mode is incremental unless there is a default.

11

<@U05JENRCF7C> indeed it was. Thank you so much!