Dynamically setting source_id and destination_id in Airbyte connections based on dynamically created sources/destinations in Terraform

[slack-user-airbyte]

Summary

The user is looking for a solution to dynamically set the source_id and destination_id in Airbyte connections based on sources/destinations created dynamically in Terraform using a for_each loop. They are seeking help on how to locate these dynamically generated resources.

Question

Hi Guys - after some help with dynamically setting the source_id and destination_id within connections based on the result of other source/destinations created from a for_each loop (ie. and hence don’t have an explicit definition in terraform).

Essentially I have Airbyte which has
• xN sources
• xN destinations
• xN connections (where a connection needs to dynamically link to a source and destination to retrieve the actual UUID)
So in the connection file there are two items like:

  source_id                 = airbyte_source_postgres.test_admit_tf.source_id```
These both work fine if there are sources/destinations with:

```resource "airbyte_destination_snowflake" "test_admit_tf" { ...}
resource "airbyte_source_postgres" "test_admit_tf" { ...}```
But... those don't exist, because I'm creating those with a for_each loop to avoid being repetitive

```locals {
  sources_apply_test = {
    "client1" = {
        stream = "client1"
    }
    "client2" = {
        stream = "client2"
    }
    "client3" = {
        stream = "client3"
    }
    "client4" = {
        stream = "client4"
    }
  }
}

resource "airbyte_source_postgres" "sources_apply_test" {
  for_each = local.sources_apply_test 
    name          = "test_apply_${each.value.stream}_tf"
    workspace_id  = var.airbyte_config_localhost.workspace_id

    configuration = {

      source_type     = "postgres"
      database        = var.source_config_admit_test_au.database
      host            = var.source_config_admit_test_au.host
      username        = var.source_config_admit_test_au.username
      password        = var.source_config_admit_test_au.password             
      port            = var.source_config_admit_test_au.port
      jdbc_url_nonprod_australia_params = ""
      
      replication_method = {
        read_changes_using_write_ahead_log_cdc = {
          publication                           = var.source_config_admit_test_au.publication
          replication_slot                      = var.source_config_admit_test_au.replication_slot
          heartbeat_action_query                = "insert into test_apply.${each.value.stream}.cdc_keepalive values (1,now(),pg_current_wal_lsn()) on conflict (id) do update set ts=excluded.ts,pos=excluded.pos returning id,ts,pos;"
          initial_waiting_seconds               = 1200
          invalid_cdc_cursor_position_behavior  = "Re-sync data"
          lsn_commit_behaviour                  = "While reading Data"
          queue_size                            = 10000
          initial_load_timeout_hours            = 8
          # additional_properties               = ""
          # plugin                              = ""
        }
      }
      schemas = [
        "${each.value.stream}"
      ]
      ssl_mode = {require = {}}
      tunnel_method = {no_tunnel = {}}
  }
}

output "sources_apply_test_ids" {

  # Sources must be up and running before can output
  depends_on = [airbyte_source_postgres.sources_apply_test]

  value = {
    for k, v in airbyte_source_postgres.sources_apply_test : k => v.definition_id
  }
}```
How can I locate those dynamically generated resources - I know this has to be a problem the community has solved as otherwise we would need to either use static IDs (which is problematic) or not use the for_each on sources/destinations.

I've tried output but seem to be lost - hoping some can provide an example

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1727663297549099) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["airbyte", "connections", "source-id", "destination-id", "terraform", "for-each", "dynamic-resources", "output"]
</sub>