Issues with Accessing Variables from Multiple Parent Streams in Custom HTTP Connector

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is facing issues with accessing variables from multiple parent streams in a custom HTTP connector in Airbyte. They are trying to include both client_uid and property_id in the URL path but are encountering problems with interpolation. They have tried configuring the streams and transformations but are not seeing the desired results.


Question

Hi everyone,
I’m working on setting up a custom stream in Airbyte’s HTTP API connector and I’m running into some issues with accessing variables from multiple parent streams in the URL path.
Here’s what I’m trying to achieve:
• Clients Stream: Provides uid (client unique identifier).
• Properties Stream: Child of Clients, provides id (property identifier) and needs to have access to client_uid.
• Custom Stream: Needs to include both the client_uid and property_id in the URL path, similar to how the Properties stream works.
What I’ve Tried:

  1. Configured the Custom Stream with Two Parent Streams:
    • Parent Streams:
    • Properties
    • Parent Key: id
    :black_small_square:︎ Current Parent Key Value Identifier: property_id
    • Clients
    • Parent Key: uid
    ◦ Current Parent Key Value Identifier: client_uid
    • URL Path:
    • bash
    •
    •
    ◦ /api/v1/custom/{{ stream_partition.client_uid }}/{{ stream_partition.property_id }}
    • Transformations:
    ◦ Added fields for client_uid and property_id using {{ stream_partition.client_uid }} and {{ stream_partition.property_id }} respectively.
  2. Issues Encountered:
    ◦ When making a request, only the property_id is correctly interpolated in the URL; the client_uid is not appearing.
    ◦ If I remove the Clients parent stream, the property_id shows up after the / in the URL, but I still can’t get both variables to appear together.
  3. Additional Steps:
    ◦ I added a transformation in the Properties stream to include client_uid in each record, but I’m still not seeing client_uid in the Properties API response or in the Custom stream.
    Current Configuration of the Custom Stream:
    • URL Path:
    • bash
    •
    •
    • /api/v1/custom/{{ stream_partition.client_uid }}/{{ stream_partition.property_id }}
    • Parent Streams:
    • Properties
    :black_small_square:︎ Parent Key: id
    ◦ Current Parent Key Value Identifier: property_id
    • Clients
    :black_small_square:︎ Parent Key: uid
    ◦ Current Parent Key Value Identifier: client_uid
    • Transformations:
    • Add Field:
    :black_small_square:︎ Path: client_uid
    ◦ Value: {{ stream_partition.client_uid }}
    • Add Field:
    :black_small_square:︎ Path: property_id
    ◦ Value: {{ stream_partition.property_id }}
    Questions:
    • Has anyone successfully accessed variables from multiple parent streams in a custom HTTP connector?
    • Is there a better way to structure the streams or transformations to include both client_uid and property_id in the URL path?
    • Do I need to adjust how I’m including client_uid in the Properties stream so that it can be accessed in the Custom stream?
    Any insights or suggestions would be greatly appreciated!

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["custom-stream", "http-api-connector", "parent-streams", "url-path", "interpolation", "transformations"]
2

Check SubstreamPartitionRouter . parent_streams_configs is an array.

https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/partition-router#substreampartitionrouter