Adding additional fields to child records in parent streams

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

The user is inquiring about adding more than just the id for the parent record to the child record in Airbyte’s connector builder UI.


Question

Hi I’m reading through the documentation for parent streams specifically https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning/#adding-the-partition-value-to-the-record. Is it possible to add more than just the id for the parent record to the child record?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["parent-streams", "child-records", "connector-builder", "additional-fields"]
2

New message text here

3

<@U02T7NVJ6A3> I think you might be able to best answer this. This is the same question as https://airbytehq.slack.com/archives/C027KKE4BCZ/p1696254610155529 which never got an answer

4

<@U06863ML00P> this behavior is not directly supported by the Builder.

However, there is a workaround:

  1. You can configure an Add Field transformation on the parent stream, setting its Path to whatever you want to call this new field
  2. Set the Value to something like this: {{ record.id + ", " + record.name }}
  3. This will cause a new field to be added to ever record in the parent stream that contains the record’s id field, followed by a comma and a space, and then the record’s name field
  4. In the child stream, you can then set the Parent Key to this new field you added. Then when you reference that parent key with the Current Parent Key Value Identifier, it will contain the new value with the combined id and name, which you’ll need to parse out to properly use.
    This is definitely an unintuitive workaround, so I will note down this feedback so that we can deliver a more usable solution for this problem
5

<@U06863ML00P> could you share a bit more information about your use-case? What exactly do you mean by “add more” than just the parent record’s ID to the child record? Do you want to use more parent record fields in things like request parameters, or are you adding them to the child record through Add Fields transformations?

6

<@U02T7NVJ6A3> I am trying to export the group membership details for every group in Confluence. I have a parent stream that gets every group and then hit group/{{ stream_partition.group_id }}/membersByGroupId for each group. In the result set I would like to have the group id + group name. I already get the group id using the Add Fields transformation as follows but don’t know how to add the group name.

7

Got it, thank you for sharing your use case - I will raise this feature request with the team.

See my earlier comment for a workaround that might help you here.

8

Btw <@U02T7NVJ6A3> the workaround you suggested doesn’t work in my case. I need the group_id as a standalone field because I also use it in the substream as a request parameter - combining group_id and group_name wouldn’t work for that. Does that make sense?