Partitioning by Fields in Custom Connector Parent Stream

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Exploring the possibility of partitioning by specific fields in a custom connector’s parent stream and the requirement of the Parent Key parameter.


Question

Hi – I am trying to use a parent stream to enable partitioning in a custom connector. The documentation states <Partitioning | Airbyte Documentation“Each record of the parent stream is used as a partition for the current stream.”> Is it possible to partition by fields for a <Partitioning | Airbyte Documentation object>? For example, the response from my parent stream is in the format

    "table1": [
        "table1field1",
        "table1field2",
        "table1field3"
    ],
    "table2": [
        "table2field1",
        "table2field2",
        "table2field3",
        "table2field4",
        "table2field5",
        "table2field6"
    ],
    "table3": [
        "table3field1",
        "table3field2",
        "table3field3",
        "table3field4",
        "table3field5"
    ]
}```
and I would like to use `table1, table2, table3` as my partition values, but it appears that the <https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#parent-stream|"Parent Key"> parameter is required (it seems that I would like to partition by the parent key).

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1716402057038199) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["partitioning", "custom-connector", "parent-stream", "fields", "partition-values", "parent-key"]
</sub>