Partitioning by Fields in Custom Connector

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Exploring the possibility of partitioning by specific fields in a custom connector using a parent stream.


Question

Hi – I am trying to use a parent stream to enable partitioning in a custom connector. The documentation states <Partitioning | Airbyte Documentation“Each record of the parent stream is used as a partition for the current stream.”> Is it possible to partition by fields for a <Partitioning | Airbyte Documentation object>? For example, the response from my parent stream is in the format

    "table1": [
        "table1field1",
        "table1field2",
        "table1field3"
    ],
    "table2": [
        "table2field1",
        "table2field2",
        "table2field3",
        "table2field4",
        "table2field5",
        "table2field6"
    ],
    "table3": [
        "table3field1",
        "table3field2",
        "table3field3",
        "table3field4",
        "table3field5"
    ]
}```
and I would like to use `table1, table2, table3` as my partition values, but it appears that the <https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#parent-stream|"Parent Key"> parameter is required (it seems that I would like to partition by the parent key).

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1716404243992649) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["partitioning", "custom-connector", "parent-stream", "partition-by-fields", "connector-builder-ui"]
</sub>
2

With Python CDK and a custom partition router, maybe. With lowcode and builder, sounds like a nope.