Summary
The user is trying to extract a full list of ‘id’ values from a nested JSON object using dpath for the record selector in an Airbyte connector.
Question
According to https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/record-selector?_gl=1*slvvfm*_gcl_au*MTUxNDQ2OTA2Mi4xNzEyMDY1MzQx|this I can use “dpath” for record selector So I wrote some python test test out getting all the “values” I want to pass to a “sub-stream” and it spits out all the id I need in the next stream.
import dpath.util
# Load JSON data from a file
with open('input.json', 'r') as file:
data = json.load(file)
# Traverse the JSON data and find all values associated with 'id'
ids = list(dpath.util.values(data, '**/id'))
# Print the values found for 'id'
for id_value in ids:
print(id_value)```
But when I try and put that in the `**` in the record selector I just get an error. Basically I need to get a "list" of all `id` that exist in a json graph.
How can I get the full list of `id` from a json object (it is nested data) so `blah, *` does not get me there.
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here.
[Join the conversation on Slack](https://slack.airbyte.com)
[Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1715359167903769) if you want to access the original thread.
<sub>
`["dpath", "record-selector", "json-object", "nested-data", "id-values"]`
</sub>