Summary
The user is trying to inject a cursor value into a nested field in the request JSON for pagination in the Airbyte connector builder. They have attempted conditional Jinja in the request body and add field transformation but encountered errors due to the ‘response’ variable not being defined. They are seeking guidance on how to make the value in the transformation conditional or if there is a better way to inject the value without extending or monkey patching the cursor paginator.
Question
Hi - I"m trying to use the builder to hit the datadog API as a source. For pagination, the cursor value needs to be injected into a nested field in the request json - ie it would be like this
type: DefaultPaginator
page_token_option:
type: RequestOption
inject_into: body_json
field_name: page['cursor']
pagination_strategy:
type: CursorPagination
cursor_value: "{{ response.get(\"meta\", {}).get(\"page\", {}).get(\"after\", {}) }}"
stop_condition: >-
{{ not response.get("meta", {}).get("page", {}).get("after", {})
}} ```
(it needs to go to `page['cursor']`
It seems that the builder doesn't support nested elements in that field. I tried doing a conditional jinja in the request body (ie the request body value would look like this:
```{
"page": {
"limit": 25,
"cursor": "{% set cursor_value = response['meta']['page']['after'] if response is defined else '' %}{{ cursor_value }}"
},
"sort": "-timestamp",
"filter": {
"from": 1725163200000,
"query": "@service:markets-mobile @application.id:2c03180c-6541-4f87-b1c3-8897485ad074 @session.type:user"
}
}```
but I get an error that the response variable isn't defined.
I then tried doing it as an add field transformation (like so:
``` transformations:
- type: AddFields
fields:
- path:
- page
- cursor
value: >-
"{% set cursor_value = response['meta']['page']['after'] if
response is defined else '' %}{{ cursor_value }}"```
This also tells me that the variable `response` doesn't exist. Is there a way to make the value in that transformation conditional? maybe I have the syntax wrong? Is there a better way to inject this value without extending/monkey patching the cursor paginator?
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1729616015079659) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["nested-field-injection", "pagination", "connector-builder", "conditional-value", "response-variable"]
</sub>
actually realizing that the add fields thing is wrong, since it’s injecting it into the response instead of the request (duh)
but the larger question - is there syntax that will work for conditional logic in the request body json?
If you’re specifying the Cursor source field in the Pagination section of the config, you can merge what it contains using {{ next_page_token.next_page_token }}
You would then turn off the option to inject it (since this doesn’t currently support nested JSON), and then manually add it to the JSON Body of the request above.
You may need to combine this with a |default() value, like so:
{{ next_page_token.next_page_token|default('your-value-here') }}
. . . which can also just be |default('')
You can use Jinja conditionals in some contexts (like {% if next_page_token.next_page_token %}do something{% endif %}, but this doesn’t work everywhere
awesome - thanks!
does the next_page_token.next_page_token get set any time the next page cursor is in there? Somehow it exists but is empty: ie if I set this in the request body:
"page": {"cursor": "{{ next_page_token.next_page_token if next_page_token.next_page_token is not none else 'defaultString' }}", "limit": 25} ,
"sort": "-timestamp",
"filter": {
"from": 1725163200000,
"query": "@service:markets-mobile @application.id:2c03180c-6541-4f87-b1c3-8897485ad074 @session.type:user"
}
}```
it never sets it to `defaultString` but never has values, either...
It won’t exist on the first request (since there isn’t yet a response to pull the token from), but it’s odd to me if the default is an empty string vs. a proper None type.
It’s worth noting, as long as you aren’t expecting tokens with falsey values (e.g. 0), you can use the more readable default filter with its optional second value set to true:
{{ next_page_token.next_page_token|default('defaultString',true) }}
This will not only handle None correctly, but also anything that is falsey (e.g. empty string, False, numeric 0/0.0/0j, NaN, the null alias, empty []/()/{}, range(0)). This is very nice in template contexts like these where you aren’t sure exactly what value to expect, but know that it won’t be falsey.
I would also look at the value you’re specifying as the cursor field and compare the response you’re getting back . . . make sure that there isn’t a missing path segment, difference in case of the field path, etc.
thanks for the help - will try that out
it’s strange - if I select “inject” and put it into a request parameter, it’s there - ie this:
type: DefaultPaginator
page_token_option:
type: RequestOption
inject_into: request_parameter
field_name: checkCursor
pagination_strategy:
type: CursorPagination
cursor_value: "{{ response.get(\"meta\", {}).get(\"page\", {}).get(\"after\", {}) }}"
stop_condition: >-
{{ not response.get("meta", {}).get("page", {}).get("after", {})
}}```
gives a 1st request:
```"url": "<https://api.us5.datadoghq.com/api/v2/rum/events/search>" ```
and a 2nd one:
```"url": "<https://api.us5.datadoghq.com/api/v2/rum/events/search?checkCursor=eyJhZnRlciI6IkFnQUFBWks2SFVOMnF5XzVhQUFBQUFBQUFBQVlBQUFBQUVGYVN6WklWVTR5UVVGQmVXNUZSMWxyVTI1a1VtVjVVUUFBQUNRQUFBQUFNREU1TW1KaE1XUXRORE0zTmkwMFltTmxMV0psT0RNdE5UUXpaV1UzTVRGbVpXRTAiLCJ2YWx1ZXMiOlsiMjAyNC0xMC0yM1QxNjowMjoxMi45ODJaIiwtMTQyMjkxOTMyMF19>",```
but deselecting the injection and putting this in the request json:
``` request_body_json:
page:
cursor: "{{ next_page_token.next_page_token|default('xxx',true) }}"
limit: 25
sort: "-timestamp"
filter:
from: 1725163200000
query: >-
@service:markets-mobile
@application.id:2c03180c-6541-4f87-b1c3-8897485ad074
@session.type:user```
always hits the default (I have to do `default('',true)` or it barfs on the value that gets passed - so it just ends up sending
``` "body": {
"page": {
"limit": 25
},
have you ever seen it parse the cursor value correctly but not populate anything in next_page_token.next_page_token?
Another odd thing - I also tried
and
"{{ next_page_token.next_page_token if next_page_token.next_page_token is not none else 'xxx'}}"```
it seems like `next_page_token.next_page_token` is not none, but it's also not defined. (unless maybe I just have the syntax wrong? From google, it seemed like there was something different in jinja vs jinja2
What I’d give for a typeof-style operator in Jinja, lol.
That’s just odd that it’s working one way and not the other.
I can confirm I’m seeing the same thing in Builder right now though . . . it injects in body as a top-level key or in the query string, but the merge is always blank when trying to use it within my custom JSON body
ah cool - well glad I’m not nuts haha - 