Cursor Pagination for Nested Object Key in API

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Assistance needed with implementing cursor pagination for an API by passing the cursor value to a key nested within another object pagination_parameters


Question

Good morning everyone,
Looking for some assistance figuring out how to do cursor pagination for an API where I need to pass the cursor value to a key that’s nested within another object pagination_parameters


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["cursor-pagination", "nested-object-key", "api", "pagination-parameters"]
2

There aren’t many examples of this online so I’ve tried to go down the route of (1) turning on Cursor Pagination but removing the default injection and (2) trying to pass the response value in the JSON payload directly:

3

However, I keep getting the following error:

raise ValueError(f"Jinja macro has undeclared variables: {undeclared_not_in_context}. Context: {context}") ValueError: Jinja macro has undeclared variables: {'response'}

4

Does my overall strategy make sense? am I not using the response variable correctly?

5

Yes, I have streams that are similar—but note that the response dict is basically just the body (before being filtered with Record Selector), so don’t use ['body'] in the reference there.

If that doesn’t work, maybe you can share a sample/redacted response to try to pin it down more

6

Thanks for reaching out!
Yes, I have something that looks like this now:

7

I am still getting the same issue as before - I believe the issue is that at first the response variable doesn’t exist, and it doesn’t like that I’m adding it inside the JSON Payload

8

When you test the stream, is anything showing in the Response tab?

9

If I use the “inject cursor value” method then it works fine - but I can’t find a way to pass the end_cursor into that nested object from there

10

My guess is that this is an issue of the first request failing because there isn’t a response to pull from yet, or it’s injecting a non-valid value for that field into the request (in which case you should see that in the Request tab)

11

Yes I completely agree!

12

If I change it to be like this, then it works fine - but I just don’t know how to specify to write the cursor value into that nested pagination_parameters object!

13

can you paste the code from your JSON payload into a code block in here (you can make one using the triple-backtick ````` or the Code Block option in the formatting bar)

14 
  "adgroup_id": "{{ stream_partition.adgroup_id}}",
  "Content-Type": "application/json",
  "advertiser_id": "{{ stream_partition.advertiser_id}}",
  "pagination_parameters": {
    "cursor":"",
    "page_size": "1"
  }
}```
15

Try this:

  "adgroup_id": "{{ stream_partition.adgroup_id }}",
  "Content-Type": "application/json",
  "advertiser_id": "{{ stream_partition.advertiser_id }}",
  "pagination_parameters": {
{% if response %}
    "cursor":"{{ response.page_info.end_cursor }}",
{% endif %}
    "page_size": "1"
  }
}```
16

(you may be able to use {% if response.page_info.end_cursor %}, but I don’t remember how jinja handles that)

17

It doesn’t like that jina

18

another option is using something more like this as the value:
{{ response.page_info.end_cursor|default('some sane default that won't break the API') }}

19

(sorry, spending way too much time in different flavors of jinja, hard to keep track of which systems let you do what :joy:)

20

I think that if statement was a good start!