Handling pagination in Connector Builder with API response

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

Exploring how to handle pagination in Connector Builder when the API response includes a ‘paging’ object with ‘next_token’ for the next request.


Question

Hi all, I’m trying to use connector builder with pagination. My API returns something like:

  "paging": {
    "execution_id": "xxx",
    "next_token": "xx",
    "row_count": 100
  },
  "data": [ ]
}```
The next request needs to be something like `request_url&execution_id=xxx&next_token=xxx`
Any idea how I can handle this ?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1721916068495899) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["connector-builder", "pagination", "api-response", "next-token"]
</sub>
2

Yep, just use Pagination set to Cursor. This use case it outlined in the docs:
https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#cursor-pagination

The second parameter could probably be set in the Query Parameters section using a jinja reference along these lines:
{{ response.paging.execution_id|default('') }}

This would set it on subsequent requests, but it would be blank/empty on the first one. If the API allows that empty value, you should be good. if it throws an error, you may need to get a little more creative. But always try to start as simple as possible and move on from there.

3

Thanks Justin! Yes I actually tried a few different options, including the one you mention, but could not make it work!

4

I’m guessing it does not like the empty execution_id value on the first run

5

And trying this (with default values from a previous run) and pagination disabled does not work at all (even for the first 100 records)