Pagination with Page Increment in Body Object

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

User inquires about configuring pagination in a connector by setting the page number within an object in the request body instead of at the root level.


Question

Hello :slightly_smiling_face:
When doing Pagination with Page Increment , is there a way to set the page number in the body but in an object instead of root of body ?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['pagination', 'page-increment', 'request-body', 'connector']
2

Hey ! Sure :slightly_smiling_face:
The api I’m targeting use a pagination object
Just in case, <MODJO | Api V1 is the api endpoint documentation>

in the body with page and perPage values

{
  "pagination": {
    "page": 1,
    "perPage": 2
  }
}```
It return a `pagination`  object in the response with `page` , `perPage` , `totalValues` , `nextPage`  and `lastPage`
```// response
{
  "pagination": {
    "page": 1,
    "perPage": 2,
    "nextPage": 2,
    "totalValues": 7,
    "lastPage": 4
  }
} ```
I tried different solutions based on some threads I found on slack and some kapa answers but nothing worked
Basically, I'd like to be able to add the pagination parameters as in the picture below
But then I'm still not sure could I handle the `nextPage` / `lastPage`
Maybe in the body being able to use jinja as for `next_page_token` ?
I also tried using `response.get('pagination').get('nextPage', 1)`
Probably need to find a way to also use the `stop_condition` with the `response.get('pagination').get('page') &gt;= response.get('pagination').get('lastPage')`  ?

Here is the thread with kapa about that topic :slightly_smiling_face:
<https://airbytehq.slack.com/archives/C01AHCD885S/p1731407874753769>

Thanks for answering and don't hesitate if you need more informations
3

Thanks for providing that context! I’ve created a public issue for the one that I’m working on - I think it covers your case: https://github.com/airbytehq/airbyte/issues/48493

4

That should handle the request body injection at the least (injecting into pagination.perPage and pagination.page) - are you able to grab the params you need from the nested response body?

5

If you’re interested in using next page/last page instead of just having it increment, you probably want to switch from Page Increment to Cursor Pagination

6

Can you show an example using curl?

7

You can’t do this the correct way, but this is something we’ve just clocked as an issue and have started to work on fixing. If you could share your example so that I can make sure my solution aligns that would be really helpful!

8

Hey ! Sure :slightly_smiling_face:
The api I’m targeting use a pagination object
Just in case, <MODJO | Api V1 is the api endpoint documentation>

in the body with page and perPage values

{
  "pagination": {
    "page": 1,
    "perPage": 2
  }
}```
It return a `pagination`  object in the response with `page` , `perPage` , `totalValues` , `nextPage`  and `lastPage`
```// response
{
  "pagination": {
    "page": 1,
    "perPage": 2,
    "nextPage": 2,
    "totalValues": 7,
    "lastPage": 4
  }
} ```
I tried different solutions based on some threads I found on slack and some kapa answers but nothing worked
Basically, I'd like to be able to add the pagination parameters as in the picture below
But then I'm still not sure could I handle the `nextPage` / `lastPage`
Maybe in the body being able to use jinja as for `next_page_token` ?
I also tried using `response.get('pagination').get('nextPage', 1)`
Probably need to find a way to also use the `stop_condition` with the `response.get('pagination').get('page') &gt;= response.get('pagination').get('lastPage')`  ?

Here is the thread with kapa about that topic :slightly_smiling_face:
<https://airbytehq.slack.com/archives/C01AHCD885S/p1731407874753769>

Thanks for answering and don't hesitate if you need more informations
9

Thanks for providing that context! I’ve created a public issue for the one that I’m working on - I think it covers your case: https://github.com/airbytehq/airbyte/issues/48493

10

That should handle the request body injection at the least (injecting into pagination.perPage and pagination.page) - are you able to grab the params you need from the nested response body?

11

If you’re interested in using next page/last page instead of just having it increment, you probably want to switch from Page Increment to Cursor Pagination