Pagination Issue in Custom Connector

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

User is encountering a pagination issue in their custom connector due to a maximum retrieval limit of 30 pages while attempting to fetch orders with a page size configured at 100. The request is returning a 400 error status code with a message indicating that the maximum page limit has been exceeded.


Question

Hello!
I’m working on building a custom connector.
I have a problem with pagination because:
The limit of information retrieval is 30 pages, the default number of orders per page is 15 and it is possible to configure it up to 100 using the per_page parameter.
I have this configuration in my connector:

        type: DefaultPaginator
        page_size_option:
          type: RequestOption
          field_name: per_page
          inject_into: request_parameter
        page_token_option:
          type: RequestOption
          field_name: page
          inject_into: request_parameter
        pagination_strategy:
          type: PageIncrement
          page_size: 100
          start_from_page: 1```
And I’m getting this error:
`Request to <https://paulatorres.vtexcommercestable.com.br/api/oms/pvt/orders?f_creationDate=creationDate%3A%5B2024-10-15T00%3A00%3A00.000000Z+TO+2024-11-29T14%3A08%3A58.000000Z%5D&amp;page=31&amp;per_page=100> failed with status code 400 and error message Max page exceed ( 30 ), refine filter`
There is some way I can address this issue?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1732889564387249) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['pagination-issue', 'custom-connector', 'api-error', 'page-size', 'max-page-exceed']
</sub>
2

You can add an error handler whatever the API returns 400 to stop the sync or create a custom cursor and use some jinja to do the stop criteria

3

Thank you Marcos for the response. But this way, we can not be able to get the full information, isn’t it?

4

If the connection succeeded it will store the state, the next sync will continue from where it stopped