Summary
The user is facing an issue with double pagination in the connectorBuilder. They need to pull one month at a time until the date is after today and also need to page within that month. The user provided an example scenario with specific details.
Question
i have double pagination… need to pull one month at a time until the date is after today, and also need to page within that month… pass in 2022-01-01 asking for 100 per page, 150 in that month, first call gives back 100 & has_more=true, second page gives back 50 & has_more=false, then need to call with 2022-02-01. no way to do this in the builder right?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["double-pagination", "connectorbuilder", "pagination", "date-filtering", "paging"]
Based on the information provided in the extracts, the Airbyte Connector Builder currently supports common pagination methods such as Offset Increment, Page Increment, and Cursor Pagination. However, it does not explicitly mention support for nested or double pagination as you described. In your case, you are dealing with two levels of pagination: one at the month level and another at the page level within each month. This specific scenario might not be directly supported by the Connector Builder’s current pagination methods. If none of the provided pagination methods work for your API, you might need to use the low-code CDK or Python CDK to implement your custom pagination logic. The CDKs provide more flexibility and control over the data extraction process, which might be necessary for your use case. For more information, you can refer to the Connector Builder UI Pagination and Understanding the YAML file Pagination sections in the Airbyte documentation.
nevermind figured it out had to add end date into incremental sync and set split up interval