Setting up pagination in custom connector using Airbyte's low code UI tool

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

How to handle pagination in custom connectors built using Airbyte’s low code UI tool when the response contains a ‘next’ key with the new URL to request


Question

Hi everyone, if I was setting up pagination when building a c ustom connector using Airbyte’s low code UI tool, how would I be able to make a new request if in the response there is a “next” key with the value being the new url to request?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["pagination", "custom-connector", "low-code-ui-tool", "next-key", "request-url"]
2

In Airbyte’s Connector Builder, you can set up pagination using the “Cursor Pagination” method. This method is used when the API provides a “next” key in the response, which points to the URL of the next page of records. To configure this, you would need to specify the location of the next page cursor in the API response. If the cursor is located in the response body, you would select the “Response” option and provide a JSON path to the cursor. If the cursor is located in the response headers, you would select the “Headers” option and provide the header name. In some cases, you might need more control over how to obtain the cursor value and when to stop requesting pages. For this, you can use the “Custom” option and specify the “Stop Condition” using a jinja placeholder. For example, if your API also has a boolean more_results property included in the response to indicate if there are more items to be retrieved, the stop condition should be {{ response.more_results is false }}. For more detailed information, you can refer to the Cursor Pagination section in the Airbyte documentation.

3

I’ve tried this but it won’t make another request after the first one

4

Make sure you are setting the Next page cursor Path correctly - this will depend on the shape of the response and your Record Selector.

For example, if the raw HTTP response looks like:

  "status": 200,
  "body": {
     "data": {
       "next": "<url>",
       "id": 123
     }
   }
}```
and you don't have any record selector configured, then you'll need to set the `Path` to [`data` `next`]. But if you set the Record Selector to `data`, then the `Path` should just be [`next`]
5

Okay so it wouldn’t be

data.next
its
data, next

6

Yes, when you are entering the values into Path, you can either put a comma after each value or press Enter to enter in the value

7

okay thank you

so the only time we’d use dot notation is when we are doing someting like {{ record.data.next }}

8

awesome thank you for the help, i’ll give it a try!