Trouble with Pagination in REST API Stream URL Path

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is facing issues with incorporating page numbers in the stream’s URL path for pagination in a REST API. They are struggling to determine the template parameter for the page number and encountering a problem where the page number overwrites the entire stream’s URL path.


Question

Heya, I have been trying to use Pagination to get a bunch of data from a REST API, where the page number should be part of the Path and running into two problems:

  1. I can’t seem to figure out what should be the template parameter to include that page number in the stream’s URL path
  2. It seems like whenever that page number is inserted, the stream’s URL path is completely overwritten. (maybe it’s connected to the 1st point)
    As an exact example, I have a URL in the form of base URL <https://www.yr.no/api/v0/> and the stream URL Path (for testing) locations/1-305409/observations/year/ which should add up to <https://www.yr.no/api/v0/locations/1-305409/observations/year/> for the query, and then with the pagintion appending the year to the end.

If I just use these settings, the whole URL path gets replaced by the “page”, e.g. <https://www.yr.no/api/v0/1950> which is not what I would want to get to but <https://www.yr.no/api/v0/locations/1-305409/observations/year/1950>.

Any thoughts?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["pagination", "rest-api", "url-path", "template-parameter", "connector"]
2

can you x-post this in <#C027KKE4BCZ|> ?

3

All bueno I see you.

4

<@U04705U9GN4> if you can make a builder connector in a trial cloud airbyte workspace and DM me the link, I’ll take a look.

You might be right that inserting the page number in path is just not a thing. Replacing path is a thing (need to update docs and highlight that that’s what will happen, PR welcome). Query parameters or headers are supported, but not a query parameter that’s part of url path ;(

There could be a workaround if you put a placeholder in the stream URL, but the problem is that the page number is not exposed as an interpolation variable.

If you’ll be stuck in that situation, it’d require a CDK change to wire things up so the page number can be inserted there nicely.

I assume you don’t control the API?

5

I did look at the <Pagination | Airbyte Documentation docs> but I didn’t see anything specific for the Path insertion.

I did try `{{ next_page_token[‘next_page_token’] }} but that didn’t seem to do the trick.

I’m attaching the declarative connector as well, that I’ve been been testing with.

6

Also, if there’s a better suggested way to iterate through a bunch of years, I’m all ears! (besides hand-listing the years in parametrized requests :see_no_evil: which does work, and I can add it to the Path just fune)