Issue with URL injection in pagination API

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Airbyte is modifying the API URL when injecting the ‘nextPageUrl’ into the path, causing a 404 error for requests beyond the first page. Need help troubleshooting.


Question

Hi all. I’m using a cursor pagination API to pull an element called ‘nextPageUrl’ out of the response, and injecting it into path in order to get the next page of records. For some reason, almost inexplicably, it seems that airbyte is scrubbing out part of the API URL upon injection – it should be 'https://pi.pardot.com/api/v5/objects/…“, and airbyte is always forcing the second page response URL to be 'https://pi.pardot.com/v5/objects/…” (without the ‘/api/’ before v5). It even does this if I force the prefix of the API URL to always be https://pi/pardot.com/api/v5/ with a plaintext prefix in front of the nextPage token. The result is a 404 error for all requests beyond the first page. Any ideas on what might be causing this? Thanks!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["pagination-api", "url-injection", "airbyte", "api", "404-error"]
2

New message text here

3

Kapa’s above reference to Github Issue #18136 is indeed the same issue I’m facing. <Low-code SDK: CursorPagination removes base path from from url_base · Issue #18136 · airbytehq/airbyte · GitHub reply> to that issue by edfincham pointed me to a sufficient workaround – I set the base_url in global config to ‘https://pi.pardot.com/api’, and then the cursor value to ‘api/v5/…’ without a leading forward slash. Working now. Thank you :slight_smile:

4

<@U06EJSPJDDZ> it looks a strange bug, happy you found a workaround!