Troubleshooting Cursor Pagination for Connector

Summary

User is experiencing issues with cursor pagination in a specific connector they are developing.


Question

Troubleshooting cursor pagination for a connector im working on :thread:



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['troubleshooting', 'cursor-pagination', 'connector']

There’s a header repsonse Link that includes a value like this

"Link": "&lt;<https://api.bugsnag.com/projects/MyProjectIDHere/events?base=2024-11-27T16%3A49%3A20Z&amp;offset%5Bcount%5D=30&amp;offset%5Bevent_ids%5D%5B%5D=67460735010ce401a56b0000&amp;offset%5Bevent_ids%5D%5B%5D=67460735010cfaa7e3080000&amp;offset%5Bevent_ids%5D%5B%5D=67460735010ceb4929b90000&amp;offset%5Btimestamp%5D=2024-11-26T17%3A36%3A53.000Z&amp;offset%5Btotal_count%5D=703>&gt;; rel=\"next\""

If I choose Link as the path for the next page and inject into path, the injected value is

<https://api.bugsnag.com/%3C/projects/MyProjectIDHere/events?base=2024-11-27T16%3A49%3A20Z&amp;offset%5Bcount%5D=30&amp;offset%5Bevent_ids%5D%5B%5D=67460735010ce401a56b0000&amp;offset%5Bevent_ids%5D%5B%5D=67460735010cfaa7e3080000&amp;offset%5Bevent_ids%5D%5B%5D=67460735010ceb4929b90000&amp;offset%5Btimestamp%5D=2024-11-26T17%3A36%3A53.000Z&amp;offset%5Btotal_count%5D=703%3E;%20rel=%22next%22>

which is returning 404 because of the URL encoding

is there any way to clean this up w/ a transform or similar so that pagination works as expected? cc <@U01MMSDJGC9> I had tried opening a support ticket regarding this

I’d recommend to try using Custom Cursor, something like {{ headers.get("Link") | regex_search('\$offset[count]=([^&amp;]+)') }} let me know if you could get this solve if not let’s schedule a quick call to help you :slightly_smiling_face:

That’s enough to get me started, thx Marcos!

Could you find a solution <@U021P0L72CV>?

Haven’t attempted yet, we had to setup new orchestration and upgrade airbyte to get that function

Please keep me updated!