Summary
User is experiencing issues with cursor pagination in a specific connector they are developing.
Question
Troubleshooting cursor pagination for a connector im working on
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
['troubleshooting', 'cursor-pagination', 'connector']
There’s a header repsonse Link that includes a value like this
"Link": "<<https://api.bugsnag.com/projects/MyProjectIDHere/events?base=2024-11-27T16%3A49%3A20Z&offset%5Bcount%5D=30&offset%5Bevent_ids%5D%5B%5D=67460735010ce401a56b0000&offset%5Bevent_ids%5D%5B%5D=67460735010cfaa7e3080000&offset%5Bevent_ids%5D%5B%5D=67460735010ceb4929b90000&offset%5Btimestamp%5D=2024-11-26T17%3A36%3A53.000Z&offset%5Btotal_count%5D=703>>; rel=\"next\""
If I choose Link as the path for the next page and inject into path, the injected value is
<https://api.bugsnag.com/%3C/projects/MyProjectIDHere/events?base=2024-11-27T16%3A49%3A20Z&offset%5Bcount%5D=30&offset%5Bevent_ids%5D%5B%5D=67460735010ce401a56b0000&offset%5Bevent_ids%5D%5B%5D=67460735010cfaa7e3080000&offset%5Bevent_ids%5D%5B%5D=67460735010ceb4929b90000&offset%5Btimestamp%5D=2024-11-26T17%3A36%3A53.000Z&offset%5Btotal_count%5D=703%3E;%20rel=%22next%22>
which is returning 404 because of the URL encoding
is there any way to clean this up w/ a transform or similar so that pagination works as expected? cc <@U01MMSDJGC9> I had tried opening a support ticket regarding this
I’d recommend to try using Custom Cursor, something like {{ headers.get("Link") | regex_search('\$offset[count]=([^&]+)') }}
let me know if you could get this solve if not let’s schedule a quick call to help you
That’s enough to get me started, thx Marcos!
Could you find a solution <@U021P0L72CV>?
Haven’t attempted yet, we had to setup new orchestration and upgrade airbyte to get that function