Summary
Issue with pagination using offset in Connector Builder for API, specifically when setting limit above 1. API supports pagination but offset increment is not working as expected.
Question
Hi All - I am fairly new to Airbyte and I am attempting setup a new connector using the connector builder. I am using this api https://cloud.getdbt.com which supports limits and offset. However when using pagination with offset increment pagination is not working for any limit value above 1 i.e if I set limit to 1 it paginates but if the limit is >1 then I simply get the limit set but no further offset above the first one. The api supports pagination and I am currently replicating calls that I use in another tool. Any advice on this?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["pagination", "offset", "connector-builder", "api", "limit", "pagination-issue"]
Hey there! Is you post the manifest (export from builder) and link to the api docs page about the endpoint you are trying to fetch, I’m happy to take a look.
It’s likely that some criteria for determining if it’s the last page kicks in when it should not.
<@U065RJ879QT> is the “Export Yaml” option? The endpoints are <dbt Developer Hub Runs>, and <dbt Developer Hub Events>
see attached. Thanks for looking into this and let me know if you need more info.
Okay I have a few things for you 
First off, here’s a video of how I fixed things: https://www.loom.com/share/efa1cf797b9649369374b2a3bb05c54e?sid=0da9c666-5dbe-4dc8-a3ce-0a156c48a09e
And the manifest.
I’ve taken the liberty to add more streams, and make a PR with this connector upstream: https://github.com/airbytehq/airbyte/pull/44541
Thanks for looking into this so promptly! FYI the audit_logs endpoints are only available on API V3
Ooops. Well, the connector is now in the cloud and oss registry, so you can just use it 