Summary
User is experiencing a limitation with the sync process of a custom connector built using the connector builder, specifically that it stops after processing 10,000 records and is seeking guidance on how to increase this limit.
Question
Hi Team,
Build a new custom connector using connector builder however, sync finishes after 10k records. how/where to increase the limit ?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
['custom-connector', 'connector-builder', 'sync-limit', 'record-limit']
There isn’t any such limit in Airbyte. I suspect that this is an issue with pagination, and the API is probably returning 10K results per page and you’re only getting a single page. I would check your pagination settings, as well as look at the requests being sent/received when testing in Builder (assuming this is a Builder connection).
Also make sure that the API you’re calling doesn’t have a default limit in the returned results you need to override. This is uncommon, but I have seen it.
<@U035912NS77> Thank you for swift reply. Very helpful. Indeed, it is pagination issue. How to test pagination whilst building connection ? I set path as pagination however, there is no preview for pagination
It will preview the pagination when it’s working correctly. This is only briefly mentioned in the tutorial, but you can see what it looks like in this video (this is the same video from the https://docs.airbyte.com/connector-development/connector-builder-ui/tutorial#adding-incremental-reads|docs, but I’m linking to it on Loom so I can take you to the exact second it shows the UI for partitions/pages):
https://www.loom.com/share/d52259513b664119a842809a4fd13c15?t=50
If you don’t see the pagination UI, it likely means it isn’t seeing the element you’re telling it to get the token from, or there’s some other config issue