Summary
The user is inquiring about the possibility of using pagination and partitioning together in Airbyte’s custom connector, specifically for a use case involving multiple tables with the same schema and a need to fetch all data using pagination. The user initially assumed offset would be updated as offset + limit, but found in the documentation that it is based on the number of rows fetched.
Question
Hi team, is it possible to use pagination and partitioning together in Airbyte’s custom connector? I have a use case where I have multiple tables of the same schema. And the tables have more rows that my API needs to paginate and fetch all the data. Initially, I had looped the tables in the code assuming Airbyte updates offset as offset + limit, but in documentation, offset is updated based on the number of rows fetched.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.