Trouble configuring pagination for Microsoft Dynamics custom connector in Airbyte

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Setting up pagination for a custom connector in a self-hosted Airbyte instance to bring Microsoft Dynamics data into BigQuery. Currently syncing one endpoint with 20,000 records but need help implementing pagination for more records.


Question

Hello everyone,
I’m working on setting up a custom connector in my self-hosted Airbyte instance to bring Microsoft Dynamics data into BigQuery. We’re currently syncing one of the endpoints and have successfully retrieved 20,000 records. However, the endpoint contains more than 20,000 records, so we’re looking to implement pagination. I’m having trouble configuring the pagination and would appreciate your help in getting it set up.

API Endpoint we are using: https://api.businesscentral.dynamics.com/v2.0/<tenant_id>/Production/ODataV4/Company(‘<id>’)/G_LBudgetEntries

Response we are getting after hitting the API endpoint:
“@odata.context”: “https://api.businesscentral.dynamics.com/v2.0/<tenant_id>/Production/ODataV4/$metadata#Company(‘<id>’)/G_LBudgetEntries”,
“value”: [ it contains 20,000 records],
“@odata.nextLink”: “https://api.businesscentral.dynamics.com/v2.0/<tenant_id>/Production/ODataV4/Company(‘<id>’)/G_LBudgetEntries
?aid=FIN&$skiptoken=537549”

Any help would be appreciated.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["custom-connector", "pagination", "microsoft-dynamics", "bigquery", "airbyte-instance"]
2

That’s a custom builder connector?

Use cursor pagination. Next page cursor is in the response body, in nextLink.

Select “inject cursor to http request” and choose option “path”.

That should do it. Happy to record a video, ping me if you’ll need help.

3

Hi Natik, thanks for your response. I tried what you suggested. Is this the correct way to provide the parameters?

4

I think the key should be @data.nextLink but the general idea seems right.