Custom Connector Pagination Issue

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

User is facing challenges with pagination in a custom connector using the Builder, specifically with injecting parameters into a nested JSON structure for a POST request body.


Question

Hello :slightly_smiling_face: (if this is the wrong channel, let me know and I will happily move it elsewhere!)

I am building a custom connector with the Builder. I want to paginate data from an API with an offset. The API am connecting to have some requirements that I fail to meet with the Builder. The request must be a POST request with the following body:

    "search": {
      "from_date": "2024-11-26",
      "to_date": "2024-11-27"
    },
    "config": {
      "limit": 1,
      "offset": 0
    }
  },```
Using the Pagination module, I can select `Inject Limit into outgoing HTTP Request` and Body JSON payload - but this does not support nested structures as far as I can see, meaning I cannot inject it into the `config` dict in the request body, only in root.
I have tried multiple work-arounds, like in the below picture with `next_page_token['next_page_token']` , but I cannot seem to get it to work. Any tips or pointers very much appreciated! Thank you in advance :slightly_smiling_face:

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1732719180186089) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['custom-connector', 'pagination', 'api', 'post-request', 'nested-structure']
</sub>
2

<@U082A7P32H5> When you test this stream with that config, what do you see being sent in the Request tab? (both on the first and subsequent requests?)

Are you getting some type of error, or is it just not working as expected? and if the latter, how is it misbehaving?

3

Okay, so I have figured it out.

  "config": {
    "limit": 5,
    "offset": "{{ next_page_token['next_page_token'] | default(0) }}"
  },
}```
This request body seems to work, but only if Pagination module is "enabled" (but without any injecting in the Pagination module). I am new to pagination in Airbyte, and did not at first understand how it works.
4

Ah, yes, it needs to be enabled, just not injected. Good job sorting it out!