Injecting limit and offset values into nested path in JSON body using Connector Builder for GraphQL API with Offset Increment pagination strategy

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

When using the Connector Builder to connect to a GraphQL API with Offset Increment pagination, the user is seeking guidance on how to inject limit and offset values into a nested path under the variables key in the JSON body to enable pagination and Incremental Sync.


Question

Utilizing the Connector Builder, I am trying to create a connection to a GraphQL API and implement pagination. The API is using the Offset Increment pagination strategy. In order to create the appropriate HTTP POST request, I need to inject the limit and offset into the JSON body, but it needs to be keys nested under a variables object. An example body is:

  "query": "GraphQL query string",
  "variables": {
    "limit": num,
    "offset": num,
  }
}```
Is there a way to specify that the `OffsetIncrement` strategy injects the values into a nested path inside the JSON body (either using the UI or the low-code YAML definition)? I'm hoping it is just some syntax that I'm missing as being able to inject items under the `variables` key would also be quite useful to enable Incremental Sync (the API I'm working with also allows for `updatedAt` filters utilizing a similar mechanism)

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1708113278128069) if you want to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["connector-builder", "graphql-api", "offset-increment", "pagination", "http-post-request", "json-body", "variables-key", "incremental-sync"]
</sub>
2

Based on the provided extract, the Connector Builder in Airbyte currently supports injecting values into the query parameters, headers, or body. However, the extract does not provide specific details on how to inject values into a nested path inside the JSON body, such as the variables object in your example. Unfortunately, without additional information or extracts, I cannot provide a confident answer on whether there’s a specific syntax or method to achieve this using the Connector Builder’s UI or the low-code YAML definition. For more detailed information, I would recommend referring to the Airbyte documentation or seeking advice from the Airbyte community or Slack channel.

3

<@U04SE163WC8> Is there a variable that can be used to interpolate the offset value?

4

Hmm, I don’t know enough to give you a confident answer yet. The extract provided does not mention a variable for interpolating the offset value. However, you can find more information about incremental syncs, which might include details about offset values, on the incremental sync concept page.

5

Hmm, I don’t know enough to give you a confident answer yet. The extract provided does not mention any specific variable for interpolating the offset value. However, it does mention that the Connector Builder supports injecting values into various parts of the request, which could potentially include an offset value. For more detailed information, you may want to refer to the Airbyte Documentation.

6

For posterity for anyone who find this later: I was able to solve pagination and incremental sync through the use of custom parameter injection into the request body as detailed <Pagination | Airbyte Documentation for pagination> and <Incremental sync | Airbyte Documentation for incremental sync>

The only item that I needed to hardcode into the JSON body was the limit, which was no huge loss

7

Sample request body:

  "query": "GraphQL query string",
  "variables": {
    "offset": "{{ next_page_token['next_page_token'] or 0 }}",
    "pageSize": 1000,
    "updatedAtEnd": "{{ stream_interval.end_time }}",
    "updatedAtStart": "{{ stream_interval.start_time }}"
  }
}```