Trouble with URL encoding LastEvaluatedKey param in Connector Builder for pagination

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

The user is facing an issue with URL encoding the LastEvaluatedKey parameter in the Connector Builder while trying to build a tap from the Awtomic API for pagination.


Question

I’m attempting to use the Connector Builder to build a tap from Awtomic (https://api.awtomic.com). I’m running into a particular issue with the way this API handles pagination - on each request, it returns a LastEvaluatedKey param that is issued to the following request. This should translate neatly into a cursor based pagination method as described on the Airbyte docs, but I’m having trouble URLencoding the string in LastEvaluatedKey . Screenshot here with additional context:

Has anyone had any success URL encoding params to be injected into a follow-up request using the Connector Builder?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["connector-builder", "url-encoding", "pagination", "lastevaluatedkey", "awtomic-api"]
2

{{ response[‘LastEvaluatedKey’] | urlencode }} doesn’t really work on strings

3

Could you share an example of what response['LastEvaluatedKey'] looks like?

4

Also looking at the API docs, you may need to change that to a lowercase l, e.g. response['lastEvaluatedKey'] instead of response['LastEvaluatedKey']

5

Hi Lake -
response[“lastEvaluatedKey”] looks like:

    "PK": {
      "S": "SHOP#*****.<http://myshopify.com|myshopify.com>"
    },
    "SK": {
      "S": "#SC#827****38791"
    }
  }```
6

i was wrong about urlencoding, looks like Airbyte does that out of the box

7

I just tried this for the Cursor Value: {{ response.get("lastEvaluatedKey", {}) }}

it returns responses from the API, but in the debug toolbar I cant’ quite tell if the param is being injected into the URL

8

this is all I see in the Request body:

  "url": "<https://api.awtomic.com/subscriptions>",
  "parameters": {
    "limit": [
      "100"
    ]
  },
  "headers": {
    "User-Agent": "python-requests/2.31.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "x-api-key": "****"
  },
  "http_method": "GET",
  "body": ""
}```
9

are you sure that I need to lowercase the l when parsing the Response object? this is what the response looks like:

I have the injected param lowercased, which is what the API expects.

10

Ah, I was just going off of the <https://docs.awtomic.com/reference/getcustomersubscriptions|API docs> where it looked like a lowercase l, but if that is how your response looks then you should match what is there.

It looks like it is not currently being injected; if it was, it would appear in the parameters object in your Request tab.

I would recommend setting Next page cursor to Response instead of Custom, and setting Path to LastEvaluatedKey

11

but got a 404:

Request to <https://api.awtomic.com/subscriptions?lastEvaluatedKey=PK&amp;lastEvaluatedKey=SK&amp;limit=100> failed with status code 400 and error message invalid lastEvaluatedKey - Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/connector_builder/message_grouper.py", line 249, in _read_stream
    yield from AirbyteEntrypoint(so```
12

and this is the raw Request:

  "url": "<https://api.awtomic.com/subscriptions>",
  "parameters": {
    "lastEvaluatedKey": [
      "PK",
      "SK"
    ],
    "limit": [
      "100"
    ]
  },
  "headers": {
    "User-Agent": "python-requests/2.31.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "x-api-key": "****"
  },
  "http_method": "GET",
  "body": ""
}```
13

so i dont think it’s formatting the LastEvaluatedKey correctly

14

is it a case of the LastEvalutedKey being null in the first response?

15

seeing

    "lastEvaluatedKey": [
      "PK",
      "SK"
    ],
    "limit": [
      "100"
    ]
  },```
in the params led me down the path of url encoding, but now I’m not sure.

<@U02T7NVJ6A3> any other ideas to try? i’ve been at this for a couple of days, maybe will have to move to the low-code connector if we need to parse the key from the Response and stringify it
16

So it looks like the value of LastEvaluatedKey in your response is an object. What exact format of lastEvaluatedKey does the API expect in your request for the next page? Are you able to test a request and provide an example of how exactly that parameter should be formatted?

17

yes. turns out the API expects the entire JSON value of LastEvaluatedKey , urlencoded

18

<@U02FACC3815> it looks like Airbyte doesn’t properly url encode dictionary objects by default when setting the Path to a field which has an object value.

However, I think I have found a workaround for this: try switching back to Custom Next page cursor, and in Cursor Value, put this:
{{ response['LastEvaluatedKey'] | tojson | urlencode }}
It seems that urlencode doesn’t properly encode the object when it is passed in as a raw object, so this workaround first converts the object to a JSON string with | tojson, and then passes that resulting string to urlencode

19

Lake, closer!

  "url": "<https://api.awtomic.com/subscriptions>",
  "parameters": {
    "lastEvaluatedKey": [
      "%7B%22PK%22%3A%20%7B%22S%22%3A%20%22SHOP%23soundstrue.myshopify.com%22%7D%2C%20%22SK%22%3A%20%7B%22S%22%3A%20%22%23SC%238276017351%22%7D%7D"
    ],
    "limit": [
      "100"
    ]
  },
  "headers": {
    "User-Agent": "python-requests/2.31.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "x-api-key": "****"
  },
  "http_method": "GET",
  "body": ""
}```
but still getting a 404
20

that doesn’t look properly urlencoded though