Trouble with URL encoding LastEvaluatedKey param in Connector Builder for pagination

Slack Knowledge Base Q&A Connector Development
, , , ,
21

That is just the format that we display the URL and parameters in. The actual url of the request will look like
<https://api.awtomic.com/subscriptions?lastEvaluatedKey=%7B%22PK%22%3A%20%7B%22S%22%3A%20%22SHOP%23soundstrue.myshopify.com%22%7D%2C%20%22SK%22%3A%20%7B%22S%22%3A%20%22%23SC%238276017351%22%7D%7D&amp;limit=100>
We have an <Issues · airbytehq/airbyte · GitHub issue> for actually showing the full url-encoded URL instead of separating out the parameters like we do today, but this shouldn’t affect the actual request being made

22

You can see the actual URL that is being requested in the ERROR logs if you are receiving a 404 error, it will say something like Request to &lt;url encoded url&gt; failed with status 404 and error message ...

23

ok, good to know.

I seems the Awtomic API is expecting this format for the encoding:

url = "<https://api.awtomic.com/subscriptions?limit=2&lastEvaluatedKey=%20%20%20%20%20%22PK%22%3A%20%7B%20%20%20%20%20%20%20%22S%22%3A%20%22SHOP%23XXXX.myshopify.com%22%20%20%20%20%20%7D%2C%20%20%20%20%20%22SK%22%3A%20%7B%20%20%20%20%20%20%20%22S%22%3A%20%22%23SC%238278638791%22%20%20%20%20%20%7D%20%20%20%7D>"

24

That’s a bit odd, because when I decode that value it returns a string in which there is an extra closing curly brace at the end which doesn’t match any opening brace (plus a bunch of unnecessary spaces)

25

might have copied weirdly, hang oin

26

<https://api.awtomic.com/subscriptions?limit=500&amp;lastEvaluatedKey=%7B%22PK%22%3A%20%7[…]0%22SK%22%3A%20%7B%22S%22%3A%20%22%23SC%238168833223%22%7D%7D>

27

so it would seem that | tojson | urlencode is doing something similar

28

I’m not really understanding that encoding, as when I url-decode that string I get this: %7B%22PK%22%3A%20%7[%E2%80%A6]0%22SK%22%3A%20%7B%22S%22%3A%20%22%23SC%238168833223%22%7D%7D which just seems like another encoded string (which itself doesn’t seem to be able to be url-decoded)

29

i’m just doing this in a Python script now.

so LastEvaluatedKey of {'PK': {'S': 'SHOP#<http://XXXX.myshopify.com|XXXX.myshopify.com>'}, 'SK': {'S': '#SC#8226537671'}}
gets encoded as

<https://api.awtomic.com/subscriptions?limit=500&lastEvaluatedKey=%7B%22PK%22%3A%20%7B%22S%22%3A%20%22SHOP%XXXX.myshopify.com%22%7D%2C%20%22SK%22%3A%20%7B%22S%22%3A%20%22%23SC%238226537671%22%7D%7D>

30

which decodes correctly back to {'PK': {'S': 'SHOP#<http://XXXX.myshopify.com|XXXX.myshopify.com>'}, 'SK': {'S': '#SC#8226537671'}}

31

in my python code I’m using

        encoded_key = urllib.parse.quote(json.dumps(last_evaluated_key))
        url = f"{base_url}&amp;lastEvaluatedKey={encoded_key}"```
32

when I try this in Airbyte:

33

the encoded URL is
Request to <https://api.awtomic.com/subscriptions?lastEvaluatedKey=%257B%2522PK%2522%253A%2520%257B%2522S%2522%253A%2520%2522SHOP%2523soundstrue.myshopify.com%2522%257D%252C%2520%2522SK%2522%253A%2520%257B%2522S%2522%253A%2520%2522%2523SC%25238277295303%2522%257D%257D&amp;limit=100> failed with status code 400 and error message invalid lastEvaluatedKey

34

and I couldn’t decode it back

35

<@U02T7NVJ6A3> any further ideas or suggestions for me to test?

36

<@U02FACC3815> Ah, it actually looks like the URL is going through one extra layer of encoding in the Airbyte case, since when I decode that ^ string it gives me what looks like the properly url-encoded form of the original payload.

So I’d suggest changing the Cursor Value to simply
{{ response['LastEvaluatedKey'] | tojson }}
and then the connector should automatically url-encode that when sending the request

37

ah i had such high hopes … :disappointed:

  "url": "<https://api.awtomic.com/subscriptions>",
  "parameters": {
    "lastEvaluatedKey": [
      "PK",
      "SK"
    ],
    "limit": [
      "100"
    ]
  },
  "headers": {
    "User-Agent": "python-requests/2.31.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "x-api-key": "****"
  },
  "http_method": "GET",
  "body": ""
}```
38

bu no, it’s an invalid key

39

are there any other jinja formatters built in, such as tostring?

40

You could try adding | string after the tojson and see if that helps, but I’m now suspecting that this is a bug in the auto-urlencoding logic. It seems like even if you pass it a stringified object, it will decode that into an object and then improperly encode that