Nested Cursor Pagination Support for Plaid Connector in Python CDK

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

The user is building a connector for Plaid in Python CDK but facing issues with nested cursor pagination. They are inquiring about the timeline for enabling nested cursors to support APIs like Plaid.


Question

i’m building a connector for Plaid, but one endpoint uses a nested cursor for pagination (i.e., a json body param like “options”:{“cursor”:”value”}), which is unsupported right now (tried using options.cursor with no luck)

since the Python CDK is being phased out, what’s the timeline for enabling nested cursors to support apis like Plaid?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["plaid-connector", "nested-cursor-pagination", "python-cdk", "api-support", "timeline"]
2

Wait up, Python CDK is not phased out. It’s not going anywhere. We’re refactoring it and driving it to 1.0 and we will work in it in the future. In fact, a lot of features land in it first, then propagate to low-code CDK.

But that’s besides the point! I just helped <@U06PL8174LX> with a situation where she needed to set a nested HTTP request json body value. TLDR is you can set the variable named options to the value of { "cursor": "value" }.

Now, if you have more than one key in options to set and you want to set them in different spots in the Builder, that won’t work. But for a simple cursor pagination, this should work — it’s exactly what worked for Elvira.

3

thanks, i double checked and it seems Python CDK is just being rejected for pull requests

regarding injecting the cursor inside options, the UI just offers a “Field name” input box, so not sure how to set it like you’re proposing

4

Field name: options

Value: { “cursor”: “{{whatever}}” }

5

do you mean in pagination i can somehow use the cursor to set a variable? the pagination section doesn’t have a “Value” input box

6

Hmmm. If you give me the yaml + api credentials, I can try and play around with it

7

i can’t but i’m setting up <API - Transactions | Plaid Docs endpoint>, trying to paginate using options.offset

8

Ah, right, sorry. Understood, plaid is rather sensitive, and I can get test credentials myself I think.

9

So you just need to paginate transactions list?

10

Are you using airbyte cloud btw?

11

i’m using airbyte cloud, although running tests locally too

12

Have you made Plaid connector from scratch, or used source-plaid that we already have? It does have the transaction stream with a paginator, but it’s not available on cloud out of the box, I need to figure out why

13

i started one from scratch because it’s missing in cloud

14

anyway i’m now just curious about whether it’s possible to use the nested offset field for pagination or not

15

also, according to the https://docs.airbyte.com/integrations/sources/plaid|docs, the plaid connector just retrieves balances

16

i found this out in the community plaid connector: it actually has the transactions endpoint, and properly implemented pagination