Referencing cursor value in incremental syncs for low-code connector

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

When building a low-code connector using the UI with cursor paged pagination for incremental syncs, how can you reference the cursor value for future syncs? The documentation mentions filtering according to a cursor field, but it seems to expect a datetime value rather than a string.


Question

I’m building a low-code connector using the UI. The API in question doesn’t use datetime to perform incremental syncs, but rather cursor paged pagination to continue from where you left off.

Looking at the docs for cursor paged pagination, it doesn’t appear to store this value for future syncs. Is there a way I can reference this value in incremental syncs? I see <Incremental Syncs | Airbyte Documentation according to Cursor Field>, but this appears to be a datetime value, no a string.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["low-code-connector", "ui", "cursor-paged-pagination", "incremental-syncs", "cursor-value", "filtering", "yaml-file"]
2

I’m not entirely sure, but AFAIK incremental does not care about what you will use for pagination, whether it’s datetime or not. As long as it’s your pagination cursor and it increments, setting stream to be incremental should just save this value to stream state?

3

It definitely seems to want incrementals to be a time field. I’m running 0.50.16, perhaps that has changed in newer versions? I’ll check

4

Yeah, latest version also requires incremental to use a datetime, instead of the last fetched cursor. Would be nice to have this functionality!

5

Alrighty, let me go look around and ask the team

6

<@U069EMNRPA4> Hi - just following up on this.

If this isn’t feasible, likely going to change our approach to use Step Functions and/or just a simple container with some python code.

7

Yep. I was wrong. Lowcode only supports date time cursors for incremental. ;-(

8

Roger that <@U069EMNRPA4> - any plans to support cursor based incrementals?

9

Long term I think it makes sense (even assumed it’s possible already above). But that’s not on our plan for the next 1-2 months yet. File an issue for us with an example api that needs the behavior!