Selecting a good cursor field when there aren't any

Connector Development
1

I’m building out a connector for Signrequest, and the Signrequest stream itself needs to be incremental but the object itself doesn’t have any fields that are good candidates for cursor fields. Documentation here

The closest thing I can find to dates are the “signed_on” fields within the objects of the “signers” array. I have a couple questions based on this:

  • Is it wise to select a cursor field that might not be present in every object?
  • When defining the cursor field in code (specifically Python), how exactly do we reference a field that’s within an object in an array? I’ve done “parent/child” for fields in nested objects on a different connector.
3

Hi @cornjuliox,

Is it wise to select a cursor field that might not be present in every object?

This is an interesting question. I’d advise you not to try to implement an incremental stream if the source API does not offer an obvious queriable cursor. In Signrequest API I don’t see any parameters that could be a good candidate for incremental load.

When defining the cursor field in code (specifically Python), how exactly do we reference a field that’s within an object in an array? I’ve done “parent/child” for fields in nested objects on a different connector.

You can’t, It’s only possible to declare nested cursor fields in an object, not in an array of object. In the case of a nested cursor field you can use a list to declare the cursor field: cursort_field = ["parent", "child"]. Unfortunately, nested cursor are not supported by normalization.

4

Thank you for the information. Just out of curiosity, if I were to add my own timestamp to each record, perhaps when the data is collected in parse_response() would that work, even as a “synthetic” cursor? I.e it has no relation to the data itself and serves only as a cursor for Airbyte to use when the next sync occurs.

5

Yes it would work but as you’re source API does not offer a filtering feature you will still re-read the same data on all sync.

6

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.