I’m building out a connector for Signrequest, and the Signrequest stream itself needs to be incremental but the object itself doesn’t have any fields that are good candidates for cursor fields. Documentation here
The closest thing I can find to dates are the “signed_on” fields within the objects of the “signers” array. I have a couple questions based on this:
- Is it wise to select a cursor field that might not be present in every object?
- When defining the cursor field in code (specifically Python), how exactly do we reference a field that’s within an object in an array? I’ve done “parent/child” for fields in nested objects on a different connector.
Is it wise to select a cursor field that might not be present in every object?
This is an interesting question. I’d advise you not to try to implement an incremental stream if the source API does not offer an obvious queriable cursor. In Signrequest API I don’t see any parameters that could be a good candidate for incremental load.
When defining the cursor field in code (specifically Python), how exactly do we reference a field that’s within an object in an array? I’ve done “parent/child” for fields in nested objects on a different connector.
You can’t, It’s only possible to declare nested cursor fields in an object, not in an array of object. In the case of a nested cursor field you can use a
list to declare the cursor field:
cursort_field = ["parent", "child"]. Unfortunately, nested cursor are not supported by normalization.
Thank you for the information. Just out of curiosity, if I were to add my own timestamp to each record, perhaps when the data is collected in
parse_response() would that work, even as a “synthetic” cursor? I.e it has no relation to the data itself and serves only as a cursor for Airbyte to use when the next sync occurs.
Yes it would work but as you’re source API does not offer a filtering feature you will still re-read the same data on all sync.