Incremental sync state for API without cursor value in returned data

Connector Development
1

Hello :wink:

I’m developing connector for an API that allows me to specify lastmodified as a query param and returns rows without any information about modification time. I want to emit state after all slices are processed. My 2 questions:

  • is it idiomatic to override AbstractSource.read and yield state message as a last operation?
  • is it sensible to emit ~now() as a cursor value for the next run?

Let me know if there’s better way!
Thanks :wink:

2

Hi @druid ,
Could you please explain why you want to emit the state only after all the slices are processed?

is it sensible to emit ~now() as a cursor value for the ne

If your source does not have the lastmodified field I think artificially creating at runtime with now could indeed be a workaround.

3

Thanks for your response! :wink:

During my tests I think I noticed that next requests within the same run used new cursor value - that’s why I went with ‘update state at the end’.
(Correct me if I’m wrong!)

4

I feel like I gave a more complete answer on your other topic. You can update the cursor value in the state by setting self._cursor_value.

5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.