Incremental sync state for API without cursor value in returned data

Hello :wink:

I’m developing connector for an API that allows me to specify lastmodified as a query param and returns rows without any information about modification time. I want to emit state after all slices are processed. My 2 questions:

  • is it idiomatic to override AbstractSource.read and yield state message as a last operation?
  • is it sensible to emit ~now() as a cursor value for the next run?

Let me know if there’s better way!
Thanks :wink:

Hi @druid ,
Could you please explain why you want to emit the state only after all the slices are processed?

is it sensible to emit ~now() as a cursor value for the ne

If your source does not have the lastmodified field I think artificially creating at runtime with now could indeed be a workaround.

Thanks for your response! :wink:

During my tests I think I noticed that next requests within the same run used new cursor value - that’s why I went with ‘update state at the end’.
(Correct me if I’m wrong!)

I feel like I gave a more complete answer on your other topic. You can update the cursor value in the state by setting self._cursor_value.