I’m developing connector for an API that allows me to specify
lastmodified as a query param and returns rows without any information about modification time. I want to emit state after all slices are processed. My 2 questions:
- is it idiomatic to override
AbstractSource.read and yield state message as a last operation?
- is it sensible to emit
~now() as a cursor value for the next run?
Let me know if there’s better way!
Hi @druid ,
Could you please explain why you want to emit the state only after all the slices are processed?
is it sensible to emit
~now() as a cursor value for the ne
If your source does not have the
lastmodified field I think artificially creating at runtime with
now could indeed be a workaround.
Thanks for your response!
During my tests I think I noticed that next requests within the same run used new cursor value - that’s why I went with ‘update state at the end’.
(Correct me if I’m wrong!)
I feel like I gave a more complete answer on your other topic. You can update the cursor value in the state by setting