I’m developing connector for an API that allows me to specify
lastmodified as a query param and returns rows without any information about modification time. I want to emit state after all slices are processed. My 2 questions:
- is it idiomatic to override
AbstractSource.read and yield state message as a last operation?
- is it sensible to emit
~now() as a cursor value for the next run?
Let me know if there’s better way!
Hi @druid ,
Could you please explain why you want to emit the state only after all the slices are processed?
is it sensible to emit
~now() as a cursor value for the ne
If your source does not have the
lastmodified field I think artificially creating at runtime with
now could indeed be a workaround.
Thanks for your response!
During my tests I think I noticed that next requests within the same run used new cursor value - that’s why I went with ‘update state at the end’.
(Correct me if I’m wrong!)
I feel like I gave a more complete answer on your other topic. You can update the cursor value in the state by setting
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.