Trying to add better incremental tests to various sources. Can someone direct me to some (good) examples, is there there some boilerplate code or method for adding these tests?
More details:
e.g. I would like to convert qualaroo source connector into an incremental stream.
e.g. I would like to add incremental tests to survey monkey source connector incremental response stream.
- is there a clear (best practices) example of defining incremental streams? Incremental streams appear to be defined inconsistently across various source connectors.
- I would like to add incremental tests to the survey monkey source connector. It is currently broken. Previously the incremental stream for responses was not pulling all the responses. I created a PR to fix that, which it did i.e. it now pulls all responses but it doesn’t work incrementally i.e. it does full overwrite. I would like a good example of incremental stream tests that I can apply to survey monkey that also conforms with airbyte’s general connector philosophy.
Hey @danieldiamond,
- is there a clear (best practices) example of defining incremental streams? Incremental streams appear to be defined inconsistently across various source connectors.
The latest approach for declaring incremental streams is to make your stream inherits from HttpStream and IncrementalMixin. This is documented here
You have to define a state getter with a @property decorated function and a state setter with a function decorated with @state.setter which is initializing the cursor_value at the start of the sync by receiving the latest state store on Airbyte database. You can then change self._cursor_value in the read_records function.
The get_updated_state function that you will find on most of our existing incremental streams is now deprecated.
I would like to add incremental tests to the survey monkey source connector.
Our source acceptance tests are covering a couple of scenarios that are described here. If you want to add additional tests you can write survey monkey specific acceptance tests in airbyte-integrations/connectors/source-surveymonkey/integration_tests/ .
I would like a good example of incremental stream tests that I can apply to survey monkey that also conforms with airbyte’s general connector philosophy.
I’m not sure that I can share an example that can specifically apply to survey monkey. As a rule of thumb, our connectors labeled as Generaly Available have an increased test coverage compared to the other connectors. I’d suggest you check their specific integration or unit tests as an example, but these tests will probably be very source-specific.
1 Like
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.