Summary
When prototyping an API connector in the Builder, the user is working with an API that provides both created_at and updated_at fields. They want to pull an incremental stream using the created_at field for fetching only values from the previous sync, while also pulling all updated values using the updated_at field. Since the updated_at field does not include created_at, the user needs to make two different API calls. The question is whether it is possible to handle this scenario within the same stream or if they need to create two separate streams and later merge and de-duplicate the data themselves.
Question
Hello! question on Incremental streams. Prototyping API connector in the “Builder”
The API I am working on provides both created_at and updated_at fields. I’d like to pull an incremental stream using created_at field so only values from the previous sync are fetched, but also I’d like to pull all updated values. For this I need to make another call using updated_at parameter. In my case updated_at doesn’t include created_at, so I have to use 2 different calls.
Is it possible to do it in a same stream? or do I need to create 2 streams and later merge and de-dup them myself?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.