Incremental Sync Query with Updated Field

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Adding search criteria with updated field for incremental sync


Question

Hey everyone,

"search": {"updated": [{"operator": ">", "value": "2024-07-10T13:21:35+00:00"}]}
That is what I would have to “add” to my query to get my incremental sync to work.

Any idea about how I could do that?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["incremental-sync", "query", "updated-field", "search-criteria"]
2 
cursor_field: updated
cursor_datetime_formats:
  - '%Y-%m-%dT%H:%M:%S%z'
datetime_format: '%Y-%m-%dT%H:%M:%S%z'
start_datetime:
  type: MinMaxDatetime
  datetime: '{{ config["start_date"] }}'
  datetime_format: '%Y-%m-%dT%H:%M:%SZ'
start_time_option:
  inject_into: request_parameter
  type: RequestOption
  field_name: search
  value: '{"updated":[{"operator":">","value":"{{ cursor }}"}]}'```
3

The above snippet of the YAML is what I “hope” to do - which doesnt work though, as there exists no “value:” in start_time_option.

4

My first try was like this:

cursor_field: updated
cursor_datetime_formats:
  - '%Y-%m-%dT%H:%M:%S%z'
datetime_format: '%Y-%m-%dT%H:%M:%S%z'
start_datetime:
  type: MinMaxDatetime
  datetime: '{"updated":[{"operator":">","value":{{ config["start_date"] }}}]}'
  datetime_format: ''
start_time_option:
  inject_into: request_parameter
  type: RequestOption
  field_name: search```
5

But here it will naturally throw an error, as the validation of the datetime regarding the datetime_format will fail. (defaults to cursor_datetime_formats if empty)

6

This is a case where you’ll likely have to use the the {{ stream_interval.start_time }} merge within the parameters or JSON body of the request. There are some notes on that in the docs under <Incremental sync | Airbyte Documentation Parameter Injection>.

It’s worth noting that depending on the API, sometimes you need to supply defaults or switching logic so this doesn’t break on the first request—but you’ll find more details around that in the low-code docs (which will usually point you in the right direction on how to handle it in Builder)

7

Thanks for the reply, I will try that!