Summary
The user is looking to customize the incremental sync behavior in a connector to use a custom filter in the API endpoint query parameter. The filter format requires passing a condition based on the last updated date from the previous successful sync state.
Question
Hi Team… Is there anyway I can change the incremental sync behavior to following? The api endpoint am working on building a connector has weird way to accept cursor fields
Query Parameter has a key “conditions” – which needs to have filter passed as following
?conditions=lastupdated>=[YYYY-MM-DD]
YYYY-MM-DD will be from last successful sync state
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["incremental-sync", "custom-filter", "api-connector", "query-parameter", "last-updated-date"]
Yes, you can customize the incremental sync behavior to accommodate the specific requirements of your API endpoint. If the API requires special handling that can’t be achieved with the standard “Inject start time / end time into outgoing HTTP request” option, you can disable this injection and use the generic parameter section at the bottom of the stream configuration form. This allows you to freely configure query parameters, headers, and properties of the JSON body using jinja expressions and available variables. You can also use these variables as part of the URL path. For example, if your API requires setting both start and end time in a query parameter, you can use the stream_interval variable to configure a query parameter with “key” query and “value” last_event_time BETWEEN TIMESTAMP "{{stream_interval.start_time}}" AND TIMESTAMP "{{stream_interval.end_time}}" to filter down to the right window in time. Here is an example of how you might set it up: yaml incremental_sync: type: DatetimeBasedCursor <...> start_time_option: type: RequestOption field_name: "conditions[gte]" inject_into: "request_parameter" end_time_option: type: RequestOption field_name: "conditions[lte]" inject_into: "request_parameter" Please refer to the documentation for more details on custom parameter injection in incremental syncs.
<@U04SE163WC8> what are all the yaml options we need to have for incremental-sync ?
For configuring incremental_sync in your YAML file, you need to specify the following options: 1. DatetimeBasedCursor: This can optionally set request options through the start_time_option and end_time_option fields. The respective values can be set on the outgoing HTTP requests by specifying where it should be injected. Here is an example: yaml incremental_sync: type: DatetimeBasedCursor start_datetime: "2021-02-01T00:00:00.000000+0000", end_datetime: "2021-03-01T00:00:00.000000+0000", step: "P1D" start_time_option: type: "RequestOption" field_name: "created[gte]" inject_into: "request_parameter" end_time_option: type: "RequestOption" field_name: "created[lte]" inject_into: "request_parameter" (Source: Incremental syncs) 2. CustomIncrementalSync: This is an incremental component whose behavior is derived from a custom code implementation of the connector. It has properties like class_name, cursor_field, and $parameters. Here is an example: yaml "source_railz.components.MyCustomIncrementalSync" (Source: CustomIncrementalSync) Remember, when a stream is read incrementally, a state message will be output by the connector after reading all the records, which allows for checkpointing. On the next incremental sync, the prior state message will be used to determine the next set of records to read. (Source: Incremental Syncs) For more details on incremental syncs, you can refer to the Incremental syncs section.
<@U04SE163WC8> is the following statement parsed by jinja
field_name: "created[lte]"
and gets converted to created<= ?
Sorry <@U066M8RPPPX>, I am experiencing especially high traffic right now, please try again. 
<@U066M8RPPPX> this should be achievable with a configuration like this:
Set the Key of the query parameter to conditions, and set the value to lastupdated>=[{{stream_interval.start_time}}] to pull the date from the incremental sync state