Based on the information provided in the extracts, it seems like you have correctly set up the incremental_sync
configuration for your substream. However, there are a few things you might want to check: 1. Cursor Field: Ensure that the cursor_field
is correctly set to the field that represents the “last changed” or “updated at” timestamp in your data. In your case, it should be submitted_at
if that’s the field in your data that represents when the record was last updated. 2. Datetime Format: Make sure the datetime_format
matches the format of the date and time in your cursor_field
. You have mentioned two different datetime formats in your configuration ('%Y-%m-%d+%H:%M:%S'
and '%Y-%m-%d %H:%M:%S EST'
). Ensure that these formats correctly match the format of the submitted_at
field in your data. 3. Start and End Datetime: The start_datetime
and end_datetime
fields are used to specify the time range for fetching records. If you’re using the MinMaxDatetime
type for start_datetime
, make sure the datetime
field is correctly set to represent the start of the time range for fetching records. The end_datetime
is usually set to the current date and time when the sync is started to sync all changes that happened so far. 4. Request Options: The start_time_option
and end_time_option
fields are used to set request options. Make sure the field_name
is correctly set to the request parameter that should be used to filter the data based on the start and end time. Also, ensure that inject_into
is set to request_parameter
if the start and end time should be added as a query parameter to the request. If you’ve checked all these points and you’re still facing issues, it might be helpful to refer back to the Airbyte documentation or ask for help on the Airbyte forum. Remember, incremental syncs work by pulling only the data that has changed since the previous sync. This is usually implemented using a cursor value (like a timestamp) that delineates which data was pulled and which data is new. The cursor value is typically an updated_at
timestamp. This cursor means that records whose updated_at
value is less than or equal to that cursor value have been synced already, and that the next sync should only export records whose updated_at
value is greater than the cursor value.