Configuring Incremental Substreams in Connector Builder

Based on the information provided in the extracts, it seems like you have correctly set up the incremental_sync configuration for your substream. However, there are a few things you might want to check: 1. Cursor Field: Ensure that the cursor_field is correctly set to the field that represents the “last changed” or “updated at” timestamp in your data. In your case, it should be submitted_at if that’s the field in your data that represents when the record was last updated. 2. Datetime Format: Make sure the datetime_format matches the format of the date and time in your cursor_field. You have mentioned two different datetime formats in your configuration ('%Y-%m-%d+%H:%M:%S' and '%Y-%m-%d %H:%M:%S EST'). Ensure that these formats correctly match the format of the submitted_at field in your data. 3. Start and End Datetime: The start_datetime and end_datetime fields are used to specify the time range for fetching records. If you’re using the MinMaxDatetime type for start_datetime, make sure the datetime field is correctly set to represent the start of the time range for fetching records. The end_datetime is usually set to the current date and time when the sync is started to sync all changes that happened so far. 4. Request Options: The start_time_option and end_time_option fields are used to set request options. Make sure the field_name is correctly set to the request parameter that should be used to filter the data based on the start and end time. Also, ensure that inject_into is set to request_parameter if the start and end time should be added as a query parameter to the request. If you’ve checked all these points and you’re still facing issues, it might be helpful to refer back to the Airbyte documentation or ask for help on the Airbyte forum. Remember, incremental syncs work by pulling only the data that has changed since the previous sync. This is usually implemented using a cursor value (like a timestamp) that delineates which data was pulled and which data is new. The cursor value is typically an updated_at timestamp. This cursor means that records whose updated_at value is less than or equal to that cursor value have been synced already, and that the next sync should only export records whose updated_at value is greater than the cursor value.