Incremental Sync Issue with Custom Connector

Based on the information provided in the extracts, it seems like you’re facing an issue with incremental syncs on your custom connector when there’s no data for certain days. This could be due to how the connector is handling the state or the cursor field for incremental syncs. In some cases, the connector might not be saving the state correctly, which could lead to skipping dates until it encounters a day with data. You can check the state after a sync in the Settings tab in the Connection page to see if your custom connector is saving the state correctly (source). If your API doesn’t have a “last_modified” field or equivalent, you might need to implement a workaround. One approach could be to fetch all the data for the first sync, and then fetch the last month’s data for each subsequent sync and append the new data. However, this approach might not be ideal if there are no changes in the data (source). Another approach could be to pull data in overlapping time periods to prevent potential problems with incomplete data. For example, you could grab data every week with lastmodified=now - 8 days (source). If your API doesn’t allow filtering records by a date field and only provides a paginated “feed” of data ordered from newest to oldest, you could set the “API time filtering capabilities” option to “No filter”. The connector will then automatically request the next page of records until the cutoff datetime is encountered (source). Please note that these are general suggestions and the exact solution might depend on the specifics of your API and connector. If you’re still facing issues, it might be helpful to post your question on the Airbyte forum or GitHub for more targeted assistance.