Custom Connector Incremental Sync Date Range Issue

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

The user is facing an issue with a custom connector where the DatetimeBasedCursor class truncates the end_datetime parameter at the current datetime instead of using the specified date range. They are looking for a way to achieve the desired behavior or considering modifying the code to remove the end date limitation.


Question

I’m building a custom connector for an API that supports start_date and end_date parameters. With every sync I’d like to query a static date range +/- 2 weeks (4 weeks in total) and deduplicate objects based on an ID. I enabled Incremental sync as DatetimeBasedCursor and set the start_datetime to {{ now_utc() - duration('P14D') }} and end_datetime to {{ now_utc() + duration('P14D') }} . However the connector does not respect this and keeps using the current date as the end date. It looks like DatetimeBasedCursor class always truncates the end_datetime parameter at current datetime: https://github.com/airbytehq/airbyte/blob/4652becd0b1ad1ab1d3968399c7975af834d2467/airbyte-cdk/python/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py#L181|datetime_based_cursor.py. Is there a way to achieve this behaviour? Or would be the process of changing the code to remove the end_date limitation?

A similar question was posted here: https://airbytehq.slack.com/archives/C01AHCD885S/p1710491351292609


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["custom-connector", "incremental-sync", "datetimebasedcursor", "end-date", "date-range", "code-modification"]