To convert UNIX timestamps to date from a field in a record, you can override the behavior of the stream slicer in Airbyte. The stream slicer by default supports standard datetime formats, but you can alter this to handle UNIX timestamps. Here’s an example of how you can configure the DeclarativeStream to handle UNIX timestamps: yaml incremental_sync: type: DatetimeBasedCursor cursor_field: "created" datetime_format: "%s" cursor_granularity: "PT1S" start_datetime: datetime: "{{ config['start_date'] }}" datetime_format: "%Y-%m-%d %H:%M:%S" end_datetime: datetime: "{{ today_utc() }}" datetime_format: "%Y-%m-%d" step: "P1W" start_time_option: field_name: "created_after" inject_into: "request_parameter" end_time_option: field_name: "created_before" inject_into: "request_parameter"
In this configuration, the datetime_format
is set to "%s"
, which is the format for UNIX timestamps. The cursor_field
is set to "created"
, which is the field that contains the UNIX timestamps. The start_time_option
and end_time_option
are set to "created_after"
and "created_before"
respectively, which could be used to set up incremental syncing. This configuration was used in the Delighted connector to handle a similar situation. Source: Forum Post: Handling UNIX timestamps with Low-Code CDK