Summary
The user is facing an issue where a timestamp column without null values is not being recognized as a valid cursor column in Airbyte, resulting in an error message.
Question
Even though I have no null values in my cursor field, and the column is having type timestamp with zone still seeing errors that it is not valid column. Can someone help me with this issue
2024-02-29 00:32:01 - Additional Failure Information: io.airbyte.commons.exceptions.ConfigErrorException: The following tables have invalid columns selected as cursor, please select a column with a well-defined ordering with no null values as a cursor. {tableName='<table>', cursorColumnName='CREATED_AT', cursorSqlType=TIMESTAMP_WITH_TIMEZONE, cause=Unsupported cursor type}
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["timestamp-with-zone", "cursor-column", "invalid-column", "error"]
Based on the information provided in the extracts, it appears you are encountering an issue similar to one reported with the Snowflake source connector in Airbyte, where using a TIMESTAMP_WITH_TIMEZONE column as a cursor for incremental sync with deduplicated history results in an error. The error message you’re seeing indicates that the column type TIMESTAMP_WITH_TIMEZONE is not supported as a cursor field, despite having no null values and a well-defined ordering. Here’s the relevant extract for context: > Environment > - Airbyte version: 0.40.32 > - Source Connector and version: Snowflake 0.1.29 > > Current Behavior > Similar to #13603 getting an error when attempting to run an incremental sync with deduplicated history on any table with a TIMESTAMP_WITH_TIMEZONE column as cursor: > 2023-02-14 20:18:41 - Additional Failure Information: io.airbyte.commons.exceptions.ConfigErrorException: The following tables have invalid columns selected as cursor, please select a column with a well-defined ordering with no null values as a cursor. {tableName='redacted', cursorColumnName='HDB_LAST_SYNC', cursorSqlType=TIMESTAMP_WITH_TIMEZONE, cause=Unsupported cursor type},... This issue was documented in a closed GitHub issue (source) for the Snowflake source connector, but it’s plausible that similar limitations could apply to other source connectors as well. Actionable Steps: 1. Check Airbyte Version: Ensure you are using the latest version of Airbyte, as updates and bug fixes are regularly released. The issue you’re encountering might have been addressed in a newer version. 2. Select a Different Cursor Field: If possible, select a different column as your cursor field that does not use the TIMESTAMP_WITH_TIMEZONE data type. Typically, a simple TIMESTAMP or an INTEGER (if applicable) can serve as a reliable cursor field. 3. Contribute or Follow Up: Since you’re experiencing this issue and if it’s not yet resolved in the latest version, consider contributing to the discussion on the relevant GitHub issue or creating a new issue if one doesn’t already exist. This can help the Airbyte team prioritize and address the problem. Unfortunately, without more specific details about the source connector you’re using (other than the similarity to the Snowflake issue), this is the best guidance that can be provided based on the extracts. If your issue is with a different source connector, the general advice still applies, but the specifics might vary.