Inquiring about the capability of CDK to perform incremental syncs using a version integer as a cursor instead of a timestamp.
hi – does the CDK allow for incremental syncs where the cursor is a version integer instead of a timestamp? i have an object where lock_version is the only reliable cursor
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Yes, the CDK (Connector Development Kit) in Airbyte does allow for incremental syncs using a cursor that is not a timestamp. While a common cursor value is an updated_at timestamp, the cursor can be any value that delineates which data was pulled and which data is new. This means you can use an integer such as lock_version as your cursor. Here’s a general example of how incremental syncs work, using a timestamp as the cursor: sql SELECT * FROM table WHERE cursor_field >= 'last_sync_max_cursor_value' In your case, if lock_version is your cursor, the SQL query would look something like this: sql SELECT * FROM table WHERE lock_version >= 'last_sync_max_lock_version' Remember, the cursor field is used to identify new or updated records since the last sync. If modifications to the underlying records are made without properly updating the cursor field (lock_version in your case), then the updated records won’t be picked up by the Incremental sync as expected. For more information on incremental syncs, you can refer to the Airbyte documentation.
<@U04SE163WC8> but i don’t see an option in the CDK UI to define an integer cursor
Hmm, I don’t know enough to give you a confident answer yet. However, the Airbyte documentation provides some insights on how to define a cursor field in the sync settings dropdown for certain sources like Postgres. You can find more details here. For more advanced usage, you might want to check out the Airbyte Protocol for more technical details about the configuration data model. If you’re developing a connector, the Connector Development Tutorials might be of help. If you still have questions, I recommend posting your question on the Airbyte Forum or checking out the Airbyte Github for similar issues.