Summary
When building a custom connector and wanting to make it incremental with a datetime field nested within ‘attributes.datetime’, how can the datetime field be provided in the cursor?
Question
Hi team, I am building custom connector and want to make it incremental. but my datetime field is not at root level instead it is nested attributes.datetime. How can I provide datetime field in cursor??
here is the response
“data”: [
{
“type”: “event”,
“id”: “”,
“attributes”: {
“timestamp”: 1728646977,
“event_properties”: {
“Subject”: “”,
“Recipient Email Address”: “”,
“$event_id”: “”
},
“datetime”: “2024-10-11T11:42:57+00:00”,
“uuid”: “”
}]
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["custom-connector", "incremental", "datetime-field", "nested-structure", "cursor"]
It appears that there are two fields; attributes.timestamp and datetime. While one is formatted as a UNIX timestamp and the other as a datetime, they represent the same value. If that’s always true, just use datetime at the top level.
If you aren’t already, you may also need to use a <Record processing | Airbyte Documentation Selector> of data (or data *) to treat the nested array as the top level.
If you do need to use a nested field, you’ll need to use an <Record processing | Airbyte Documentation“Add Field” Transformation> to copy that field to the top level, and then a <Record processing | Airbyte Documentation“Remove Field” Transformation> to remove the original copy. That added field can then be used as the Incremental Cursor field.