Error in Zoho CRM to BigQuery integration with incorrect time format

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The error is caused by a KeyError due to incorrect time format in the ‘Modified_Time’ field. It seems to be related to ISO datetime conversion function.


Question

Regarding my issues between Zoho CRM and BigQuery, I think I found the offending lines in the logs, but I am unsure of what I can do to fix the error. Is it possible that some records have an incorrect time format, seeing as it looks like the error is coming from the ISO datetime conversion function?

2024-08-13 14:46:03 source > Setting state of incremental_deals_zoho_crm_stream stream to {'Modified_Time': '2024-08-13T10:33:51-04:00'}
2024-08-13 14:46:03 source > Read 0 records from incremental_deals_zoho_crm_stream stream
2024-08-13 14:46:03 source > Finished syncing incremental_deals_zoho_crm_stream
2024-08-13 14:46:03 source > SourceZohoCrm runtimes:
Syncing stream incremental_contacts_zoho_crm_stream 0:00:00.070913
Syncing stream incremental_deals_zoho_crm_stream 0:00:00.089149
Syncing stream incremental_price__books_zoho_crm_stream 0:00:00.061487
2024-08-13 14:46:03 source > 'Modified_Time'
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 122, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 114, in read
    stream_is_available, error = stream_instance.check_availability(logger, self)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 190, in check_availability
    return self.availability_strategy.check_availability(self, logger, source)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 45, in check_availability
    get_first_record_for_slice(stream, stream_slice)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py", line 38, in get_first_record_for_slice
    return next(records_for_slice)
  File "/airbyte/integration_code/source_zoho_crm/streams.py", line 90, in read_records
    latest_cursor_value = datetime.datetime.fromisoformat(record[self.cursor_field])
KeyError: 'Modified_Time'```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1723664123152929) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["zoho-crm", "bigquery", "error", "keyerror", "iso-datetime", "time-format"]
</sub>
2

I’m guessing that a non-incremental sync would fix the issues (due to the cursor value becoming irrelevant) but this would be pretty taxing to completely refetch ~150k records per sync

3

It looks like the state format uses this modified column that is not valid for the actual cursor that the connector wants to use. Worth filing a GH issue, we’d need to check if there was a recent change and why the name doesn’t work anymore.

4

Got it, thanks Natik! Any advice on the details I should provide in the GH issue so that it is as clear as possible?