Step: The issue occurred when upgrading to the latest version
Description: The source-jira connector fails with a KeyError for ‘projectId’ when attempting to sync board_issues. Based on the commits, it appears this was introduced in version 0.3.1.
When retrieving boards for the affected project there is a board with a location that looks like this:
“location”: {
“userId”: 1234,
“userAccountId”: “1234abcd”,
“displayName”: “Some Name”,
“avatarURI”: “https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/1234…”,
“name”: “Some Name”
}
This location does not have a projectId field and therefore there is a KeyError when looking for one. It seems like there should be a check to see if this field exists and maybe some case to use the projectId that is used in the request if one is not present.
Full Traceback below:
2023-01-19 00:27:01 source > Encountered an exception while reading stream board_issues
Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py”, line 111, in read
yield from self._read_stream(
File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py”, line 180, in _read_stream
for record in record_iterator:
File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py”, line 290, in _read_full_refresh
for record_data_or_message in record_data_or_messages:
File “/airbyte/integration_code/source_jira/streams.py”, line 232, in read_records
for board in read_full_refresh(self.boards_stream):
File “/airbyte/integration_code/source_jira/utils.py”, line 23, in read_full_refresh
for record in records:
File “/airbyte/integration_code/source_jira/streams.py”, line 192, in read_records
for board in super().read_records(**kwargs):
File “/airbyte/integration_code/source_jira/streams.py”, line 90, in read_records
yield from super().read_records(**kwargs)
File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py”, line 413, in read_records
yield from self._read_pages(
File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py”, line 430, in _read_pages
yield from records_generator_fn(request, response, stream_state, stream_slice)
File “/airbyte/integration_code/source_jira/streams.py”, line 81, in parse_response
yield self.transform(record=record, **kwargs)
File “/airbyte/integration_code/source_jira/streams.py”, line 197, in transform
record[“projectId”] = str(record[“location”][“projectId”])
KeyError: ‘projectId’
Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:
It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.
Thank you for your time and attention.
Best,
The Community Assistance Team
Sorry. issue another problem in Sprint_issues stream
Log:
File “/airbyte/integration_code/source_jira/streams.py”, line 1071, in read_records
for board in read_full_refresh(self.boards_stream):
File “/airbyte/integration_code/source_jira/utils.py”, line 23, in read_full_refresh
for record in records:
File “/airbyte/integration_code/source_jira/streams.py”, line 192, in read_records
for board in super().read_records(**kwargs):
File “/airbyte/integration_code/source_jira/streams.py”, line 90, in read_records
yield from super().read_records(**kwargs)
File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py”, line 413, in read_records
yield from self._read_pages(
File “/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py”, line 430, in _read_pages
yield from records_generator_fn(request, response, stream_state, stream_slice)
File “/airbyte/integration_code/source_jira/streams.py”, line 81, in parse_response
yield self.transform(record=record, **kwargs)
File “/airbyte/integration_code/source_jira/streams.py”, line 197, in transform
record[“projectId”] = str(record[“location”][“projectId”])
KeyError: ‘projectId’
2023-01-27 08:14:03 source > Finished syncing sprint_issues
I did not find a solution, but I did find a workaround of rolling the version back to 0.3.0. This unfortunately means I can’t benefit from any further updates until this bug is patched.
Thanks for the clarification and sorry to hear the latest version isn’t working. Would it be possible to make an issue on github repo about this if you haven’t already?