Load all JIRA Projects from Source

Is it possible to retrieve all projects from the JIRA source connector withut explicitly listing them? If I don’t list any project keys the source test fails, I have 100+ projects to potentially import.

Basic normalization also fails for this connector - multiple streams fail with this error
Unsupported nested cursor field {’.’.join(self.cursor_field)} for stream {self.stream_name}\

Hey could you share the logs of the job so that we have better understanding on what is failing

2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_ctes/jira_airbyte/application_roles_ab1.sql from application_roles
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_ctes/jira_airbyte/application_roles_ab2.sql from application_roles
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_ctes/jira_airbyte/application_roles_ab3.sql from application_roles
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_tables/jira_airbyte/application_roles.sql from application_roles
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_ctes/jira_airbyte/avatars_ab1.sql from avatars
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_ctes/jira_airbyte/avatars_ab2.sql from avatars
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_ctes/jira_airbyte/avatars_ab3.sql from avatars
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_tables/jira_airbyte/avatars.sql from avatars
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_ctes/jira_airbyte/board_issues_ab1.sql from board_issues
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_ctes/jira_airbyte/board_issues_ab2.sql from board_issues
2022-06-02 15:18:49 e[42mnormalizatione[0m >   Generating airbyte_views/jira_airbyte/board_issues_stg.sql from board_issues
2022-06-02 15:18:49 e[42mnormalizatione[0m > {"type": "LOG", "log": {"level": "FATAL", "message": "Unsupported nested cursor field fields.updated for stream board_issues\nTraceback (most recent call last):\n  File \"/usr/local/bin/transform-catalog\", line 8, in <module>\n    sys.exit(main())\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py\", line 104, in main\n    TransformCatalog().run(args)\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py\", line 36, in run\n    self.process_catalog()\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py\", line 64, in process_catalog\n    processor.process(catalog_file=catalog_file, json_column_name=json_col, default_schema=schema)\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/catalog_processor.py\", line 76, in process\n    nested_processors = stream_processor.process()\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/stream_processor.py\", line 294, in process\n    self.generate_scd_type_2_model(from_table, column_names),\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/stream_processor.py\", line 699, in generate_scd_type_2_model\n    cursor_field = self.get_cursor_field(column_names)\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/stream_processor.py\", line 984, in get_cursor_field\n    raise ValueError(f\"Unsupported nested cursor field {'.'.join(self.cursor_field)} for stream {self.stream_name}\")\nValueError: Unsupported nested cursor field fields.updated for stream board_issues"}}
2022-06-02 15:18:49 e[42mnormalizatione[0m > {"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1654183129210.775, "error": {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "Unsupported nested cursor field fields.updated for stream board_issues", "stack_trace": "Traceback (most recent call last):\n  File \"/usr/local/bin/transform-catalog\", line 8, in <module>\n    sys.exit(main())\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py\", line 104, in main\n    TransformCatalog().run(args)\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py\", line 36, in run\n    self.process_catalog()\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py\", line 64, in process_catalog\n    processor.process(catalog_file=catalog_file, json_column_name=json_col, default_schema=schema)\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/catalog_processor.py\", line 76, in process\n    nested_processors = stream_processor.process()\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/stream_processor.py\", line 294, in process\n    self.generate_scd_type_2_model(from_table, column_names),\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/stream_processor.py\", line 699, in generate_scd_type_2_model\n    cursor_field = self.get_cursor_field(column_names)\n  File \"/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/stream_processor.py\", line 984, in get_cursor_field\n    raise ValueError(f\"Unsupported nested cursor field {'.'.join(self.cursor_field)} for stream {self.stream_name}\")\nValueError: Unsupported nested cursor field fields.updated for stream board_issues\n", "failure_type": "system_error"}}}
2022-06-02 15:18:49 e[42mnormalizatione[0m >

Is retrieving all projects possible or each key needs to be specified?

Hey I think the issue is same as https://github.com/airbytehq/airbyte/issues/8473. Do you mind taking this over the github issue

I don’t care about the normalization error, will use my own. More concerned with the project keys, do they need to be specified or is there a wildcard I can use to retrieve them all?

I don’t think we have a wildcard option right now

Any ideas why the JIRA source would be consuming 100% of CPU?

Unless it goes in a loop. If there is a problem do you mind creating a new topic with logs of that sync

The issue is with the issue_properties stream. It takes more than 2 days to execute vs 45 minutes for all other streams

Got it. Kindly create a new ticket/issue with the information so that team can look into