Summary
User is facing issues with custom cursor pagination in Jira Data Center connector. They are encountering errors when trying to reference ‘startAt’ and ‘total’ properties in the JSON response. The stop condition in the sync job is not working as expected.
Question
Hi all,
I have the following setup to connect to a Jira Data Center instance. I’m having trouble trying to get the custom cursor pagination working as I’d expect. I tried using the AI tool here https://airbytehq.slack.com/archives/C01AHCD885S/p1709697465397369 and even though I set it up similarly I ran into errors when running the test connection. When I do a Test Connection I receive a json response that looks like this:
"status": 200,
"body": {
"expand": "schema,names",
"startAt": 0,
"maxResults": 50,
"total": 632,
"issues:" [truncated]
}
}```
Reading documentation I believe I should be able to reference the startAt and total properties by doing something like `response.body.startAt` or `response['body']['startAt']`. However, if I do that I receive this error:
```ERROR
Something went wrong in the connector. See the logs for more details. - Traceback (most recent call last):
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 129, in read
yield from self._read_stream(
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 224, in _read_stream
for record in record_iterator:
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 284, in _read_full_refresh
for record_data_or_message in stream_instance.read_full_refresh(configured_stream.cursor_field, logger, self._slice_logger):
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 121, in read_full_refresh
yield from self.read_records(
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 104, in read_records
yield from self.retriever.read_records(self.get_json_schema(), stream_slice)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 324, in read_records
for stream_data in self._read_pages(record_generator, self.state, stream_slice):
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 294, in _read_pages
next_page_token = self._next_page_token(response)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 258, in _next_page_token
return self._paginator.next_page_token(response, self._records_from_last_response)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/paginators/default_paginator.py", line 193, in next_page_token
return self._decorated.next_page_token(response, last_records)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/paginators/default_paginator.py", line 105, in next_page_token
self._token = self.pagination_strategy.next_page_token(response, last_records)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/paginators/strategies/cursor_pagination_strategy.py", line 59, in next_page_token
token = self.cursor_value.eval(config=self.config, last_records=last_records, response=decoded_response, headers=headers)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/interpolation/interpolated_string.py", line 40, in eval
return self._interpolation.eval(self.string, config, self.default, parameters=self._parameters, **kwargs)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/interpolation/jinja.py", line 91, in eval
return self._literal_eval(self._eval(default, context), valid_types)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/interpolation/jinja.py", line 109, in _eval
return self._environment.from_string(s).render(context)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 1, in top-level template code
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/jinja2/environment.py", line 466, in getitem
return obj[argument]
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'body'
ERROR
None - Traceback (most recent call last):
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/connector_builder/message_grouper.py", line 249, in _read_stream
yield from AirbyteEntrypoint(source).read(source.spec(self.logger), config, configured_catalog, {})
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 163, in read
yield from self.source.read(self.logger, config, catalog, state)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 167, in read
yield from super().read(logger, config, catalog, state)
File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 177, in read
raise AirbyteTracedException(message=error_message, failure_type=FailureType.config_error)
airbyte_cdk.utils.traced_exception.AirbyteTracedException: None```
If I change it to `response['startAt']` though it will at least make a successful request and show the results. However, the Stop Condition doesn't work as I'd expect it to. My sync job essentially just ran forever despite already pulling in all of the results. The stop condition is setup like so: `{{ response['body']['startAt'] >= response['body']['total'] }}`
Screenshot of full stream definition:
Does anyone know if how I'm approaching referencing these properties correct? Thanks in advance!
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1709773319044249) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["jira-data-center", "custom-cursor-pagination", "json-response", "sync-job", "stop-condition"]
</sub>