Source Close - Stream email_activity sync fail

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Amazon Linux EC2 t2.large
  • Memory / Disk: 8Gb/50Gb
  • Deployment: Docker
  • Airbyte Version: 0.39.4
  • Source name/version: Close.com 0.2.0
  • Destination name/version: BigQuery
  • Step: Sync
  • Description:

I am encountering the following error when syncing email_activity

"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"Request to https://api.close.com/api/v1/activity/email/?_skip=250100&_limit=100&date_created__gt=2021-03-01T00%3A00%3A00Z&date_created__lte=2022-02-28T00%3A00%3A00Z failed with status code 400 and error message **The skip you set is larger than the maximum skip for this resource (max_skip = 250000)**.
Please refine your search query.","stack_trace":"Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 122, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 112, in read
    yield from super().read(logger, config, catalog, state)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 125, in read
    raise e
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 111, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 180, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 243, in _read_incremental
    for message_counter, record_data_or_message in enumerate(records, start=1):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 115, in read_records
    for record in self.retriever.read_records(sync_mode, cursor_field, stream_slice, stream_state):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 374, in read_records
    for record in records_generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 430, in _read_pages
    yield from records_generator_fn(request, response, stream_state, stream_slice)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 415, in parse_records_and_emit_request_and_responses
    yield from self.parse_response(response, stream_slice=stream_slice, stream_state=stream_state)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 326, in parse_response
    raise ReadException(error_message)
3

This error message is indicating that the connector is encountering an issue when making a request to the Close.com API. The specific issue is that the _skip parameter in the request is set to a value larger than the maximum allowed value of 250000.

The _skip parameter is used to paginate the results of a query by skipping a certain number of rows in the response. For example, if you are requesting 100 rows at a time and want to retrieve the second page of results, you would set _skip to 100.

It looks like you are trying to skip more rows than the API allows, which is causing the request to fail with a status code of 400 and the error message “The skip you set is larger than the maximum skip for this resource (max_skip = 250000)”.

To resolve this issue, you will need to modify your request to set the _skip parameter to a value less than or equal to 250000. You may also want to consider modifying your query to return fewer results, as it seems like you are trying to retrieve a very large number of rows.

Hope this help!

4

Hi @sajarin ,
Yes that seems to be the issue. However, I have no control on the request that is made, it is part of the Airbyte developed connector. And yes I am trying to sync the whole data, which is quite large.
Would you please be able to file this as a bug ?
Thanks

5

I filed the following issue : Close.com connector fails to synchronize large number of items · Issue #20959 · airbytehq/airbyte · GitHub