Rate Limit Exception on Custom Zendesk Connector

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

User is experiencing a rate limit exception when syncing a custom connector for Zendesk’s Export Search endpoint, specifically when attempting to extract a large number of tickets created on a specific date. The issue arises after reading approximately 15k records, leading to a 429 error response indicating API rate limit exceeded.


Question

Hello everyone

I’m trying to use a custom connector on the Zendesk Support Export Search endpoint to extract tickets created on a specific date (Nov 18th). Using the count endpoint on Postman I know that there are around 17k tickets created on this date. For some reason, when I start the sync the connector manages to read up to 15k records, but when it gets to the last “batch” of 5k tickets it starts receiving a rate limit exception. I also did some other tests with less than 5k tickets, but then what happens is that the rate limit exception happens right away. I let the exception run some times too, but even if it slept for more than a minute like the example below it doesn’t work.

2024-11-19 18:36:11 source > Marking stream tickets as STARTED
2024-11-19 18:36:11 source > Syncing stream: tickets 
2024-11-19 18:36:12 destination > Begin writing to the destination...
2024-11-19 18:36:13 destination > Creating StreamWriter for landing_dev:zendesk_tickets
2024-11-19 18:36:14 source > Marking stream tickets as RUNNING
2024-11-19 18:40:15 platform > Records read: 5000 (190 MB)
2024-11-19 18:44:26 platform > Records read: 10000 (381 MB)
2024-11-19 18:48:34 platform > Records read: 15000 (571 MB)
2024-11-19 18:50:35 source &gt; Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://&lt;subdomain&gt;.<http://zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&amp;page%5Bafter%5D=&lt;next_token&gt;&amp;per_page=100&amp;query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z|zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&amp;page%5Bafter%5D=&lt;next_token&gt;&amp;per_page=100&amp;query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z>, Response Code: 429, Response Text: {"error":"APIRateLimitExceeded","description":"Rate limit for Export API (api/v2/search/export) exceeded. Please wait 1 minute and try again."})
2024-11-19 18:50:35 source &gt; Retrying. Sleeping for 20 seconds
2024-11-19 18:50:56 source &gt; Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://&lt;subdomain&gt;.<http://zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&amp;page%5Bafter%5D=&lt;next_token&gt;&amp;per_page=100&amp;query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z|zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&amp;page%5Bafter%5D=&lt;next_token&gt;&amp;per_page=100&amp;query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z>, Response Code: 429, Response Text: {"error":"APIRateLimitExceeded","description":"Rate limit for Export API (api/v2/search/export) exceeded. Please wait 1 minute and try again."})
2024-11-19 18:50:56 source &gt; Retrying. Sleeping for 80 seconds
2024-11-19 18:52:42 source &gt; Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://&lt;subdomain&gt;.<http://zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&amp;page%5Bafter%5D=&lt;next_token&gt;&amp;per_page=100&amp;query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z|zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&amp;page%5Bafter%5D=&lt;next_token&gt;&amp;per_page=100&amp;query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z>, Response Code: 429, Response Text: {"error":"APIRateLimitExceeded","description":"Rate limit for Export API (api/v2/search/export) exceeded. Please wait 1 minute and try again."})
2024-11-19 18:52:42 source &gt; Retrying. Sleeping for 20 seconds```
I'm not using the official connector because I need a massive extraction, going back a few years, so my plan is to use a custom connector using the UI builder on the Export Search endpoint and running month-by-month for some days to be able to extract all tickets. (When we tested with the official connector it was very slow and was receiving Gateway timeout exceptions and re-attempting before even a million records).

Does someone has any experience with this? Any different extraction strategies for a large amount of tickets is also very welcome.

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1732052518689089) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['zendesk', 'custom-connector', 'rate-limit', 'api', 'export-search', 'tickets']
</sub>
2

I recommend reading documentation https://developer.zendesk.com/api-reference/introduction/rate-limits/

> If the rate limit is exceeded, the API responds with a <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429|429 Too Many Requests> status code. The response also has a Retry-After header that tells you how many seconds to wait before retrying API requests. Ensure your code handles 429 errors and waits the Retry-After interval before retrying requests.
>
> For tips on avoiding and handling 429 errors, see <https://developer.zendesk.com/documentation/ticketing/using-the-zendesk-api/best-practices-for-avoiding-rate-limiting/|Best practices for avoiding rate limiting>.
> Export Search Results GET /api/v2/search/export?query={query} 100 requests per minute per account
> Zendesk might limit requests if it detects an unusual spike in requests from all sources for the account, including internal product requests. For example, this can happen in a denial-of-service attack. The account-wide limit is 100,000 requests per minute.

3

I recommend reading documentation https://developer.zendesk.com/api-reference/introduction/rate-limits/

> If the rate limit is exceeded, the API responds with a <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429|429 Too Many Requests> status code. The response also has a Retry-After header that tells you how many seconds to wait before retrying API requests. Ensure your code handles 429 errors and waits the Retry-After interval before retrying requests.
>
> For tips on avoiding and handling 429 errors, see <https://developer.zendesk.com/documentation/ticketing/using-the-zendesk-api/best-practices-for-avoiding-rate-limiting/|Best practices for avoiding rate limiting>.
> Export Search Results GET /api/v2/search/export?query={query} 100 requests per minute per account
> Zendesk might limit requests if it detects an unusual spike in requests from all sources for the account, including internal product requests. For example, this can happen in a denial-of-service attack. The account-wide limit is 100,000 requests per minute.