How to add delay between API hits in a connector for better rate limiting and API usage management.
can someone please tell me how can i add delay between api hits in a connector, need it urgently please help
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
To handle rate limits in your connector, you should implement a backoff mechanism. This is a common way to gracefully handle APIs that enforce rate limits, ensuring that your connector process does not fail. Unfortunately, the provided extracts do not contain specific instructions or code examples on how to implement backoff in an Airbyte connector. However, backoff generally involves adding a delay between API requests, which increases after each failed request, up to a maximum limit. For more detailed instructions, you may want to refer to the Airbyte documentation or other resources on implementing backoff mechanisms in your specific programming language or framework.
I am being blocked by zoho for around 40 mins because i exceed 100 requests per minute. I just need to keep the count below 100 that’s all.
<@U04SE163WC8> I am being blocked by zoho for around 40 mins because i exceed 100 requests per minute. I just need to keep the count below 100 that’s all.
It seems like you’re hitting the rate limits set by Zoho’s API. According to the Zoho CRM documentation, each Zoho CRM edition has a limit in a 24-hour rolling window. To avoid hitting these rate limits, you need to manage your API calls in a way that they don’t exceed the limit. Unfortunately, the extracts provided do not contain specific information on how to implement this in Airbyte. However, as a general best practice, Airbyte recommends handling rate limits gracefully, often through implementing backoff as mentioned in the Airbyte’s connector development best practices. If you’re still encountering issues, it might be helpful to raise an issue in the Airbyte Github repository for further assistance.