Summary
Custom connector to Chartmetric API encounters 429 rate limit issue despite implementing Backoff Strategy with Wait time and Retry-After headers. Connector stops working after 5 attempts.
Question
hi guys, has anyone build custom connector to Chartmetric API? i create one for myself and it works perfect while testing but now i struggle with 429 and rate limit exceed. i add Backoff Strategy using Wait time from header and Retry-After but after 5 five attempts connector has final error and stop working
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["custom-connector", "chartmetric-api", "rate-limit", "backoff-strategy", "retry-after"]
Hi there, Rate limiting is a common issue when dealing with APIs. In Airbyte, the most common way to handle rate limits is to implement backoff. This means your connector should gracefully handle rate limits without failing the connector process source. You mentioned that you’ve implemented a Backoff Strategy using “Wait time from header” and “Retry-After”. This is a good approach as it allows the connector to wait before retrying a request based on the value specified in the API response source. However, if you’re still encountering issues, it’s possible that there are other factors to consider when trying to slow down a connector. For instance, a user on the Airbyte forum mentioned that their connector never received a 429 or hit exponential backoff when testing locally, but when run from a locally-hosted Airbyte instance, it hit backoffs regularly source. If you’re still encountering rate limit issues that are not automatically retried successfully, you can create an issue on the Airbyte Github page source. Also, please note that in some cases, like with the LinkedIn Ads connector, after 5 unsuccessful attempts, the connector will stop the sync operation source. I hope this helps! If you have any other questions, feel free to ask.
The Backoff strategy is working and you want to change the retries or is not working?