Summary
The user is looking for ways to pro-actively limit the number of API requests in a connector to avoid reaching the hard rate limit. They have found information on rate limiting in the Airbyte documentation and a discussion thread, but are unsure how to implement it effectively.
Question
Hi everyone,
I am working on a new connector for an API with hard rate limit at 100 request per minute. I would like for this connector to limit the number of request pro-actively to avoid api shutdown which happen when reaching the limit.
It seems not possible at the moment - https://docs.airbyte.com/connector-development/cdk-python/http-streams#rate-limiting only dealing with backoff when errors occurs.
However, someone in a discussion tried to had a sleep read_records or in send function - https://discuss.airbyte.io/t/how-do-you-correctly-modify-a-connector-to-adhere-to-rate-limits/1754
It is not very clear for me how to do it and whether it is a good idea or not.
Do you have an opinion on this matter? Do you other way to perform this ? Do you have any tips for me ?
Thank you for your help and have a good day,
Olivier
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.