Pro-actively Limiting API Requests in Connector


The user is looking for ways to pro-actively limit the number of API requests in a connector to avoid reaching the hard rate limit. They have found information on rate limiting in the Airbyte documentation and a discussion thread, but are unsure how to implement it effectively.


Hi everyone,
I am working on a new connector for an API with hard rate limit at 100 request per minute. I would like for this connector to limit the number of request pro-actively to avoid api shutdown which happen when reaching the limit.
It seems not possible at the moment - only dealing with backoff when errors occurs.
However, someone in a discussion tried to had a sleep read_records or in send function -
It is not very clear for me how to do it and whether it is a good idea or not.
Do you have an opinion on this matter? Do you other way to perform this ? Do you have any tips for me ?
Thank you for your help and have a good day,

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["api-connector", "rate-limit", "pro-active-limiting", "connector-development", "rate-limiting-strategy"]