Client-side throttle/quota/rate limit for Salesforce source

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Is there a way to apply a client-side throttle/quota/rate limit for a Salesforce source in Airbyte? Seeking workarounds to prevent exceeding API quota and potentially disrupting client’s production flows.


Question

Is there any way at all to apply a client side throttle / quota / rate limit for a source like Salesforce? https://airbytehq.slack.com/archives/C01AHCD885S/p1710269903380649 I see some github issues around the topic that indicate it’s not support currently but curious if anyone has any workarounds. Context: I am intending to use the connector to ingest data from my customers salesforce instances with the ‘Powered By’ setup. In this scenario, my tool exhausting the clients salesforce rate limits and breaking their production flows is a worst case scenario and would destroy credibility / relationship, so I need a rock solid way to ensure that my tools API usage never exceeds a pre-agreed on % of the API quota. Would appreciate input from anyone with experience or thoughts on how to approach this. Whether or not I can find a way to satisfy this requirement is likely a make-or-break for me for using Airbyte for this use case.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["client-side-throttle", "quota", "rate-limit", "salesforce-connector", "api-usage", "workarounds"]
2

After a bit of digging I see that the singer tap for salesforce specifically has a config for quota percentage which I can probably use for my purpose https://github.com/search?q=repo%3Aairbytehq%2Ftap-salesforce%20quota_percent_total&type=code

But it isn’t mentioned in the Airbyte docs so unclear if it can actually be configured. I will have to test I guess. https://docs.airbyte.com/integrations/sources/salesforce#reference

A generalized solution that applies to any connector would still be good

3

I agree, rate limiting should be a standard part of airbyte