Cross-sync generic Rate Limiting

Hello!

For a specific partner API, I need to enforce very strict daily rate limits (no more than X calls for a specific day, across retries ) and report on these in our UI.

The API won’t enforce this for us as this limit is set by our customer to prevent our usage impacting other usages (the rate limit is shared by different services).

My first instinct was to store the number of API calls done in the connector’s state, but I’m running into two issues:

  • The protocol defines that the state is only updated when the destination acknowledges (by outputting it) the state message. This means that in case of errors, the state would not be updated and the rate limit not properly enforced. This also complexifies the state a lot.
  • I need to consume the state outside of Airbyte (in our UI), which would not be generic at all since the state of this rate limit would not be defined in the protocol.

I feel like Airbyte could create a generic service to handle complex rate limits, store their state and report back on their usage.

In addition to providing an useful feature to all connectors, this would also improve the overall visibility on the connectors themselves.

I’m interested in Airbyte’s view on this topic:

  • if you feel like this is not something that should be part of airbyte’s core, how would you go about implementing that externally?
  • if you feel like it should be part of the core, I would be willing to contribute to its development either in the core itself, or externally first so we can test it for our usages and later give you feedback on it!

A quick example similar to what we’re trying to do (in HubSpot when you configure their Salesforce integration):

Thank you @cptjacky for this great feedback and interesting use case.
You are right about the current state protocol.
I understand your monitoring need that is not yet met with Airbyte, but could you please detail if you only want to report on the number of requests made by connectors or if you also want to make the connector behave according to this metric?

The approach Airbyte has taken for rate-limit management at the moment is to switch to exponential backup strategies mode once rate limit are hit. It is not pacing the requests according to a pre-defining number of requests per minute or a hard limit such as the one you have for your use case. In other words, a connector requests a source as fast as possible until an error response (429) from the source.

IMO the CDK could expose an object to the connector with global metrics such as the one you need, and we could choose to expose these metrics in the API at the source level and job level for monitoring purposes.

The workaround I could suggest at the moment is to override the HttpStream.read_records method and implement your own logic around rate limiting there. There might be something to do with our optional cache mechanism. By reading the cache you could be aware of the number of requests sent to the source. You are also free to use whatever external state storage that might work for you, but I understand it’s not an ideal solution.

I’ll reach out to our connector team to give you more details if this is something in the roadmap and what might be their suggestion for this kind of problem.

Thank you also for the quick answer!

could you please detail if you only want to report on the number of requests made by connectors or if you also want to make the connector behave according to this metric?

We are indeed trying to make the connector behave according to this metric!
I want to make sure we never go over that limit (and potentially multiple limits: daily, hourly, …), which means potentially stopping the sync early if we reach the daily limit (and possibly loose some data if we can’t checkpoint correctly), or more simply letting it sleep (I’d consider this non-ideal for our use case, but each connector might work differently).

IMO the CDK could expose an object to the connector with global metrics such as the one you need, and we could choose to expose these metrics in the API at the source level and job level for monitoring purposes.

This is indeed what I personally think would be the best solution, but this involves correctly defining the protocol and datastore depending on the exact needs (the ideal datastore wouldn’t be the same if we wanted to allow rate limiting at the second or minute level or if we only need daily rate limits for example).

The workaround I could suggest at the moment is to override the HttpStream.read_records method and implement your own logic around rate limiting there. There might be something to do with our optional cache mechanism. By reading the cache you could be aware of the number of requests sent to the source. You are also free to use whatever external state storage that might work for you, but I understand it’s not an ideal solution.

This is indeed the workaround I was imagining here! Basically defining our own service to handle this, and extend the CDK for our use (in the more generic way possible, as we’ll likely need it for several connectors).
But this is not ideal long-term and I would love to collaborate and contribute toward a long-term solution that would improve Airbyte’s capabilities!

I’ll reach out to our connector team to give you more details if this is something in the roadmap and what might be their suggestion for this kind of problem.

Thank you very much!