Slow performance with nested API calls in connector builder for Zoom

Summary

Nested API calls in connector builder for Zoom are causing slow performance, specifically with extracting data for meetings.


Question

Why does nested API calls are extremely slow I am trying to build a connector using UI builder for zoom, the first-level data for Users is done in 3 minutes. But extracting data for meetings is taking forever



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["nested-api-calls", "connector-builder", "zoom", "slow-performance", "meetings"]

Based on the information provided in the extracts, there are a few considerations that might help address the issue of slow performance when extracting data for meetings using a connector built with the Airbyte UI builder for Zoom. 1. Zoom API Rate Limits: The performance of connectors, especially for services like Zoom, can be significantly affected by the API’s rate limits. The Zoom connector accesses endpoints that are restricted by standard Zoom requests limitation. If the connector hits the rate-limiting threshold, it has been designed to retry appropriately. This could be one reason for the slow performance when making nested API calls, as the rate limits might be more easily reached with more complex data extraction like that for meetings. For more details on Zoom’s rate limits, you can check their API documentation. Extract from the source: “Most of the endpoints this connector access is restricted by standard Zoom requests limitation, with a few exceptions. For more info, please check zoom API documentation. We’ve added appropriate retries if we hit the rate-limiting threshold.” 2. Feedback and Issues: If you are experiencing rate limit issues that are not automatically retried successfully or if the performance issue persists, it is recommended to create an issue on the Airbyte GitHub. This way, the Airbyte team can investigate and potentially provide solutions or optimizations for the connector. Extract from the source: “Please create an issue if you see any rate limit issues that are not automatically retried successfully.” 3. Investigating Latency Issues: While there isn’t a direct extract addressing the investigation of latency issues for the Zoom connector specifically, there is a general approach mentioned for investigating latency issues with the connector builder. Questions like identifying the root cause and adding instrumentation to understand where time is spent are part of the investigation process. Applying a similar approach to your situation might help in diagnosing the slow performance for nested API calls. Extract from the source: “- What is the root cause? - Can we add instrumentation to understand where we spend time?” Unfortunately, without more specific details or extracts related to the technical aspects of handling nested API calls or optimizing performance for such scenarios in the Airbyte UI builder, this is the extent of the guidance that can be provided based on the provided extracts. If the issue persists, consider reaching out directly to Airbyte support or the community for more targeted assistance.

How many users are there, roughly? Nested streams will make an http request for each user, and if there’s a set backoff on rate limit, that could take a lot of time.

<@U069EMNRPA4> - It has 17k users and I am pulling data for 2 day range

I am assuming with the rate limit of zoom 60req per sec 17k users * 3 which will be max API calls if I user has 600 meetings. which should take 14 minutes to complete the calls without back off issues

but its simply not completing anytime

users are extracted in 3 minutes which is 5 api calls

I am curious to see what is causing this delay is there a way to see which call is sent and how much time it took