Iterate across multiple endpoints? (e.g. for a set of ad ids)

I am building out a connector to serve the “ad stats” endpoint from the Snapchat API (the current connector does not include this). To fetch stats for all ads associated with a given organization, I need to return responses from a list of endpoints, iterating over a set of ad_ids.

For example we might have

ad_ids = [‘foo’,‘bar’,‘baz’]

And a URL constructed like <url_base>/<ad_id>/<query parameters>

So we would want to get data for
<url_base>/foo/<query parameters>
<url_base>/bar/<query parameters>
<url_base>/baz/<query parameters>

I am familiar with the design pattern of having classes returning a single endpoint with pagination logic, but I’ve not yet implemented something like this. Would appreciate any pointers!

EDIT TO ADD: It looks like I want to use something like this, but I can’t find any examples of existing sources and am having trouble implementing it: HTTP-API-based Connectors | Airbyte Documentation

Are you getting the ad_ids dynamically from an endpoint or are you going to give by the connector spec.json what ad_ids you want to retrieve?

Hi Marcos! ad_id values come dynamically from a different endpoint: Snapchat Marketing API Reference{ad_account_id}/ads

(to begin with, we will probably provide ad_account_id via the source config, but that might be fetched dynamically later as well.)

What about using SubHttpStream? See here:

See one implementation here: is missing the cache part but gives you the idea.

Oh! I had found the HttpSubStream but from the example in the documentation I was not able to figure out how to implement it. But the example you shared is helpful, I’ll give it another try!

Is the caching bit essential for what I need to do? That was one thing I found confusing in the HttpSubStream documentation (HTTP-API-based Connectors | Airbyte Documentation).

Thanks so much for your help figuring this out!

It’s not essential but helps a lot because with cache because your substream need to get info from a read records from the parent one. Let me know if you need any further assistance.

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.