Iterate across multiple endpoints? (e.g. for a set of ad ids)

Connector Development
1

I am building out a connector to serve the “ad stats” endpoint from the Snapchat API (the current connector does not include this). To fetch stats for all ads associated with a given organization, I need to return responses from a list of endpoints, iterating over a set of ad_ids.

For example we might have

ad_ids = [‘foo’,‘bar’,‘baz’]

And a URL constructed like <url_base>/<ad_id>/<query parameters>

So we would want to get data for
<url_base>/foo/<query parameters>
<url_base>/bar/<query parameters>
<url_base>/baz/<query parameters>

I am familiar with the design pattern of having classes returning a single endpoint with pagination logic, but I’ve not yet implemented something like this. Would appreciate any pointers!

EDIT TO ADD: It looks like I want to use something like this, but I can’t find any examples of existing sources and am having trouble implementing it: HTTP-API-based Connectors | Airbyte Documentation

2

Are you getting the ad_ids dynamically from an endpoint or are you going to give by the connector spec.json what ad_ids you want to retrieve?

3

Hi Marcos! ad_id values come dynamically from a different endpoint: Snapchat Marketing API Reference

https://adsapi.snapchat.com/v1/adaccounts/{ad_account_id}/ads

(to begin with, we will probably provide ad_account_id via the source config, but that might be fetched dynamically later as well.)

4

What about using SubHttpStream? See here: https://docs.airbyte.com/connector-development/cdk-python/http-streams

See one implementation here: https://github.com/airbytehq/airbyte/blob/86cae0028d070f16254884e78e04f421ddd96c3e/airbyte-integrations/connectors/source-kyriba/source_kyriba/source.py#L123-L152 is missing the cache part but gives you the idea.

5

Oh! I had found the HttpSubStream but from the example in the documentation I was not able to figure out how to implement it. But the example you shared is helpful, I’ll give it another try!

Is the caching bit essential for what I need to do? That was one thing I found confusing in the HttpSubStream documentation (HTTP-API-based Connectors | Airbyte Documentation).

Thanks so much for your help figuring this out!

6

It’s not essential but helps a lot because with cache because your substream need to get info from a read records from the parent one. Let me know if you need any further assistance.

7

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.