I am building out a connector to serve the “ad stats” endpoint from the Snapchat API (the current connector does not include this). To fetch stats for all ads associated with a given organization, I need to return responses from a list of endpoints, iterating over a set of ad_ids.
For example we might have
ad_ids = [‘foo’,‘bar’,‘baz’]
And a URL constructed like
So we would want to get data for
I am familiar with the design pattern of having classes returning a single endpoint with pagination logic, but I’ve not yet implemented something like this. Would appreciate any pointers!
EDIT TO ADD: It looks like I want to use something like this, but I can’t find any examples of existing sources and am having trouble implementing it: HTTP-API-based Connectors | Airbyte Documentation
Are you getting the
ad_ids dynamically from an endpoint or are you going to give by the connector spec.json what
ad_ids you want to retrieve?
ad_id values come dynamically from a different endpoint: Snapchat Marketing API Reference
(to begin with, we will probably provide ad_account_id via the source config, but that might be fetched dynamically later as well.)
Oh! I had found the HttpSubStream but from the example in the documentation I was not able to figure out how to implement it. But the example you shared is helpful, I’ll give it another try!
Is the caching bit essential for what I need to do? That was one thing I found confusing in the HttpSubStream documentation (HTTP-API-based Connectors | Airbyte Documentation).
Thanks so much for your help figuring this out!
It’s not essential but helps a lot because with cache because your substream need to get info from a read records from the parent one. Let me know if you need any further assistance.