Handling Trackonomics API response in Custom Connector Builder

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

When building a Custom Connector for Trackonomics API, the response includes a request_id that needs to be used in a subsequent API call to get the actual data. This is likely to allow time for data generation. How can the Connector Builder be configured to handle this?


Question

Hi, I’m trying to build a Custom Connector to hit Trackonomics’ API. I’m able to authenticate and create a stream. The challenge I have with this API is that the response is not the data I want, but rather a request_id which I can then use in a subsequent API call to get the actual results of the data. I guess this is to allow the endpoint time to generate the data. How do I use the builder to handle this configuration?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["custom-connector", "trackonomics-api", "response-handling", "connector-builder", "subsequent-api-call"]
2

New message text here

3

<@U04P92HCXPF> Check out Parent Stream for your use case: https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#parent-stream

4

<@U02T7NVJ6A3> that article was very helpful - thank you. I was able to
• create a stream that passes in all the parameters for the API which returns a request_id
• create a 2nd stream that calls the first as a partitioned substream (aka Parent)

5

I now need to be able to paginate and increment.
For pagination I won’t know if there is a need to paginate the parent stream until I get the result from the child stream. Here is an example output from the child stream:

    "total": 10000,
    "recordsTotal": 37849,
    "recordsFiltered": 37849,
    "pageNum": 1,
    "numOfPages": 4```
I'm trying to figure out where to pass in the incremental page numbering. Is Airbyte able to read the numOfPages and understand to add `page=#` depending on its iteration?
6

How you need to pass pagination parameters into requests is dependent on your API, but you should be able to just set a page number and page limit on your requests, and the connector builder will automatically stop making new requests when it receives a response with fewer records than the page limit you’ve configured.

See the pagination page on the docs for more details: https://docs.airbyte.com/connector-development/connector-builder-ui/pagination

7

Do I set the pagination information on the parent or child connector?

8

You can set it on whichever stream is calling an endpoint that accepts pagination inputs