To make multiple API requests in the same stream, you can use the Airbyte’s CDK (Connector Development Kit) which provides the HTTPStream
class. This class allows you to define the base URL and the path for your API requests, as well as how to parse the resulting response. Here’s a brief overview of how you can set it up: 1. Define the base URL via the url_base
property. 2. Implement the abstract path
function to set the path for your API request. 3. Implement the parse_response
function to instruct the stream how to parse the API response. This function should return an Iterable
, whose elements are each later transformed into an AirbyteRecordMessage
. For example, if you need to call the API four times, you can set up four different paths or modify the parameters of your API request in the path
function. If you need to make requests based on the response of a previous request, you might want to look into nested streams and caching as suggested in a forum post. For more detailed information, you can refer to the HTTP Streams section in the Airbyte’s CDK documentation. Please note that the actual implementation might vary depending on the specifics of the API you are working with.