Implementing a Stateful Connector for Bing Webmaster API in Airbyte

Summary

The user is looking for guidance on implementing a stateful connector for the Bing Webmaster API in Airbyte to handle the response mapping to the corresponding request tuples.


Question

Hi, I have a little special requirements and wonder if I can realize them using Airbyte. To import the PageQueryStats from Bing Webmaster API I need to:

  1. iterate over a list tuples (page, siteUrl) and send a request for each tuple. The Bing API replies with a list of metrics for this tuple – but it does not return the input tuple.
  2. then, for each response, add the (page, siteUrl) tuple to the response tuple so that we know where these metrics belong to.
    However, during experiments I observed that next_page_token and request_params are called multiple times before parse_response is called. If there was one call to parse_response after each request then it would be easy to keep track of which response belongs to which request. However, because there are multiple requests until the first parse_response is invoked I see no possibility of adding the right (page, siteUrl) pair to the response.

Do you have any idea how to implement such a stateful Airbyte connector?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["stateful-connector", "bing-webmaster-api", "airbyte", "response-mapping"]