Summary
How to add unique URL parameters for each stream in a YAML based connector to specify which fields should be returned with the stream
Question
what should be simple and likely is, I just can’t figure out how to do it: YAML based connector, each stream needs some unique URL parameters set, specifically which fields should come back with the stream (e.g., fields=id,address,annualRevenue,billingAddress,…) - how do I get this added to each stream uniquely? Each stream has a different list of fields that need to be updated in URL.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["yaml-based-connector", "unique-url-parameters", "stream-fields"]
To add unique URL parameters to each stream in a YAML-based connector, you can use the “Data Retriever” component of the stream configuration. Specifically, the “Requester” sub-component allows you to define how to prepare HTTP requests to send to the source API, including the base URL and path, request options provider, HTTP method, authenticator, and error handler components. If the API has special requirements that can’t be handled by the standard “Inject Parameter / Parent Key Value into outgoing HTTP Request” option, you can disable injection in the component and use the generic parameter section at the bottom of the stream configuration form. Here, you can freely configure query parameters, headers, and properties of the JSON body using jinja expressions and available variables. This would allow you to add unique fields to the URL parameters for each stream. Here’s an example of how you might configure this for a stream: yaml streams: - name: stream_name ... retriever: requester: base_url: <https://api.example.com> path: /endpoint http_method: GET authenticator: ... error_handler: ... request_options_provider: query_parameters: fields: id,address,annualRevenue,billingAddress,... In this example, replace stream_name with the name of your stream, and replace the fields value with the list of fields you want to retrieve for that stream. Repeat this configuration for each stream, adjusting the fields value as needed. For more details, refer to the Understanding the YAML file and Low-code connector development sections of the Airbyte documentation.
Since each stream defines its own requester, you should be able to set the required request parameters specific to each stream in each stream’s own requester.request_options_provider component
This is what the relevant part of my streams block looks like and it’s not adding the url params… am I doing it wrong…
$ref: "#/definitions/base_stream"
name: "client_corporation"
requester:
type: CustomRequester
class_name: "source_bullhorn_ats.http.BhHttpRequester"
request_options_provider:
query_parameters:
fields: id,address,annualRevenue,billingAddress
primary_key: "id"
$parameters:
path: "/query/ClientCorporation"```
definitions:
selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path: []
requester:
type: CustomRequester
class_name: "source_bullhorn_ats.http.BhHttpRequester"
url_base: "<http://it.does.not.matter.com|it.does.not.matter.com>"
http_method: "GET"
authenticator:
type: CustomAuthenticator
class_name: "source_bullhorn_ats.auth.BullhornAuthenticator"
username: "{{ config['username'] }}"
password: "{{ config['password'] }}"
client_id: "{{ config['client_id'] }}"
client_secret: "{{ config['client_secret'] }}"
retriever:
type: SimpleRetriever
record_selector:
$ref: "#/definitions/selector"
paginator:
type: NoPagination
requester:
$ref: "#/definitions/requester"
base_stream:
type: DeclarativeStream
retriever:
$ref: "#/definitions/retriever"
client_corporation_stream:
$ref: "#/definitions/base_stream"
name: "client_corporation"
requester:
type: CustomRequester
class_name: "source_bullhorn_ats.http.BhHttpRequester"
request_options_provider:
query_parameters:
fields: id,address,annualRevenue,billingAddress
primary_key: "id"
$parameters:
path: "/query/ClientCorporation"
streams:
- "#/definitions/client_corporation_stream"
check:
type: CheckStream
stream_names:
- "client_corporation"
spec:
type: Spec
documentation_url: <https://docs.airbyte.com/integrations/sources/bullhorn-ats>
connection_specification:
title: Bullhorn ATS Spec
type: object
required:
- username
- password
- client_id
- client_secret
additionalProperties: true
properties:
username:
type: string
description: >-
Your API Access Key. See <a
href="<https://exchangeratesapi.io/documentation/>">here</a>. The key is
case sensitive.
airbyte_secret: false
password:
type: string
description: >-
Your API Access Key. See <a
href="<https://exchangeratesapi.io/documentation/>">here</a>. The key is
case sensitive.
airbyte_secret: true
client_id:
type: string
description: >-
Your API Access Key. See <a
href="<https://exchangeratesapi.io/documentation/>">here</a>. The key is
case sensitive.
airbyte_secret: false
client_secret:
type: string
description: >-
Your API Access Key. See <a
href="<https://exchangeratesapi.io/documentation/>">here</a>. The key is
case sensitive.
airbyte_secret: true ```
It looks like you’re using a CustomRequester, so your custom requester class would need to explicitly set the request parameters in its initialization function
oh, ok… not explicitly right? as in, I can still control that in the stream level, right?
if so, what am I setting?
sorry, daft question, I’m sure
Your custom component can still take in arguments from the yaml manifest