Troubleshooting Parent Stream Dependency in Connector Development

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

The user is developing a connector using the Low-code connector development framework and is facing an issue with a stream that depends on a parent stream. The connector is making a call to the endpoint with an incorrect placeholder value. The user needs assistance in identifying what is wrong or missing in the manifest.yaml file.


Question

Hello everyone,
I am looking into developing a connector using the <https://docs.airbyte.com/connector-development/config-based/low-code-cdk-overview|Low-code connector development> framework. I am stuck in the section where I have a stream that depends on a parent stream. Here is the extract from my manifest.yaml file:


definitions:
  selector:
    type: RecordSelector
    extractor:
      type: DpathExtractor
      field_path: []

  requester:
    type: HttpRequester
    url_base: "<https://customapi.goldcast.io>"
    http_method: "GET"
    authenticator:
      type: ApiKeyAuthenticator
      header: "Authorization"
      api_token: "Token {{ config['access_key'] }}"

  retriever:
    type: SimpleRetriever
    record_selector:
      $ref: "#/definitions/selector"
    paginator:
      type: NoPagination
    requester:
      $ref: "#/definitions/requester"

  base_stream:
    type: DeclarativeStream
    retriever:
      $ref: "#/definitions/retriever"

  events_stream:
    $ref: "#/definitions/base_stream"
    name: "events"
    description: "Events stream. Events are the base object of Goldcast."
    primary_key: "id"
    $parameters:
      path: "/event/"

  webinars_stream:
    $ref: "#/definitions/base_stream"
    name: "webinars"
    description: "Webinars stream. Webinares are one of the objects that compose an event."
    primary_key: "id"
    parent_stream_configs:
      - type: ParentStreamConfig
        stream: "#/definitions/events_stream"
        parent_key: "id"
        partition_field: "id"
    $parameters:
      name: "webinars"
      primary_key: "id"
      path: "/event/webinars/{{ stream_slice.id }}"
    transformations:
      - type: AddFields
        fields:
          - path: ["event_id"]
            value: "'{{ stream_slice.id }}'"
    retriever:
      $ref: "#/definitions/retriever"
      record_selector:
        $ref: "#/definitions/retriever/record_selector"
        extractor:
          field_path: ["id", "id"]
      requester:
        $ref: "#/definitions/requester"
        error_handler:
          type: DefaultErrorHandler
          description: "405 - cMethod \"GET\" not allowed."
          response_filters:
            - http_codes: [405]
              action: IGNORE

streams:
  - "#/definitions/events_stream"

check:
  type: CheckStream
  stream_names:
    - "events"

spec: 
  documentation_url: <https://docs.airbyte.com/integrations/sources/exchangeratesapi>
  connection_specification:
    $schema: <http://json-schema.org/draft-07/schema#>
    title: goldcast.io Source Spec
    type: object
    required:
      - access_key
    additionalProperties: true
    properties:
      access_key:
        type: string
        description: >-
          Your API Access Key. See <a
          href="<https://help.goldcast.io/hc/en-us/articles/22931655725723-How-To-Create-an-API-Token-in-Goldcast>">here</a>. The key is
          case sensitive.
        airbyte_secret: true```
As I hope you can understand from my yaml file :smile:, for each record of the events stream, I want to make a call to the webinars stream to the path `"/event/webinars/{{ stream_slice.id }}"`
where `{{ stream_slice.id }}`  is the id of the event (primary key of the events stream). Could you please tell me what is wrong / missing in my file, currently the connector tries to call the endpoint `"/event/webinars/"`  which is not allowed. I am guessing it because the placeholder `{{ stream_slice.id }}`  is not correct ?
Many thanks in advance for your help :pray:

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1713186034372479) if you want to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["connector-builder", "low-code-connector-development", "parent-stream-dependency", "manifest.yaml", "stream-slice.id", "endpoint-call"]
</sub>
2

I think you could try this:

            - type: ParentStreamConfig
              stream: "#/definitions/events_stream"
              parent_key: id
              partition_field: webinar_id```
and then use `{{ stream_partition.webinar_id }}`  in the webinars url
3

Hello <@U06PR044WBA>, thank you for your suggestion. Tried this and got the same result. It seems that {{ stream_partition.webinar_id }} is not giving any value as airbyte is making a GET request on "/event/webinars/"

4

Found the issue. Adding the following configuration for the webinars stream works:

    $ref: "#/definitions/base_stream"
    name: webinars
    description: "Webinars stream. Information on the webinars of events."
    primary_key: "id"
    $parameters:
      partition_router:
        - type: SubstreamPartitionRouter
          parent_stream_configs:
            - type: ParentStreamConfig
              parent_key: id
              partition_field: event
              stream:
                $ref: '#/definitions/events_stream'
      path: "/event/webinars/{{ stream_partition.event }}"```
The issue was that I was using stream_slice and not stream_partition I beleive.
FYI - <@U06PR044WBA>, <@U05FB419LMR>