Issue with 'discover' method in low-code connector implementation

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is facing an issue where the ‘discover’ method in their low-code connector implementation is not displaying any streams in the catalog, despite ‘spec’, ‘check’, and ‘read’ methods working as expected. They are implementing OAuth2 authentication with ‘client_credentials’ grant type.


Question

hello :wave:
I am developing a low-code connector, with 1 stream defined in my manifest.yaml and implementing OAuth2 authentication with client_credentials grant type.
Both spec, check and read properly works as expected (<https://docs.airbyte.com/connector-development/cdk-python/basic-concepts|defined here>)… but the discover does not work, as it displays no stream:
{"type":"CATALOG","catalog":{"streams":[]}}
Do you have an idea on how to fix it ? As I can not use my connector currently (no stream to enable in connection :disappointed: )
Thanks a lot !


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["low-code-connector", "manifest.yaml", "OAuth2", "client-credentials", "discover-method", "streams"]
2

My manifest.yaml :


type: DeclarativeSource

check:
  type: CheckStream
  stream_names:
    - integrations

definitions:
  base_requester:
    type: HttpRequester
    url_base: '{{ config["api_base_url"] }}'
    authenticator:
      type: OAuthAuthenticator
      client_id: '{{ config["client_id"] }}'
      client_secret: '{{ config["client_secret"] }}'
      grant_type: client_credentials
      token_refresh_endpoint: '{{ config["token_refresh_endpoint"] }}'
      scopes:
        - aggregator:read

  streams:
    integrations:
      type: DeclarativeStream
      name: integrations
      primary_key:
        - Id
      retriever:
        type: SimpleRetriever
        requester:
          $ref: '#/definitions/base_requester'
          path: /integrations
          http_method: GET
        record_selector:
          type: RecordSelector
          extractor:
            type: DpathExtractor
            field_path: []
      schema_loader:
        type: InlineSchemaLoader
        schema:
          $ref: '#/schemas/integrations'

streams:
  - $ref: '#/definitions/streams/integrations'

spec:
  type: Spec
  connection_specification:
    type: object
    $schema: <http://json-schema.org/draft-07/schema#>
    required:
      - api_base_url
      - client_id
      - client_secret
      - token_refresh_endpoint
    properties:
      api_base_url:
        type: string
        title: API Base URL
        description: May be different in DEV and PROD environments
        default: <https://my-default-url>
        order: 0
      client_id:
        type: string
        order: 1
        title: Client ID
        airbyte_secret: true
      client_secret:
        type: string
        order: 2
        title: Client secret
        airbyte_secret: true
      token_refresh_endpoint:
        type: string
        title: Token refresh endpoint
        description: May be different in DEV and PROD environments
        default: <https://my-token-refresh-url>
        order: 3
    additionalProperties: true

metadata:
  autoImportSchema:
    integrations: true
  yamlComponents:
    global:
      - authenticator
  testedStreams:
    integrations:
      streamHash: 4752d1d4d214744323ece3edc301afbd76d5ceee
      hasResponse: true
      responsesAreSuccessful: true
      hasRecords: true
      primaryKeysArePresent: true
      primaryKeysAreUnique: true

schemas:
  integrations:
    type: object
    $schema: <http://json-schema.org/schema#>
    additionalProperties: true
    required:
      - Id
    properties:
      ConnectionType:
        type:
          - number
          - 'null'
      DisplayName:
        type:
          - string
          - 'null'
      Id:
        type: string
      Name:
        type:
          - string
          - 'null'
      Protocol:
        type:
          - number
          - 'null'```
3

If it can help, when I add dummy in my check to force an error:

  type: CheckStream
  stream_names:
    - integrations
    - dummy```
=> the `check` now fails with error:
```Expected one of dict_keys(['integrations'])```
... but no change on `discover` , still no stream:
```{"type":"CATALOG","catalog":{"streams":[]}}```
4

It works if I add many Python dependencies (run.py, source.py, main.py, etc) but I would like to make it work with no Python at all :pray: :thinking_face:

5

:exploding_head: really painful as it is impossible to debug !

… but finally identified the root cause by chance:

the metadata.yaml had :

  baseImage: docker.io/airbyte/source-declarative-manifest:6.1.0```
I changed to:
```connectorBuildOptions:
  baseImage: docker.io/airbyte/source-declarative-manifest:6.5.2```
and it works! :fire: