Passing array for one of the field in spec.yaml file

I have a requirement where I need to pull multiple data from the api. Right now what are doing we have created a custom connector as a source where I’m pulling some data which is schema less and we transform the data into JSON and dump it into the Postgres. I have a spec file which looks like this

  title: Surveycto Spec
  type: object
    - server_name
    - username
    - password
    - form_id
      type: string
      title: Server Name
      description: The name of the SurveryCTO server
      order: 0
      type: string
      title: Username
      order: 1
      type: string
      title: Password
      airbyte_secret: true
      order: 2
      type:  string
      title: Form ID
      order: 3
class SoStream(HttpStream, ABC):
    primary_key = None

    def __init__(self, config: Mapping[str, Any], **kwargs):
        self.server_name = config['server_name']
        self.form_id = config['form_id']
        #base64 encode username and password as auth token
        user_name_password = f"{config['username']}:{config['password']}"
        self.auth_token = self._base64_encode(user_name_password)
    def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
        return None

    def _base64_encode(self,string:str) -> str:
        return base64.b64encode(string.encode("ascii")).decode("ascii")

    def url_base(self) -> str:
        return f"https://{self.server_name}"

    def request_params(
        self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, any] = None, next_page_token: Mapping[str, Any] = None
    ) -> MutableMapping[str, Any]:
        return {'date': 0}

    def request_headers(
        self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
    ) -> Mapping[str, Any]:
        return {'Authorization': 'Basic ' + self.auth_token }

    def parse_response(
        response: requests.Response,
        stream_state: Mapping[str, Any],
        stream_slice: Mapping[str, Any] = None,
        next_page_token: Mapping[str, Any] = None,
    ) -> Iterable[Mapping]:
        response_json = response.json()

        for data in response_json:
                yield data
            except Exception as e:
                msg = f"""Encountered an exception parsing schema"""
                raise e
    def path(
        stream_state: Mapping[str, Any] = None, 
        stream_slice: Mapping[str, Any] = None, 
        next_page_token: Mapping[str, Any] = None
    ) -> str:
        return self.form_id

Right now it pulls data from the particular form_id which is string now. I need to pass multiple form_id in the spec file. Which can be an array but when I change it to an array and run the code again its giving me this error

Config validation error: ‘test_data’ is not of type ‘array’

how can I pass multiple form IDs here and what changes should I make so that I can pull data from different form_id?

Hey @siddhant3030! Could you give me a bit more info - what is ‘test_data’, it’s in a separate file, right? Could you provide that here as well? I think the bug might be there.

Test data is the name of the form_id from where I’m calling this API basically

This will give me a JSON response which is a list. But in order to do that I was passing as a string basically

      type:  string
      title: Form ID
      order: 3

It pulls data when the form_id is a string.

Now what I want to do is basically pass multiple form_id which will be an array
and then somehow it can call the API for each form_id and give a list of that response

[“test_data”, “test_data1”]

Still looking into this, but I’ve found a thread that might be helpful to you where a user was trying to do a similar thing:

The solution implements stream slices:

I did implement this. But somehow I’m still getting the same error

2022-10-06 19:31:36 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):99 - Config validation error: 'test_data' is not of type 'array'
Traceback (most recent call last):
  File "/airbyte/integration_code/", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/", line 123, in launch
    for message in
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/", line 96, in run
    check_config_against_spec_or_exit(connector_config, source_spec)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/utils/", line 160, in check_config_against_spec_or_exit
    raise Exception("Config validation error: " + validation_error.message) from None
Exception: Config validation error: 'test_data' is not of type 'array'
2022-10-06 19:31:36 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):96 - Error checking connection, status: Optional.empty, exit code: 1

I don’t know why it’s not able to take the array type for the field. Is something wrong with YAML file?

      type:  array
      title: Form ID
      description: Unique identifier for one of your forms
      order: 3

Could you please share the complete code for the connector? That way I could test it out locally!