Running connector docker image with catalog as NoneType

Connector Development
1

I’m currently building Workday connector using Singer-tap. I discovered the configured_catalog.json that matches the ConfiguredAirbyteCatalog format. When I sync data with command
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-workday-singer:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json,

I got the error msg:

y@y source-workday-singer % docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-workday-singer:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
{"type": "LOG", "log": {"level": "INFO", "message": "Starting discover"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Downloading XSD to determine table schema \"INT_DWH_Publisher_Combined_Journal_Lines_Report\"."}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished discover"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Starting sync."}}
{"type": "LOG", "log": {"level": "ERROR", "message": "CRITICAL 'NoneType' object has no attribute 'get_selected_streams'\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "Traceback (most recent call last):\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "  File \"/usr/local/bin/tap-workday-raas\", line 8, in <module>\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "    sys.exit(main())\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "  File \"/usr/local/lib/python3.9/site-packages/singer/utils.py\", line 229, in wrapped\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "    return fnc(*args, **kwargs)\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "  File \"/usr/local/lib/python3.9/site-packages/tap_workday_raas/__init__.py\", line 55, in main\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "    do_sync(args.config, args.catalog, args.state)\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "  File \"/usr/local/lib/python3.9/site-packages/tap_workday_raas/__init__.py\", line 30, in do_sync\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "    for stream in catalog.get_selected_streams(state):\n"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "AttributeError: 'NoneType' object has no attribute 'get_selected_streams'\n"}}
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 117, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 108, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/singer/singer_helpers.py", line 168, in read
    for line, text_wrapper in SingerHelper._read_lines(p):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/singer/singer_helpers.py", line 202, in _read_lines
    raise Exception(f"Underlying command {process.args} failed with exit code {process.returncode}")
Exception: Underlying command tap-workday-raas --config /tmp/tmp61loj8ts/config.json --properties singer_rendered_catalog.json  failed with exit code 1

I tried to do the discover and sync data in tap-workday-raas directory following this, and it successed. However, the output properties.json file from discover has different format from the configure_catalog.json that is genereated by docker (properties.json has “stream” field of string, but configured_catalog.json has “stream” field must be a map).
My configured_catalog.json looks like this:

image
image912×956 68.8 KB

The properties.json is like this
Screen Shot 2022-06-09 at 2.12.24 PM
Screen Shot 2022-06-09 at 2.12.24 PM712×1546 54.7 KB

2

Hi Yifan,
Could you please assign this issue to yourself if you are working on this connector?
Did you try to use the output of the discover command to build the configured_catalog.json ?
I would suggest you give a look to other singer-based connectors to check how they manage this, such as source-kustomer-singer
I would also help for debuggingging if you could open a draft PR on our repo so that we can pull your branch and try to reproduce this problem on our side too.

3

I used docker to get the configured_catalog.json from discover command, but the format is not ready-to-use. Thus, I reformated it according to the one in source-kustomer-singer and there’s no format error. I cannot get the configured_catalog.json by running python main.py discover --config secrets/config.json as this and I cannot have base_python.

4

Ok, thanks for the clarification.
Are you making headways or are you blocked by this issue?

5

I cannot use from base_python.entrypoint import launch since I cannot install base_python package. Or which package shall I install so that I can use base_python?
So I used docker to check and discover, it worked. My problem is still the READ process in which I have the correctely formated configured_catalog.json in my command, but it’s recognized None.

7

Could you please open a draft PR so that I can try to reproduce this problem and try to debug this?
We do not have a Workday account at the moment, but if you open a PR our team will try to create one to run the acceptance tests. Nevertheless, without an account, I should still be able to reproduce your problem as I think the error is raised before reading data from the API.

8

I drafted a PR here.