Troubleshooting connector generation and setup

Summary

The user is facing an issue with running the generated connector. The ‘source-fastspring’ command is not found after generating the connector using the connector generator.


Question

Is something wrong with my environment? Trying to follow this:

https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/install-dependencies

I just generated the connector using the connector generator, selecting Python HTTP API Source and naming it fastspring and:

total 64
-rw-r--r--@ 1 dossy  staff  8800 Feb 28 15:14 README.md
-rw-r--r--@ 1 dossy  staff  1603 Feb 28 15:14 acceptance-test-config.yml
drwxr-xr-x@ 9 dossy  staff   288 Feb 28 15:14 integration_tests
-rw-r--r--@ 1 dossy  staff   138 Feb 28 15:14 main.py
-rw-r--r--@ 1 dossy  staff  1108 Feb 28 15:14 metadata.yaml
-rw-r--r--@ 1 dossy  staff     5 Feb 28 15:14 requirements.txt
drwxr-xr-x@ 3 dossy  staff    96 Feb 28 15:14 secrets
-rw-r--r--@ 1 dossy  staff   792 Feb 28 15:14 setup.py
drwxr-xr-x@ 7 dossy  staff   224 Feb 28 15:14 source_fastspring
drwxr-xr-x@ 6 dossy  staff   192 Feb 28 15:14 unit_tests

$ cd connectors/source-fastspring

$ poetry run source-fastspring spec
Command not found: source-fastspring```
:thinking_face: :shrug:

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1709151644541239) if you want to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["connector", "connector-generation", "python-http-api-source", "troubleshooting", "command-not-found"]
</sub>

Based on the information provided from the extracts, it seems like you’ve successfully generated a new connector named fastspring using the Airbyte connector generator and have navigated to the connector’s directory. However, you’re encountering an issue when trying to run the poetry run source-fastspring spec command, receiving a “Command not found” error. This issue might be occurring because the command you’re trying to execute doesn’t directly correspond to a script or entry point defined within your project’s pyproject.toml file (which is managed by Poetry). Typically, after generating a connector, you would need to install its dependencies using poetry install before running any commands. However, the specific command format poetry run source-&lt;name&gt; spec might not be applicable directly unless it’s explicitly defined in your project’s configuration. In the extracts provided, there’s a mention of running commands like poetry run source-&lt;name&gt; spec after navigating to the connector’s directory and installing dependencies (source). If you’ve already run poetry install and are still facing this issue, it’s possible that there’s a misunderstanding or a missing step in the setup process. One common approach to testing your source in development, as mentioned in the extracts, involves directly invoking the main.py script (source). You might try running your source commands using the main.py script directly within your virtual environment. For example: bash poetry run python main.py spec This approach assumes that main.py is the entry point for your connector and is set up to handle the spec command. If this doesn’t work, you may need to review the main.py file to understand how it’s structured and how it expects commands to be invoked. If you’re still encountering issues, it might be helpful to review the documentation and tutorials provided by Airbyte more closely to ensure all steps have been followed correctly. Unfortunately, without more specific details on the setup and configuration of your fastspring connector, it’s challenging to provide a more precise solution.

Installing dependencies from lock file

No dependencies to install or update

Installing the current project: airbyte (0.1.0)
Warning: The current project could not be installed: No file/folder found for package airbyte
If you do not want to install the current project use --no-root.
If you want to use Poetry only for dependency management but not for packaging, you can set the operating mode to "non-package" in your pyproject.toml file.
In a future version of Poetry this warning will become an error!```

In case anyone wonders:

Python 3.10.13

$ poetry --version
Poetry (version 1.8.1)

$ git log --no-decorate --oneline
6f5c9a33 Docs/PyAirbyte: created pyairbyte folder for docs (#35690)```

Looks like the docs got updated a little prematurely - the PR to move the generator to poetry is <https://github.com/airbytehq/airbyte/pull/35411|not quite in>. Sorry about that - the docs previous to <update doc to reference poetry by alafanechere · Pull Request #35414 · airbytehq/airbyte · GitHub change> should apply to your current state of the generator having run (tldr you probably want python main.py wherever poetry run source-exchange-rates-tutorial is)

Or you could checkout the “update the generator” pr branch (or wait a little bit til it’s merged) and rerun the generator to get the new generator that uses poetry

Okay, thanks for confirming that it’s not something I’m doing wrong or is otherwise wrong with my machine/environment!

Hello <@U047ANT3J84>
I am facing exactly the same situation trying to follow the guide how to build the first Python connector. Do you know which version of the documentation I should follow to make it work?

Hey <@U05F5HRM6KV>! The PR to update the generator just went in - if you want to pull that, you’ll get the most updated version that aligns with the documentation. If you have any problems with the new code, please let me know!

Hello. I have one more question about the updates:
I am not sure if the example Docker example https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/use-connector-in-airbyte|here is up to date with the poetry approach defined by the new generator.

Hey <@U0668M7G22W>, thanks for pointing that out. I’d encourage you to use airbyte-ci if you can, as we’re moving away from connector-specific dockerfiles in favor of a shared base image.

That being said if you’re talking specifically about the entrypoints, I think you’re right - they need to be updated! I can take a look at that or would be happy to help you contribute a docs update (knowing exactly what steps works for you would be helpful to validate the changes)

Hello again <@U047ANT3J84> ,
After using the new generator version, I have tried to build the connector using the airbyte-ci but it failed with the following error message:
Failed to build connector image for platform linux/amd64: lstat main.py: no such file or directory
I am not sure if this issue is related to the new project folder structure adding the main.py file inside src folder.

After moving the code to the root level, the airbyte-ci was able to build

Im glad you’ve gotten yourself unblocked! I think this is a bug, have created a ticket to look into it: <Issues · airbytehq/airbyte · GitHub this issue on GitHub>

If you react on the issue (just a thumbs up is fine) it will remain in the public repository and you can track its progress. Also feel free to comment with any extra info I missed.