Summary
The user is facing issues while building a Python HTTP integration source for an Airbyte connector. The build process is unable to find the source directory or main.py. After removing generated files from the src directory, the build command works, but there are problems with schema discovery when adding the connector to Airbyte OSS.
Question
Hi There, I hope that everyone is doing well today. I was wondering if someone may be able to share a little insight into an issue I have been having. I’ve started the process of building a Python HTTP integration source. I’ve been following through with the tutorial that is part of the documentation. Everything seems to be working great up until the time I need to call:
airbyte-ci connectors --name source-shopify-partners-api build
during the build process it was not able to find my source directory or main.py
After looking through some issues on GH, I saw there may be an issue with the way things are now being generated and placed in a src/ directory. I’ve removed the generated files from the src directory, and the build command appears to work.
I’m pushing the image up to docker hub, and then trying to add he newly built connector to AirByte OSS, through the web ui. This seems to go smoothly, until you try to create the connection and add a destination. Upon trying to setup the connection, AirByte is not able to discover the schema. Is there something else that may need adjusted to ensure the schema.json files are referenced correctly?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["python-http-integration", "airbyte-connector", "source-directory", "schema-discovery", "docker-hub"]
<@U02UC3T12PJ> do you mind giving some help here?
For Reference, this is currently what my pyproject.toml looks like:
requires = ["poetry-core>=1.0.0", ]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
version = "0.1.0"
name = "source-shopify-partners"
description = "Source implementation for shopify-partners."
authors = ["Airbyte <contact@airbyte.io>", ]
license = "MIT"
readme = "README.md"
documentation = "<https://docs.airbyte.com/integrations/sources/shopify-partners>"
homepage = "<https://airbyte.com>"
repository = "<https://github.com/airbytehq/airbyte>"
include = ["source_shopify_partners/schemas/*.json"]
[[tool.poetry.packages]]
include = "source_shopify_partners"
[[tool.poetry.packages]]
include = "main.py"
[tool.poetry.dependencies]
python = "^3.9,<3.12"
airbyte-cdk = "==0.59.0"
requests = "^2.31.0"
[tool.poetry.scripts]
source-shopify-partners = "source_shopify_partners.run:run"
[tool.poetry.group.dev.dependencies]
requests-mock = "^1.11.0"
pytest-mock = "^3.12.0"
pytest = "^8.0.0"```
<@U047ANT3J84> is it the same problem you previously shared with me on a different post ?
Would believe it is the same/similar to the issue started by <@U047ANT3J84> last week:
https://github.com/airbytehq/airbyte/issues/35893
As suggested in that GH issue, I had moved the files out of the src/ directory, and this seems to allow the build to complete, but when you go to use the source connector the schema is not able to be discovered.
This is the message from the AirByte Web UI:
Initial check at setup works and goes though, but when you try to connect with a destination, this appears.
Using poetry to check discover seems to work prior to the build though.
<@U02UC3T12PJ> I’m not familiar with any issues with discover specifically nor issues from the platform. The issues that I was trying to help the user debug was just accomplishing building the connector (though it’s possible when they got around this by moving things out of src they might have run into the same error)
For me I was having the same/very similar to the initial error. when running build with airbyte-ci I was getting a similar message to that in the GH issue, and in the console it was also saying it couldn’t find my source-<name> directory or main.py. Moving those out of the src/ directory seems to fix those issues with the build process it self, but when trying to use the integration the new issues around discover come up, check does seem to work as I am able to setup the connector and get to the point of connecting it to the destination.
> Using poetry to check discover seems to work prior to the build though.
<@U06PJ7EL1CK> are you able to perform a docker run ... discover locally?
I.e. using docker and not poetry
docker run --rm -v $(pwd)/secrets:/config th3sven/source-shopify-partners:0.0.9 discover --config /config/config.json
Sorry for the Giant wall of output:
{"type": "CATALOG", "catalog": {"streams": [{"name": "relationship_installs", "json_schema": {"$schema": "<http://json-schema.org/draft-07/schema#>", "type": "object", "properties": {"events": {"type": ["null", "array"], "items": {"type": ["null", "object"], "properties": {"event_type": {"type": "string"}, "occurredAt": {"type": ["null", "string"], "format": "date-time"}, "app": {"type": "object", "properties": {"id": {"type": "string"}, "name": {"type": "string"}}}, "shop": {"type": "object", "properties": {"id": {"type": "string"}, "name": {"type": "string"}}}}}}}}, "supported_sync_modes": ["full_refresh"], "source_defined_primary_key": [["occurredAt"]]}, {"name": "relationship_uninstalls", "json_schema": {"$schema": "<http://json-schema.org/draft-07/schema#>", "type": "object", "properties": {"events": {"type": ["null", "array"], "items": {"type": ["null", "object"], "properties": {"event_type": {"type": "string"}, "occurredAt": {"type": ["null", "string"], "format": "date-time"}, "app": {"type": "object", "properties": {"id": {"type": "string"}, "name": {"type": "string"}}}, "shop": {"type": "object", "properties": {"id": {"type": "string"}, "name": {"type": "string"}}}}}}}}, "supported_sync_modes": ["full_refresh"], "source_defined_primary_key": [["occurredAt"]]}, {"name": "relationship_reactivated", "json_schema": {"$schema": "<http://json-schema.org/draft-07/schema#>", "type": "object", "properties": {"events": {"type": ["null",
Ah it was too long to post. But TLDR; yes. :rolling_on_the_floor_laughing:
huh! so no errors locally but errors in the cloud. interesting.
I think common.error is what comes out when we don’t get a schema out of the connector but we also don’t get an error. But most of the issues I would expect to lead to something funky like this i’d expect to happen during spec or check when used in the UI 
Are you doing this in airbyte cloud or your own instance?
It’s in my own instance for the time being. We’re actually evaluating OSS vs Cloud as part of a exercise for a project we’re working on, but this was a missing api for the data we need to collect.
No worries, was just wondering if I could get in and take a look myself
Ah Okay! If you didn’t mind using wireguard I don’t mind letting you in at all. It’s just a lab server we’re working on.
Or a screen share if you wanted to poke around