Internal Server Error when deploying container based connector

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

The user is facing an Internal Server Error when trying to deploy a container based connector. Docker run commands like spec, check, discover, and read are failing despite using a YAML template from the generator without any modifications.


Question

Low code CDK using YAML and command line. Everything is working. Going to deploy it: built container and adding container based connector to UI, I get “Internal Server Error: Get Spec job failed.” I am completely lost at where to go given the fact that my docker run commands (spec, check, discover, read) are failing, but I used a YAML template from the generator and didn’t remove anything. What am I missing? Completely lost…


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["low-code-cdk", "yaml", "command-line", "container-based-connector", "internal-server-error", "docker-run", "yaml-template"]
2

I’m sorry to hear that you’re encountering issues with deploying your connector. Based on the information provided in the extracts, there are a few things you could try to resolve the “Internal Server Error: Get Spec job failed” issue. 1. Run spec and check commands on the connector: As suggested by a support agent in one of the forum posts, you could try running the spec and check commands on the connector to confirm that the connector is valid and it is able to connect to the source. Here’s how to run them in Python: python from airbyte-integrations/connectors/source-<source-name> python main.py spec python main.py check --config secrets/config.json (Source: Forum Post: Internal Server Error: Get Spec job failed > Answers Part 1) 2. Check Docker logs: Another user found a solution by checking the Docker logs for the Airbyte server. They found an error message indicating a missing module, which they were able to fix by updating the requirements in the setup.py file. They also updated the logic for the image in source_definitions.yaml and source_specs.yaml, and ensured that the Dockerfile followed the same logic as other connectors. bash docker logs container_name (Source: Forum Post: Python custom source build - Internal Server Error: Get Spec job failed > Official Answer) 3. Check dependencies: In another case, a user was able to resolve the issue by adding necessary dependencies to the setup.py file. python “google-api-python-client==1.12.8”, “google-auth-httplib2==0.0.4”, “google-auth-oauthlib==0.4.2”, “oauth2client==4.1.3” (Source: Forum Post: Internal Server Error: Get Spec job failed > Answers Part 2) Please try these suggestions and let me know if they help resolve your issue.

3

My issues are fixed:

  1. I was having an issue with my local python - once I could rerun that, I could rerun the python run commands
  2. I took the default Dockerfile but had to uncomment the final two lines in the online version
    version online

COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code

# The entrypoint and default env vars are already set in the base image
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]```
4

version that worked

5 

COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code

# The entrypoint and default env vars are already set in the base image
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]```