We am unable to use any JWT library as part of our custom connector development. Our intention is to use jwt in order to create an RSA 256 Bearer Token for authenticating our requests to a REST API Source. We have tried the following libraries and still failed: pyjwt, python-jose, jwcrypto, authlib. If we try to include these libraries in our import statements in source.py and include the necessary installation command in the Dockerfile (pip install pyjwt), we are unable to deploy the resulting Airbyte connector docker image as a new Custom Connector within Airbyte. We get the following below errors (attached as screenshots) when we try to add a new connector, or change an existing connector with the jwt import statement. Can you help us make sense of these errors? Does Airbyte support using jwt libraries for creating JWT Tokens for RSA 256?
Hello Anwar, the code for the connector is available in Github?
You must put the external libraries inside the setup.py file. See example here: https://github.com/airbytehq/airbyte/blob/147100f06b22a225086bbe6825970ae6fc76a700/airbyte-integrations/connectors/source-facebook-marketing/setup.py#L8-L13
Is your connector working running commads from the README instructions?
Unfortunately we cannot push this code to Github as the code contains sensitive information.
Even after adding the dependency to setup.py, we are still getting the exact error as above. This is what I have in setup.py for MAIN_REQUIREMENTS:
MAIN_REQUIREMENTS = [
And here is our Dockerfile:
FROM python:3.7.11-alpine3.14 as base
build and load all requirements
FROM base as builder
upgrade pip to the latest version
RUN apk --no-cache upgrade
&& pip install --upgrade pip
&& apk --no-cache add tzdata build-base
&& pip install PyJWT
COPY setup.py ./
install necessary packages to a temporary folder
RUN pip install --prefix=/install .
build a clean environment
copy all loaded and built libraries to a pure basic image
COPY --from=builder /install /usr/local
add default timezone settings
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
RUN echo “Etc/UTC” > /etc/timezone
bash is installed for more convenient debugging.
RUN apk --no-cache add bash
copy payload code only
COPY main.py ./
COPY source_cpm ./source_cpm
ENV AIRBYTE_ENTRYPOINT “python /airbyte/integration_code/main.py”
ENTRYPOINT [“python”, “/airbyte/integration_code/main.py”]
What is the output after installing the pyenv?
source .venv/bin/activatepip install -r requirements.txtpip install '.[tests]'
python main.py specpython main.py check --config secrets/config.json
because after adding to setup.py your connector should be able to import and use the library.
Just to be clear, the issue is not that we are unable to run the connector locally on our machine using the command line. The issue is that we cannot integrate/install this connector on Airbyte UI. Please also note that we are using the open source version of Airbyte, not the official paid cloud offering of Airbyte.
Do I need to run the following commands in the docker image that we are building for the connector?
pip install -r requirements.txt
pip install ‘.[tests]’
I am building the connector as a docker image using the following command:
docker build . -t airbyte/source-cpm:dev
This results in a docker image (airbyte/source-cpm) which I then try to deploy within the Airbyte UI:
This of course results in the error: Internal Server Error: Get Spec job failed.
Do we need to run the source and pip install commands you mentioned as part of the docker image build process for our connector?
You should be able to add new connectors in the Airbyte UI using the Open Source. Can you share the server logs to identify the error for adding the connector? Also you can intercept the API call in developer tools → network tab.
We were able to resolve our problem. Pointing us to the README was very helpful and we were able to figure it out from there.
Thank you for your help, cheers!
Happy to hear that! Let us know if you need anything else!