Unable to use JWT library as part of custom connector development

[Saad]

Hello!

We am unable to use any JWT library as part of our custom connector development. Our intention is to use jwt in order to create an RSA 256 Bearer Token for authenticating our requests to a REST API Source. We have tried the following libraries and still failed: pyjwt, python-jose, jwcrypto, authlib. If we try to include these libraries in our import statements in source.py and include the necessary installation command in the Dockerfile (pip install pyjwt), we are unable to deploy the resulting Airbyte connector docker image as a new Custom Connector within Airbyte. We get the following below errors (attached as screenshots) when we try to add a new connector, or change an existing connector with the jwt import statement. Can you help us make sense of these errors? Does Airbyte support using jwt libraries for creating JWT Tokens for RSA 256?

Thank you!

[marcosmarxm]

Hello Anwar, the code for the connector is available in Github?
You must put the external libraries inside the setup.py file. See example here: https://github.com/airbytehq/airbyte/blob/147100f06b22a225086bbe6825970ae6fc76a700/airbyte-integrations/connectors/source-facebook-marketing/setup.py#L8-L13

Is your connector working running commads from the README instructions?

[Saad]

Unfortunately we cannot push this code to Github as the code contains sensitive information.

Even after adding the dependency to setup.py, we are still getting the exact error as above. This is what I have in setup.py for MAIN_REQUIREMENTS:

MAIN_REQUIREMENTS = [
“airbyte-cdk~=0.1”,
“PyJWT==2.3.0”
]

Any ideas?

[Saad]

And here is our Dockerfile:

FROM python:3.7.11-alpine3.14 as base

build and load all requirements

FROM base as builder
WORKDIR /airbyte/integration_code

upgrade pip to the latest version

RUN apk --no-cache upgrade
&& pip install --upgrade pip
&& apk --no-cache add tzdata build-base
&& pip install PyJWT

COPY setup.py ./

install necessary packages to a temporary folder

RUN pip install --prefix=/install .

build a clean environment

FROM base
WORKDIR /airbyte/integration_code

copy all loaded and built libraries to a pure basic image

COPY --from=builder /install /usr/local

add default timezone settings

COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
RUN echo “Etc/UTC” > /etc/timezone

bash is installed for more convenient debugging.

RUN apk --no-cache add bash

copy payload code only

COPY main.py ./
COPY source_cpm ./source_cpm

ENV AIRBYTE_ENTRYPOINT “python /airbyte/integration_code/main.py”
ENTRYPOINT [“python”, “/airbyte/integration_code/main.py”]

LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/source-cpm

[marcosmarxm]

What is the output after installing the pyenv?

source .venv/bin/activatepip install -r requirements.txtpip install '.[tests]'

and

python main.py specpython main.py check --config secrets/config.json

because after adding to setup.py your connector should be able to import and use the library.

[Saad]

Just to be clear, the issue is not that we are unable to run the connector locally on our machine using the command line. The issue is that we cannot integrate/install this connector on Airbyte UI. Please also note that we are using the open source version of Airbyte, not the official paid cloud offering of Airbyte.

Do I need to run the following commands in the docker image that we are building for the connector?
source .venv/bin/activate
pip install -r requirements.txt
pip install ‘.[tests]’

I am building the connector as a docker image using the following command:

docker build . -t airbyte/source-cpm:dev

This results in a docker image (airbyte/source-cpm) which I then try to deploy within the Airbyte UI:

This of course results in the error: Internal Server Error: Get Spec job failed.

Do we need to run the source and pip install commands you mentioned as part of the docker image build process for our connector?

Thank you!

[marcosmarxm]

You should be able to add new connectors in the Airbyte UI using the Open Source. Can you share the server logs to identify the error for adding the connector? Also you can intercept the API call in developer tools → network tab.

[Saad]

We were able to resolve our problem. Pointing us to the README was very helpful and we were able to figure it out from there.

Thank you for your help, cheers!

[marcosmarxm]

Happy to hear that! Let us know if you need anything else!

[marcosmarxm]

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.