Issue with custom source integration using Python CDK and Dockerfile

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Custom source integration developed using Python CDK and packaged with Dockerfile is causing a timeout issue. Pod is created with the provided image but the main pod shows no logs and is terminated.


Question

I have an issue, trying to add custom source developed using python CDK and packaged with recommended Dockerfile.

Request just throws timeout. I see pod created with image I provided (image pull successful) with name like “e-file-source-spec-”. Main pod doesn’t have any logs and its status is terminated.

Env: Airbyte helm chart airbyte-0.45.22 deployed in AWS EKS
Dockerfile:


COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code```
Any suggestion how to debug this?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1716372035043039) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["custom-source", "python-cdk", "dockerfile", "timeout-issue", "pod-termination", "debugging", "airbyte-helm-chart", "aws-eks"]
</sub>
2

Thanks <@U05JENRCF7C>!

I see that there is no executable for any of these commands, I guess it’s because I’m building it with Dockerfile instead of airbyte-ci.

For now we want to use only Dockerfile, let me know if it’s possible to install it correctly using just Dockerfile

3

Roger that. I use Dockerfile in my project as well.

Just to be sure. Have you tried those docker run commands?

docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-{{dashCase name}}:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-{{dashCase name}}:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-{{dashCase name}}:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json```
I think they should work with entrypoint. `spec`, `check`, `discover`, and `read` are passed as arguments
4

<@U05JENRCF7C> Will try it now. Thanks a lot!

5

I’d pull Docker image on local machine and use commands similar to <airbyte/airbyte-integrations/connector-templates/source-python/README.md.hbs at 0d7bf96afdad9227584692a8ef6937765a1bf1fd · airbytehq/airbyte · GitHub in README.md.hbs>
You could build other Docker images from it with some updates if necessary.

Sidenotes: you may want to check the latest tutorials
https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun
https://docs.airbyte.com/connector-development/tutorials/custom-python-connector/getting-started
because they use poetry and airbyte-ci instead of pip and Dockerfile
Commands <airbyte/airbyte-integrations/connector-templates/source-python/README.md.hbs at 0d7bf96afdad9227584692a8ef6937765a1bf1fd · airbytehq/airbyte · GitHub run …> were really useful for me when developing.

6

In my project for App Store Connect API I use following Dockerfile


WORKDIR /airbyte/integration_code

COPY pyproject.toml .
COPY poetry.lock .

COPY README.md .
COPY main.py .
COPY source_appstore_analytics ./source_appstore_analytics

RUN pip install poetry==1.8.2
RUN poetry install

ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/source-appstore-analytics```
In that way I have more visibility for ENTRYPOINT configuration