To access a custom python destination in Airbyte locally, you need to build the connector and configure it properly.
I have made my custom python destination locally. How do i build it so that when I deploy airbyte locally on my machine, I can access the destination on my localhost
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
To build your custom Python destination and make it accessible on your localhost, you need to follow these steps: 1. Set up your Python Virtual Environment: This is usually done with venv. Here’s an example of how to do it: bash cd airbyte-integrations/connectors/destination-duckdb # Create a virtual environment in the .venv directory python -m venv .venv # activate the virtualenv source .venv/bin/activate # Install with the "tests" extra which provides test requirements pip install '.[tests]' Make sure you set all relevant environment variables. For example: bash BASIC_AUTH_USERNAME=airbyte BASIC_AUTH_PASSWORD=password Source 2. Test your connector: You can test your connector locally with Python and a test file. Here’s an example of how to do it: bash source .venv/bin/activate python -m pip install --upgrade pip pip install -r requirements.txt python main.py spec python main.py check --config integration_tests/config.json python main.py discover --config integration_tests/config.json cat integration_tests/messages.jsonl| python main.py write --config integration_tests/config.json --catalog integration_tests/configured_catalog.json Source 3. Build the Airbyte platform: You can build the whole Airbyte platform using the following command: bash SUB_BUILD=PLATFORM ./gradlew build Source 4. Start Airbyte: You can start Airbyte with the following command: bash airbyte-server After starting Airbyte, you should be able to access it at localhost:8000. In the destinations, you should now see your new connector. Source Please note that these instructions assume that you have already installed the necessary prerequisites such as Python, pip, Homebrew, and Docker, and that you have cloned the Airbyte repository. If you haven’t done so, you can refer to this YouTube tutorial for guidance on setting up your local development environment for Airbyte.
<@U04SE163WC8> help me write a dockerfile for my python destination. What all data do you need?
Sure, I can guide you on how to write a Dockerfile for your Python destination based on the information provided in the extracts. Here’s an example Dockerfile: Dockerfile FROM python:3.9.11 as base # build and load all requirements FROM base as builder WORKDIR /airbyte/integration_code # upgrade pip to the latest version RUN apt-get update && apt-get -y upgrade \ && pip install --upgrade pip COPY setup.py ./ # install necessary packages to a temporary folder RUN pip install --prefix=/install . # build a clean environment FROM base # RUN conda install -c conda-forge python-duckdb WORKDIR /airbyte/integration_code # copy all loaded and built libraries to a pure basic image COPY --from=builder /install /usr/local # add default timezone settings COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime RUN echo "Etc/UTC" > /etc/timezone #adding duckdb manually (outside of setup.py - lots of errors) RUN pip install duckdb # copy payload code only COPY main.py ./ COPY destination_duckdb ./destination_duckdb ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py" ENTRYPOINT ["python", "/airbyte/integration_code/main.py"] LABEL io.airbyte.version=0.1.0 LABEL io.airbyte.name=airbyte/destination-duckdb This Dockerfile is for a Python destination using DuckDB. You may need to adjust it according to your specific requirements. For instance, you might need to replace duckdb with the specific Python package your destination requires. Please note that the Dockerfile assumes that you have a setup.py file and your Python code (including a main.py file and a destination_duckdb directory) in the same directory as the Dockerfile. You can build the Docker image using the following command: bash docker build . -t airbyte/destination-duckdb:dev --no-cache This command builds the Docker image and tags it as airbyte/destination-duckdb:dev. The --no-cache option ensures that all steps are run even if they’ve been run before. You might want to remove this option if you’re not making changes that would affect the Docker build. Please refer to the tutorial for more details.
<@U04SE163WC8> I get the following error when I try to run airbyte locally after making my custom python destination
[+] Running 11/11 2.9s
✘ init Error 2.9s
✘ webapp Error 2.9s
✘ worker Error 2.9s
✘ airbyte-cron Error 2.9s
✘ airbyte-temporal Error 2.9s
✘ airbyte-api-server Error 2.9s
✘ server Error 2.9s
✘ airbyte-connector-builder-server Error 2.9s
✘ airbyte-proxy Error 2.9s
✘ db Error 2.9s
✘ bootloader Error 2.9s
Error response from daemon: manifest for airbyte/db:dev not found: manifest unknown: manifest unknown
A message from http://kapa.ai|kapa.ai
The error message you’re seeing indicates that Docker is unable to find the image airbyte/db:dev. This could be due to a few reasons: 1. The image does not exist: You might not have built the image yet. Make sure you have run the build command for your Docker image. 2. The image tag is incorrect: The tag dev might not exist for the image airbyte/db. Make sure you’re using the correct tag. 3. Docker is not able to pull the image: If the image is not locally available, Docker will try to pull it from a Docker registry. If it’s unable to do so, you might see this error. Make sure you’re connected to the internet and able to access Docker’s image registry. To resolve this issue, you can try the following steps: 1. Build the Docker image: If you haven’t done so already, build the Docker image using the docker build command. For example, if your Dockerfile is in the current directory, you can use docker build -t airbyte/db:dev . to build the image and tag it as dev. 2. Check the image tag: Use docker images to list all Docker images that are locally available. Check if airbyte/db is listed and if it has a tag dev. 3. Pull the Docker image: If the image is not locally available, you can try to pull it from Docker’s registry using docker pull airbyte/db:dev. If you’re still having trouble, you might want to check the Airbyte documentation or ask for help in the Airbyte community.
<@U04SE163WC8> I am still getting the above error, here is my dockerfile
# build and load all requirements
FROM base as builder
WORKDIR /airbyte/integration_code
# upgrade pip to the latest version
RUN apt-get update && apt-get -y upgrade \
&& pip install --upgrade pip
COPY setup.py ./
# install necessary packages to a temporary folder
RUN pip install --prefix=/install .
# build a clean environment
FROM base
# RUN conda install -c conda-forge python-duckdb
WORKDIR /airbyte/integration_code
# copy all loaded and built libraries to a pure basic image
COPY --from=builder /install /usr/local
# add default timezone settings
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
RUN echo "Etc/UTC" > /etc/timezone
# copy payload code only
COPY main.py ./
COPY destination_quickbooks ./destination_quickbooks
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/destination-destination_quickbooks```
and here is my metadata.yaml
```data:
allowedHosts:
hosts:
- <http://sandbox-quickbooks.api.intuit.com|sandbox-quickbooks.api.intuit.com>
- <http://quickbooks.api.intuit.com|quickbooks.api.intuit.com>
- <http://oauth.platform.intuit.com|oauth.platform.intuit.com>
registries:
oss:
enabled: True
cloud:
enabled: false
connectorBuildOptions:
# Please update to the latest version of the connector base image.
# Please use the full address with sha256 hash to guarantee build reproducibility.
# <https://hub.docker.com/r/airbyte/python-connector-base>
baseImage: <http://docker.io/airbyte/python-connector-base:1.0.0@sha256:dd17e347fbda94f7c3abff539be298a65af2d7fc27a307d89297df1081a45c27|docker.io/airbyte/python-connector-base:1.0.0@sha256:dd17e347fbda94f7c3abff539be298a65af2d7fc27a307d89297df1081a45c27>
connectorSubtype: database
connectorType: destination
definitionId: bc0f5066-5aee-4e05-bb68-fadfb6a6fa09
dockerImageTag: 0.1.0
dockerRepository: airbyte/destination-quickbooks
githubIssueLabel: destination-quickbooks
icon: quickbooks.svg
license: MIT
name: Quickbooks
releaseDate: TODO
releaseStage: alpha
supportLevel: community
documentationUrl: <https://docs.airbyte.com/integrations/destinations/quickbooks>
tags:
- language:python
metadataSpecVersion: "1.0"```The error you’re encountering is not directly related to your Dockerfile or metadata.yaml file. The error message indicates that Docker is unable to find the image airbyte/db:dev. This image is part of the Airbyte core services and is not related to your custom Python destination. Here are a few steps you can take to troubleshoot this issue: 1. Pull the Docker image manually: Try to manually pull the airbyte/db:dev image using the command docker pull airbyte/db:dev. If this command fails, it means that the image does not exist in the Docker registry. 2. Check your Docker Compose file: The airbyte/db:dev image is specified in the Docker Compose file used to start Airbyte. Make sure that the image is correctly specified in this file. If you’ve made changes to the Docker Compose file, try to revert them or use the original file from the Airbyte repository. 3. Rebuild Airbyte: If you’ve made changes to the Airbyte core services, you might need to rebuild Airbyte. You can do this by running ./gradlew composeBuild in the root of the Airbyte repository. 4. Check your internet connection: Docker needs an internet connection to pull images from the Docker registry. Make sure that your internet connection is working and that you’re not behind a firewall that could block Docker. If none of these steps solve your issue, I recommend asking for help in the Airbyte community.
<@U04SE163WC8> explain this error that I am getting
(.venv) @aazam-gh ➜ …/airbyte/airbyte-integrations/connectors/destination-quickbooks (qb) $ airbyte-ci connectors --name destination-quickbooks build
[15808] Error loading Python lib ‘/tmp/_MEI8tRAAq/libpython3.10.so.1.0’: dlopen: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35’ not found (required by /tmp/_MEI8tRAAq/libpython3.10.so.1.0)
The error message you’re seeing is related to a mismatch between the version of the GNU C Library (GLIBC) required by the Python library (libpython3.10.so.1.0) and the version available on your system. The Python library is looking for version GLIBC_2.35, but it’s not found on your system. This usually happens when you’re trying to run a program that was built on a system with a newer version of GLIBC. Here are a few steps you can take to resolve this issue: 1. Upgrade your system: The simplest solution is to upgrade your system to a newer version that includes GLIBC_2.35. This will likely involve upgrading your Linux distribution to a newer version. 2. Install a newer version of GLIBC: If upgrading your system is not an option, you can try to manually install a newer version of GLIBC. This is a more complex solution and it might not be possible depending on your Linux distribution. 3. Build the program on your system: If you have access to the source code of the program, you can try to build it on your system. This will ensure that it’s compatible with the version of GLIBC available on your system. In your case, since you’re building a custom Airbyte connector, you might want to check the Python version you’re using. The error message indicates that you’re using Python 3.10, which might not be compatible with the version of GLIBC on your system. You could try using an older version of Python that’s compatible with your system. If none of these steps solve your issue, I recommend asking for help in the Airbyte community.
You can build the Docker image manually and creaste a custom connector in the web ui
<@U02TQLBLDU4> So first is to run docker build -t airbyte/destination-quickbooks:dev . to build my destination and then from the root directory run the ./run-ab-platform.sh ?
<@U04SE163WC8> after building my custom connector using docker, should I rebuild airbyte for the changes to be visible in the web ui when running it on my local machine? IF yes, what are the commands to do it?
After building your custom connector using Docker, you don’t need to rebuild the entire Airbyte for the changes to be visible in the web UI. You can add your custom connector to the UI by following these steps: 1. Get the Docker coordinate of your custom connector from Dockerhub (or any image repository that Airbyte can access). 2. In the UI, go to the Admin section, and click on [+ New connector] on the top right. 3. You will be asked for the display name, the Docker repository name (repository + image name), tag, and documentation URL for that connector. Once this is filled, you will see your connector in the UI and your team will be able to use it, from the UI and Airbyte’s API too. Here is the source for your reference: Adding your connectors in the UI However, if you’re developing a connector and want to test it locally, you can follow these steps: 1. In your local airbyte repository, run the following command: bash ./run-ab-platform 2. Then, build the connector image: - Install the airbyte-ci tool to build your connector. - Running airbyte-ci connectors --name source-<source-name> build will build your connector image. - Once the command is done, you will find your connector image in your local docker host: airbyte/source-<source-name>:dev. You can find more details in the Airbyte documentation.
<@U04SE163WC8> how to run airbyte with admin privileges in local machine
<@U047WETGHFW> That’s a great question, but unfortunately I am only capable of responding to a maximum of five questions per thread. Don’t hesitate to create a new thread for any further questions you have though. I’m here to help!