Running Airbyte locally on Windows for debugging

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Documentation needed for running Airbyte locally on Windows, debugging in IDE, issues with long file paths, SDKs/tools required, and troubleshooting Elasticsearch data source.


Question

Is there documentation on how to get Airbyte running locally on a Windows machine and being able to debug the code in an IDE? Having a hard time due to a fresh checkout running into issues related to long file paths despite having that enabled on Windows, can’t seem to find documentation on all of the SDKs or tools required to build the code, etc. The Contribution Guideline’s on Github don’t seem to have the information and the local deployment section on the website links to a “Getting Started FAQ” (https://docs.airbyte.com/deploying-airbyte/local-deployment#troubleshooting) but it’s just the Github discussion forums which contains no instructions AFAICT.

My use case is we’ve tried using the Elasticsearch data source and it is failing to upload some records but the error it tries to output from the Elasticsearch java client library is null. I’d like to try running the connector locally, be able to step through the code, and see exactly what is going on.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["airbyte-platform", "windows", "debugging", "local-deployment", "sdk", "tools", "elasticsearch", "java-client-library"]
2

To get Airbyte running locally on a Windows machine, you need to follow these steps:

  1. Install Docker Engine and the Docker Compose plugin on your workstation. You can find the instructions here.
  2. After Docker is installed, you can immediately get started locally by running the following commands:
# clone Airbyte from GitHub  
git clone --depth=1 <https://github.com/airbytehq/airbyte.git>  
  
# switch into Airbyte directory  
cd airbyte  
  
# start Airbyte  
./run-ab-platform.sh
  1. In your browser, visit http://localhost:8000
  2. You will be asked for a username and password. By default, that’s username airbyte and password password. Once you deploy Airbyte to your servers, be sure to change these in your .env file:
# Proxy Configuration  
# Set to empty values, e.g. "" to disable basic auth  
BASIC_AUTH_USERNAME=your_new_username_here  
BASIC_AUTH_PASSWORD=your_new_password_here
  1. Start moving some data! source

If you encounter errors about long filenames while cloning on Windows, you can refer to the instructions here to correct it.

To build Airbyte locally, you need the following technologies:

  1. Java 21
  2. Node 16
  3. Python 3.9
  4. Docker
  5. Jq

You can use a version manager such as pyenv or jenv to switch between different language versions. source

To debug the Elasticsearch data source, you can run a version of an Airbyte connector that you are developing on. Here are the steps:

  1. In your local airbyte repository, run the following command:
./run-ab-platform
  1. Then, build the connector image:
  • Install the airbyte-ci tool to build your connector.
  • Running airbyte-ci connectors --name source-&lt;source-name&gt; build will build your connector image.
  • Once the command is done, you will find your connector image in your local docker host: airbyte/source-&lt;source-name&gt;:dev.
  1. In your browser, visit http://localhost:8000/
  2. Log in with the default user airbyte and default password password
  3. Go to Settings (gear icon in lower left corner)
  4. Go to Sources or Destinations (depending on which connector you are testing)
  5. Update the version number to use your docker image tag (default is dev)
  6. Click Change to save the changes

Now when you run a sync with that connector, it will use your local docker image. source

3

To get Airbyte running locally on a Windows machine, you need to follow these steps: 1. Install Docker Engine and the Docker Compose plugin on your workstation. You can find the instructions here. 2. After Docker is installed, you can immediately get started locally by running the following commands: bash # clone Airbyte from GitHub git clone --depth=1 <https://github.com/airbytehq/airbyte.git> # switch into Airbyte directory cd airbyte # start Airbyte ./run-ab-platform.sh 3. In your browser, visit http://localhost:8000 4. You will be asked for a username and password. By default, that’s username airbyte and password password. Once you deploy Airbyte to your servers, be sure to change these in your .env file: bash # Proxy Configuration # Set to empty values, e.g. "" to disable basic auth BASIC_AUTH_USERNAME=your_new_username_here BASIC_AUTH_PASSWORD=your_new_password_here 5. Start moving some data! source If you encounter errors about long filenames while cloning on Windows, you can refer to the instructions here to correct it. To build Airbyte locally, you need the following technologies: 1. Java 21 2. Node 16 3. Python 3.9 4. Docker 5. Jq You can use a version manager such as pyenv or jenv to switch between different language versions. source To debug the Elasticsearch data source, you can run a version of an Airbyte connector that you are developing on. Here are the steps: 1. In your local airbyte repository, run the following command: bash ./run-ab-platform 2. Then, build the connector image: - Install the airbyte-ci tool to build your connector. - Running airbyte-ci connectors --name source-&lt;source-name&gt; build will build your connector image. - Once the command is done, you will find your connector image in your local docker host: airbyte/source-&lt;source-name&gt;:dev. 3. In your browser, visit http://localhost:8000/ 4. Log in with the default user airbyte and default password password 5. Go to Settings (gear icon in lower left corner) 6. Go to Sources or Destinations (depending on which connector you are testing) 7. Update the version number to use your docker image tag (default is dev) 8. Click Change to save the changes Now when you run a sync with that connector, it will use your local docker image. source If you still encounter issues, you can check out the Getting Support documentation for options on how to get in touch with the community

4

On a fresh machine after installing Java 21, Node 16, Python 3.10 (hit a build error), Docker, and Jq I run into the following issue:

The Kotlin Gradle plugin was loaded multiple times in different subprojects, which is not supported and may break the build.
This might happen in subprojects that apply the Kotlin plugins with the Gradle 'plugins { ... }' DSL if they specify explicit versions, even if the versions are equal.
Please add the Kotlin plugin to the common parent project or the root project, then remove the versions in the subprojects.
If the parent project does not need the plugin, add 'apply false' to the plugin line.
See:
<https://docs.gradle.org/current/userguide/plugins.html#sec:subprojects_plugins_dsl>
The Kotlin plugin was loaded in the following projects: ':airbyte-cdk:java:airbyte-cdk', ':airbyte-integrations:connectors:destination-bigquery'
&gt; Task :airbyte-integrations:connectors:checkPython
Using python 3.10.11 from C:\Github\airbyte\airbyte-integrations\connectors\build\.venv (C:\Github\airbyte\airbyte-integrations\connectors\build\.venv\Scripts\python.exe)
Using pip 21.0.1 from C:\Github\airbyte\airbyte-integrations\connectors\build\.venv\lib\site-packages\pip (python 3.10)
&gt; Task :airbyte-integrations:connectors:poetryInstallAirbyteCI FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:poetryInstallAirbyteCI'.
&gt; A problem occurred starting process 'command 'C:\Github\airbyte\airbyte-integrations\connectors\build\.venv\bin\python''
* Try:
&gt; Run with --info or --debug option to get more log output.
&gt; Get more help at
<https://help.gradle.org>.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':airbyte-integrations:connectors:poetryInstallAirbyteCI'.```