I鈥檝e written a python connector, which is almost finished, but this build command fails:
./gradlew :airbyte-integrations:connectors:source-fauna:build
Checkout PR 馃帀 New source: Fauna by macmv 路 Pull Request #15274 路 airbytehq/airbyte 路 GitHub to run for yourself.
The build command outputs this:
Output
/data/macmv-programming/fauna/airbyte-fauna $ ./gradlew :airbyte-integrations:connectors:source-fauna:build
Building all of Airbyte.
/data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/connectors
Type-safe dependency accessors is an incubating feature.
> Task :airbyte-integrations:bases:base-normalization:checkPython
Using python 3.10.6 from /data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/bases/base-normalization/.venv (.venv/bin/python)
Using pip 21.3.1 from /data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/bases/base-normalization/.venv/lib/python3.10/site-packages/pip (python 3.10)
> Task :airbyte-integrations:connectors:source-fauna:checkPython
Using python 3.10.6 from /data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/connectors/source-fauna/.venv (.venv/bin/python)
Using pip 21.3.1 from /data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/connectors/source-fauna/.venv/lib/python3.10/site-packages/pip (python 3.10)
> Task :airbyte-integrations:bases:source-acceptance-test:checkPython
Using python 3.10.6 from /data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/bases/source-acceptance-test/.venv (.venv/bin/python)
Using pip 21.3.1 from /data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/bases/source-acceptance-test/.venv/lib/python3.10/site-packages/pip (python 3.10)
> Task :airbyte-commons:compileJava FAILED
> Task :airbyte-integrations:bases:base-normalization:pipInstall
[python] .venv/bin/python -m pip install pip==21.3.1
Requirement already satisfied: pip==21.3.1 in ./.venv/lib/python3.10/site-packages (21.3.1)
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/bases/base-normalization/.venv/bin/python -m pip install --upgrade pip' command.
[python] .venv/bin/python -m pip list --format=columns
Package Version
----------------- -------
attrs 22.1.0
black 22.3.0
click 8.1.3
coverage 6.3.1
flake8 4.0.1
iniconfig 1.1.1
isort 5.6.4
mccabe 0.6.1
mypy 0.930
mypy-extensions 0.4.3
packaging 21.3
pathspec 0.10.1
pip 21.3.1
platformdirs 2.5.2
pluggy 0.13.1
py 1.11.0
pycodestyle 2.8.0
pyflakes 2.4.0
pyparsing 3.0.9
pyproject-flake8 0.0.1a2
pytest 6.1.2
setuptools 60.6.0
toml 0.10.2
tomli 2.0.1
typing_extensions 4.3.0
wheel 0.37.1
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/bases/base-normalization/.venv/bin/python -m pip install --upgrade pip' command.
> Task :airbyte-integrations:connectors:source-fauna:pipInstall
[python] .venv/bin/python -m pip install pip==21.3.1
Requirement already satisfied: pip==21.3.1 in ./.venv/lib/python3.10/site-packages (21.3.1)
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/connectors/source-fauna/.venv/bin/python -m pip install --upgrade pip' command.
[python] .venv/bin/python -m pip list --format=columns
Package Version
----------------- -------
attrs 22.1.0
black 22.3.0
click 8.1.3
coverage 6.3.1
flake8 4.0.1
iniconfig 1.1.1
isort 5.6.4
mccabe 0.6.1
mypy 0.930
mypy-extensions 0.4.3
packaging 21.3
pathspec 0.9.0
pip 21.3.1
platformdirs 2.5.2
pluggy 0.13.1
py 1.11.0
pycodestyle 2.8.0
pyflakes 2.4.0
pyparsing 3.0.9
pyproject-flake8 0.0.1a2
pytest 6.1.2
setuptools 62.6.0
toml 0.10.2
tomli 2.0.1
typing_extensions 4.3.0
wheel 0.37.1
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/connectors/source-fauna/.venv/bin/python -m pip install --upgrade pip' command.
> Task :airbyte-integrations:bases:source-acceptance-test:pipInstall
[python] .venv/bin/python -m pip install pip==21.3.1
Requirement already satisfied: pip==21.3.1 in ./.venv/lib/python3.10/site-packages (21.3.1)
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/bases/source-acceptance-test/.venv/bin/python -m pip install --upgrade pip' command.
[python] .venv/bin/python -m pip list --format=columns
Package Version
---------------------- ---------
airbyte-cdk 0.1.81
attrs 22.1.0
backoff 2.1.2
black 22.3.0
certifi 2022.6.15
charset-normalizer 2.1.1
click 8.1.3
coverage 6.3.1
dataclasses-jsonschema 2.15.3
Deprecated 1.2.13
docker 5.0.3
dpath 2.0.6
fancycompleter 0.9.1
flake8 4.0.1
icdiff 1.9.1
idna 3.3
inflection 0.5.1
iniconfig 1.1.1
isort 5.6.4
Jinja2 3.1.2
jsonref 0.2
jsonschema 3.2.0
MarkupSafe 2.1.1
mccabe 0.6.1
multidict 6.0.2
mypy 0.930
mypy-extensions 0.4.3
packaging 21.3
pathspec 0.10.1
pdbpp 0.10.3
pendulum 2.1.2
pip 21.3.1
platformdirs 2.5.2
pluggy 0.13.1
pprintpp 0.4.0
py 1.11.0
pycodestyle 2.8.0
pydantic 1.9.2
pyflakes 2.4.0
Pygments 2.13.0
pyparsing 3.0.9
pyproject-flake8 0.0.1a2
pyrepl 0.9.0
pyrsistent 0.18.1
pytest 6.1.2
pytest-sugar 0.9.5
pytest-timeout 1.4.2
python-dateutil 2.8.2
pytzdata 2020.1
PyYAML 5.4.1
requests 2.28.1
setuptools 60.6.0
six 1.16.0
source-acceptance-test 0.0.0
termcolor 1.1.0
toml 0.10.2
tomli 2.0.1
typing_extensions 4.3.0
urllib3 1.26.12
vcrpy 4.2.1
websocket-client 1.4.1
wheel 0.37.1
wmctrl 0.4
wrapt 1.14.1
yarl 1.8.1
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/bases/source-acceptance-test/.venv/bin/python -m pip install --upgrade pip' command.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':airbyte-commons:compileJava'.
> error: invalid source release: 17
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 20s
20 actionable tasks: 7 executed, 13 up-to-date
/data/macmv-programming/fauna/airbyte-fauna $ java --version
openjdk 11.0.16.1 2022-08-12
OpenJDK Runtime Environment (build 11.0.16.1+1)
OpenJDK 64-Bit Server VM (build 11.0.16.1+1, mixed mode)
/data/macmv-programming/fauna/airbyte-fauna $
I tried this with both java 11 and java 17, and they both gave the same output. I鈥檓 not sure how gradle finds it鈥檚 java version, but I鈥檓 assuming it just uses the one in $PATH. Also, the way I switched versions of java didn鈥檛 actually change the path of java, instead the java in /usr/bin is just a wrapper script to fetch the right java version.
So the pip error in there was bugging me, as I had just updated pip to version 22.
So I did some googling, and I ended up doing a clean build, which failed on the octavia-cli step (which was before the invalid java version), so I ran that directly, and this shows a problem with the pip version instead:
Output
/data/macmv-programming/fauna/airbyte-fauna $ ./gradlew :octavia-cli:installLocalReqs
Building all of Airbyte.
/data/macmv-programming/fauna/airbyte-fauna/airbyte-integrations/connectors
Type-safe dependency accessors is an incubating feature.
> Task :octavia-cli:checkPython
Using python 3.10.6 from /data/macmv-programming/fauna/airbyte-fauna/octavia-cli/.venv (.venv/bin/python)
Using pip 21.3.1 from /data/macmv-programming/fauna/airbyte-fauna/octavia-cli/.venv/lib/python3.10/site-packages/pip (python 3.10)
> Task :octavia-cli:pipInstall
[python] .venv/bin/python -m pip install pip==21.3.1
Requirement already satisfied: pip==21.3.1 in ./.venv/lib/python3.10/site-packages (21.3.1)
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/octavia-cli/.venv/bin/python -m pip install --upgrade pip' command.
[python] .venv/bin/python -m pip list --format=columns
Package Version
----------------- -------
attrs 22.1.0
black 22.3.0
click 8.1.3
coverage 6.3.1
flake8 4.0.1
iniconfig 1.1.1
isort 5.6.4
mccabe 0.6.1
mypy 0.930
mypy-extensions 0.4.3
packaging 21.3
pathspec 0.10.1
pip 21.3.1
platformdirs 2.5.2
pluggy 0.13.1
py 1.11.0
pycodestyle 2.8.0
pyflakes 2.4.0
pyparsing 3.0.9
pyproject-flake8 0.0.1a2
pytest 6.1.2
setuptools 64.0.3
toml 0.10.2
tomli 2.0.1
typing_extensions 4.3.0
wheel 0.37.1
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/octavia-cli/.venv/bin/python -m pip install --upgrade pip' command.
> Task :octavia-cli:installLocalReqs FAILED
[python] .venv/bin/python -m pip install .[dev,tests]
Processing /data/macmv-programming/fauna/airbyte-fauna/octavia-cli
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
WARNING: octavia-cli 0.39.41 does not provide the extra 'dev'
Processing ./build/airbyte_api_client
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/data/macmv-programming/fauna/airbyte-fauna/octavia-cli/build/airbyte_api_client'
WARNING: You are using pip version 21.3.1; however, version 22.2.2 is available.
You should consider upgrading via the '/data/macmv-programming/fauna/airbyte-fauna/octavia-cli/.venv/bin/python -m pip install --upgrade pip' command.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':octavia-cli:installLocalReqs'.
> Python call failed: .venv/bin/python -m pip install .[dev,tests]
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See docs.gradle.org /7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 8s
13 actionable tasks: 3 executed, 10 up-to-date
/data/macmv-programming/fauna/airbyte-fauna $
The weird thing with this is at the start, I鈥檓 using pip 22, and half way through it starts using pip 21. I manually deleted the .venv directory and re-ran it, but it got to the same point, where it managed to downgrade pip. I鈥檓 not sure if this is the problem or not, but it definitely seems related.
I鈥檓 running arch linux, and uname -a gives this:
Linux neils-pc 5.19.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 05 Sep 2022 18:09:09 +0000 x86_64 GNU/Linux
Result of python --version:
Python 3.10.6
Result of java --version
openjdk 11.0.16.1 2022-08-12
OpenJDK Runtime Environment (build 11.0.16.1+1)
OpenJDK 64-Bit Server VM (build 11.0.16.1+1, mixed mode)
I also tried with java 17, and it gave the same output.
All of commands were run on the source-fauna branch, on commit 61408e841.
Also I modified the output of the gradle commands, as the forumn doesn鈥檛 allow multiple links at a time, so the gradle help links are invalid.