Gradle method "RepositoryHandler.jcenter() has been deprecated" error when running Standard Tests

Connector Development
1

When trying to run Standard Tests from airbyte root repository, I see an error that the RepositoryHandler.jcenter() method has been deprecated

has anyone seen this error before?

I’m running version * 0.32.0-alpha

Logs:

Building all of Airbyte.
/Users/Meduro/airbyte/airbyte-integrations/connectors

> Configure project :buildSrc
The RepositoryHandler.jcenter() method has been deprecated. This is scheduled to be removed in Gradle 8.0. JFrog announced JCenter's sunset in February 2021. Use mavenCentral() instead. Consult the upgrading guide for further information: https://docs.gradle.org/7.2/userguide/upgrading_version_6.html#jcenter_deprecation
        at build_ecm5b00t9l8g84tvzxs5kny4b$_run_closure1.doCall(/Users/Meduro/airbyte/buildSrc/build.gradle:6)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/Meduro/airbyte/airbyte-cdk/python/build.gradle' line: 2

* What went wrong:
Could not compile build file '/Users/Meduro/airbyte/airbyte-cdk/python/build.gradle'.
> startup failed:
  build file '/Users/Meduro/airbyte/airbyte-cdk/python/build.gradle': 2: only id(String) method calls allowed in plugins {} script block
  
  See https://docs.gradle.org/7.2/userguide/plugins.html#sec:plugins_block for information on the plugins {} block
  
   @ line 2, column 5.
         id 'airbyte-python'    id 'airbyte-docker'
         ^
  
  1 error
2

Are you with 0.32 tag version? If so, please upgrade to latest version. What tests are you trying to run?

3

Hi marcosmarxm , i’ve upgraded to * 0.39.26-alpha and I still have this issue

I’m running Standard Tests as instructed in this article Source Acceptance Tests Reference | Airbyte Documentation

4

when running

python -m pytest -p integration_tests.acceptance --pdb --last-failed

I get the error:

ImportError while importing test module '/Users/Meduro/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.venv/lib/python3.8/site-packages/_pytest/python.py:578: in _importtestmodule
    mod = import_path(self.fspath, mode=importmode)
.venv/lib/python3.8/site-packages/_pytest/pathlib.py:524: in import_path
    importlib.import_module(module_name)
../../../../opt/anaconda3/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:961: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:783: in exec_module
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
../../bases/source-acceptance-test/source_acceptance_test/tests/__init__.py:5: in <module>
    from .test_core import TestBasicRead, TestConnection, TestDiscovery, TestSpec
.venv/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
    exec(co, module.__dict__)
../../bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:16: in <module>
    from airbyte_cdk.models import (
E   ImportError: cannot import name 'TraceType' from 'airbyte_cdk.models' (/Users/Meduro/airbyte/airbyte-integrations/connectors/source-healthsites/.venv/lib/python3.8/site-packages/airbyte_cdk/models/__init__.py)
5

Did you follow the commands in the connector README inside the connector page? Are you in the connector folder when running those commands?

From this connector directory, create a virtual environment:python -m venv .venvThis will generate a virtualenv for this module in .venv/. Make sure this venv is active in your development environment of choice. To activate it from the terminal, run:source .venv/bin/activatepip install -r requirements.txt

you must install requirrements and pyenv

6

Hi ,
yes the venv was created according to the instructions, requirements were installed and this command was run from within the source directory:

python -m pytest -p integration_tests.acceptance --pdb --last-failed

the above command threw the import error:

ImportError while importing test module '/Users/Meduro/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py'.

however, the original posted error came from a comand run in the airbyte root:

./gradlew :airbyte-integrations:connectors:source-<source-name>:integrationTest.

which threw the error:

The RepositoryHandler.jcenter() method has been deprecated. This is scheduled to be removed in Gradle 8.0. JFrog announced JCenter's sunset in February 2021. Use mavenCentral() instead. Consult the upgrading guide for further information: https://docs.gradle.org/7.2/userguide/upgrading_version_6.html#jcenter_deprecation
        at build_ecm5b00t9l8g84tvzxs5kny4b$_run_closure1.doCall(/Users/Meduro/airbyte/buildSrc/build.gradle:6)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)
7

If it provides more clarity, here is the entire output when I try to run python -m pytest -p integration_tests.acceptance from inside the root of the connector

`(.venv) (base) Meduro@Meduro-2 source-healthsites % python -m pytest -p integration_tests.acceptance
Test session starts (platform: darwin, Python 3.8.5, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/Meduro/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting …
― ERROR collecting airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py ―
ImportError while importing test module ‘/Users/Meduro/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py’.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.venv/lib/python3.8/site-packages/_pytest/python.py:578: in _importtestmodule
mod = import_path(self.fspath, mode=importmode)
.venv/lib/python3.8/site-packages/_pytest/pathlib.py:524: in import_path
importlib.import_module(module_name)
…/…/…/…/opt/anaconda3/lib/python3.8/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
:1014: in _gcd_import
???
:991: in _find_and_load
???
:961: in _find_and_load_unlocked
???
:219: in _call_with_frames_removed
???
:1014: in _gcd_import
???
:991: in _find_and_load
???
:975: in _find_and_load_unlocked
???
:671: in _load_unlocked
???
:783: in exec_module
???
:219: in _call_with_frames_removed
???
…/…/bases/source-acceptance-test/source_acceptance_test/tests/init.py:5: in
from .test_core import TestBasicRead, TestConnection, TestDiscovery, TestSpec
.venv/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
exec(co, module.dict)
…/…/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:16: in
from airbyte_cdk.models import (
E ImportError: cannot import name ‘TraceType’ from ‘airbyte_cdk.models’ (/Users/Meduro/airbyte/airbyte-integrations/connectors/source-healthsites/.venv/lib/python3.8/site-packages/airbyte_cdk/models/init.py)

― ERROR collecting airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py ―
ImportError while importing test module ‘/Users/Meduro/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py’.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.venv/lib/python3.8/site-packages/_pytest/python.py:578: in _importtestmodule
mod = import_path(self.fspath, mode=importmode)
.venv/lib/python3.8/site-packages/_pytest/pathlib.py:524: in import_path
importlib.import_module(module_name)
…/…/…/…/opt/anaconda3/lib/python3.8/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
:1014: in _gcd_import
???
:991: in _find_and_load
???
:961: in _find_and_load_unlocked
???
:219: in _call_with_frames_removed
???
:1014: in _gcd_import
???
:991: in _find_and_load
???
:975: in _find_and_load_unlocked
???
:671: in _load_unlocked
???
:783: in exec_module
???
:219: in _call_with_frames_removed
???
…/…/bases/source-acceptance-test/source_acceptance_test/tests/init.py:5: in
from .test_core import TestBasicRead, TestConnection, TestDiscovery, TestSpec
.venv/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
exec(co, module.dict)
…/…/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:16: in
from airbyte_cdk.models import (
E ImportError: cannot import name ‘TraceType’ from ‘airbyte_cdk.models’ (/Users/Meduro/airbyte/airbyte-integrations/connectors/source-healthsites/.venv/lib/python3.8/site-packages/airbyte_cdk/models/init.py)

― ERROR collecting airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py ―
ImportError while importing test module ‘/Users/Meduro/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py’.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.venv/lib/python3.8/site-packages/_pytest/python.py:578: in _importtestmodule
mod = import_path(self.fspath, mode=importmode)
.venv/lib/python3.8/site-packages/_pytest/pathlib.py:524: in import_path
importlib.import_module(module_name)
…/…/…/…/opt/anaconda3/lib/python3.8/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
:1014: in _gcd_import
???
:991: in _find_and_load
???
:961: in _find_and_load_unlocked
???
:219: in _call_with_frames_removed
???
:1014: in _gcd_import
???
:991: in _find_and_load
???
:975: in _find_and_load_unlocked
???
:671: in _load_unlocked
???
:783: in exec_module
???
:219: in _call_with_frames_removed
???
…/…/bases/source-acceptance-test/source_acceptance_test/tests/init.py:5: in
from .test_core import TestBasicRead, TestConnection, TestDiscovery, TestSpec
.venv/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
exec(co, module.dict)
…/…/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:16: in
from airbyte_cdk.models import (
E ImportError: cannot import name ‘TraceType’ from ‘airbyte_cdk.models’ (/Users/Meduro/airbyte/airbyte-integrations/connectors/source-healthsites/.venv/lib/python3.8/site-packages/airbyte_cdk/models/init.py)

{“type”: “LOG”, “log”: {“level”: “INFO”, “message”: “/Users/Meduro/airbyte/airbyte-integrations/connectors/source-healthsites - SAT run - bd64d5c02e0dbba0b8719ced4acbb47909d45469 - FAILED”}}

======================= short test summary info ========================
FAILED …/…/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py
FAILED …/…/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py
FAILED …/…/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py
!!! Interrupted: 3 errors during collection !!!`

8

When specifying the python version as such

(.venv) (base) Meduro@Meduro-2 source-healthsites % python3.9 -m pytest -p integration_tests.acceptance

I was able to bypass the import errors above, however now I get an output indicating a system failure upon importing the name 'escape from jinja2

{"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1656497227410.329, "error": {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "cannot import name 'escape' from 'jinja2' (/usr/local/lib/python3.9/site-packages/jinja2/__init__.py)", "stack_trace":...
9

I searched on the web and was suggested to upgrade flask. After using pip3.9 to upgrade Flask,
I got this error:

{“type”: “TRACE”, “trace”: {“type”: “ERROR”, “emitted_at”: 1656500525251.6838, “error”: {“message”: “Something went wrong in the connector. See the logs for more details.”, “internal_message”: "cannot import name ‘get_current_traceback’ from ‘werkzeug.debug.tbtools’

Then I went and changed get_current_traceback to DebugTraceback in /usr/local/lib/python3.9/site-packages/dash/dash.py\", line 22

not sure why that was necessary in the first place however…

10

Screen Shot 2022-06-29 at 14.18.04
Screen Shot 2022-06-29 at 14.18.041326×684 77.4 KB

I was still getting the original error from the root of airbyte

> startup failed:
  build file '/Users/Meduro/airbyte/airbyte-cdk/python/build.gradle': 2: only id(String), alias(Provider), or alias(ProviderConvertible) method calls allowed in plugins {} script block
  
  See https://docs.gradle.org/7.4/userguide/plugins.html#sec:plugins_block for information on the plugins {} block
  
   @ line 2, column 5.
         id 'airbyte-python'    id 'airbyte-docker'
         ^
  
  1 error

So I separated the lines in the plugins config as follows and that seems to have helped me make progress

plugins {
    id 'airbyte-python'    
    id 'airbyte-docker'
}
11

You can always run the same command building the docker container. Try ./gradlew airbyte-integrations:connectors:source-your-connector-name:integrationTest

12

Hi marcos

thanks but I mentioned that I’ve run this command already from the airbyte root, which gave the original error I posted

however after making these recent changes I am now seeing a different error related to a previously developed connector in the repo

Can you make sense of what it means when it says the “project” with path ':airbyte-integrations:bases:base-python' cannot be found in a different connector?

(base) Meduro@Meduro-2 airbyte % ./gradlew :airbyte-integrations:connectors:source-healthsites:integrationTest      
Building all of Airbyte.
/Users/Meduro/airbyte/airbyte-integrations/connectors
Type-safe dependency accessors is an incubating feature.

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/Meduro/airbyte/airbyte-integrations/connectors/source-kobo-collect/build.gradle' line: 2

* What went wrong:
An exception occurred applying plugin request [id: 'airbyte-python']
> Failed to apply plugin 'airbyte-python'.
   > A problem occurred configuring project ':airbyte-integrations:connectors:source-kobocollect'.
      > A problem occurred evaluating project ':airbyte-integrations:connectors:source-kobocollect'.
         > Project with path ':airbyte-integrations:bases:base-python' could not be found in project ':airbyte-integrations:connectors:source-kobocollect'.
13

Looks you have other connector called kobocollect that is causing the problem

14

this is a different problem than the original problem but yes that is obvious

can you help me understand what Project with path ':airbyte-integrations:bases:base-python' could not be found in project ':airbyte-integrations:connectors:source-kobocollect'. means

15

found a discussion forum on the topic for your reference

16

Namer, sorry the long go-back messages. Please provide your OS, what Airbyte version OR head from github you’re using to run your code.
This code is used daily to generate new connectors by the team.
Also all steps you had done so far: use generator, build connector… etc ← my example

17

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.