Cannot build docker images for python destination connector

Connector Development
1

I am trying to docker build a python destination connector. My connector requires me to add a python dependency named Authlib. I have added this to my setup.py file like the following:

MAIN_REQUIREMENTS = [
    "airbyte-cdk",
    "Authlib"
]

When I add this dependency, my docker build command (docker build . -t airbyte/destination-<my-destination>:dev) fails with the following error:

#10 55.79   Stored in directory: /root/.cache/pip/wheels/c9/c7/29/3cf0033406b18958d1a7b107c38be2405ef9c081b64a264fd5
#10 55.80   Building wheel for cffi (setup.py): started
#10 56.36   Building wheel for cffi (setup.py): finished with status 'error'
#10 56.38   error: subprocess-exited-with-error
#10 56.38
#10 56.38   × python setup.py bdist_wheel did not run successfully.
#10 56.38   │ exit code: 1
#10 56.38   ╰─> [36 lines of output]
#10 56.38       running bdist_wheel
#10 56.38       running build
#10 56.38       running build_py
#10 56.38       creating build
#10 56.38       creating build/lib.linux-x86_64-3.9
#10 56.38       creating build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
#10 56.38       running build_ext
#10 56.38       building '_cffi_backend' extension
#10 56.38       creating build/temp.linux-x86_64-3.9
#10 56.38       creating build/temp.linux-x86_64-3.9/c
#10 56.38       gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
#10 56.38       c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
#10 56.38          15 | #include <ffi.h>
#10 56.38             |          ^~~~~~~
#10 56.38       compilation terminated.
#10 56.38       error: command '/usr/bin/gcc' failed with exit code 1
#10 56.38       [end of output]
#10 56.38
#10 56.38   note: This error originates from a subprocess, and is likely not a problem with pip.
#10 56.38   ERROR: Failed building wheel for cffi
#10 56.38   Running setup.py clean for cffi
#10 56.87   Building wheel for pyrsistent (pyproject.toml): started
#10 61.12   Building wheel for pyrsistent (pyproject.toml): finished with status 'done'
#10 61.13   Created wheel for pyrsistent: filename=pyrsistent-0.18.1-cp39-cp39-linux_x86_64.whl size=72096 sha256=282850b44a98a8c13c9b117f9293ef217e00f7a9959072d45954629d700fdf60
#10 61.13   Stored in directory: /root/.cache/pip/wheels/87/fe/e6/fc8deeb581a41e462eafaf19fee96f51cdc8391e0be1c8088a
#10 61.13 Successfully built destination-zir PyYAML pendulum pyrsistent
#10 61.13 Failed to build cffi
#10 61.41 Installing collected packages: jsonref, wrapt, urllib3, typing-extensions, six, PyYAML, pytzdata, pyrsistent, pycparser, multidict, MarkupSafe, idna, dpath, charset-normalizer, certifi, backoff, attrs, yarl, sentry-sdk, requests, python-dateutil, pydantic, jsonschema, Jinja2, Deprecated, cffi, vcrpy, pendulum, cryptography, authlib, airbyte-cdk, destination-zir
#10 64.40   Running setup.py install for cffi: started
#10 64.94   Running setup.py install for cffi: finished with status 'error'
#10 64.95   error: subprocess-exited-with-error
#10 64.95
#10 64.95   × Running setup.py install for cffi did not run successfully.
#10 64.95   │ exit code: 1
#10 64.95   ╰─> [36 lines of output]
#10 64.95       running install
#10 64.95       running build
#10 64.95       running build_py
#10 64.95       creating build
#10 64.95       creating build/lib.linux-x86_64-3.9
#10 64.95       creating build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
#10 64.95       running build_ext
#10 64.95       building '_cffi_backend' extension
#10 64.95       creating build/temp.linux-x86_64-3.9
#10 64.95       creating build/temp.linux-x86_64-3.9/c
#10 64.95       gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
#10 64.95       c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
#10 64.95          15 | #include <ffi.h>
#10 64.95             |          ^~~~~~~
#10 64.95       compilation terminated.
#10 64.95       error: command '/usr/bin/gcc' failed with exit code 1
#10 64.95       [end of output]
#10 64.95
#10 64.95   note: This error originates from a subprocess, and is likely not a problem with pip.
#10 64.96 error: legacy-install-failure
#10 64.96
#10 64.96 × Encountered error while trying to install package.
#10 64.96 ╰─> cffi
#10 64.96
#10 64.96 note: This is an issue with the package mentioned above, not pip.
#10 64.96 hint: See above for output from the failure.
------
executor failed running [/bin/sh -c pip install --prefix=/install .]: exit code: 1

There might be a dependency that I have to add somewhere but I cannot figure out what and where.
Please note that without adding Authlib the docker build goes fine.

2

Hello,
This is indeed a dependency issue, the cffi package, which is probably an Authlib dependency.
You might need to customize the connector docker image to install additional system dependencies if this error happens during docker build. I found the following StackOverflow with some suggested solutions
I would also suggest trying executing the connector locally in a python environment first (this is explained in the generated readme file) and check if you have a similar error.

3

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.