Summary
Error message indicates a connection issue between Airbyte and Airflow using abctl. The error is caused by a failed connection due to connection refusal on localhost. Suggestions for troubleshooting may include checking network configurations, ensuring services are running, and verifying connection settings.
Question
Hey everyone! I’m getting this error trying to connect airbyte with airflow using abctl. Any idea on how can I solve it?
HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/public/v1/health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c4b7957aa20>: Failed to establish a new connection: [Errno 111] Connection refused'))
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["error", "connecting", "airbyte", "airflow", "abctl", "connection-refused"]
Airflow is running on Docker or on host machine?
You need to connect airflow container to the same network in Docker docker network connect kind <airflow-container>
(you need to replace <airflow-container> with airflow container name, check with docker ps)
Then, change localhost:8000 to airbyte-abctl-control-plane
So both airbyte and airflow must have the same network. How can I check the network with abctl?
docker inspect airbyte-abctl-control-plane
kind is the name of the network used by airbyte-abctl-control-plane
<@U05JENRCF7C> when I changed to the abctl, my Minio connections stoped working. Could be the same problem?
Settings the network on the minio docker?
abctl creates airbyte-abctl-control-plane Docker container and runs kind cluster (https://kind.sigs.k8s.io/, think about kubernetes running in Docker container) in it.
I assume that you went from docker-compose based approach to abctl deployment, so for sure previous connections between containers stopped working.
https://docs.docker.com/engine/network/
https://docs.docker.com/engine/network/drivers/bridge/
bridge is the default network driver and also the name of the network
> When you start Docker, a <Bridge network driver | Docker Docs bridge network> (also called bridge) is created automatically, and newly-started containers connect to it unless otherwise specified. You can also create user-defined custom bridge networks. User-defined bridge networks are superior to the default bridge network.
yep, I was running with a compose
First time using a kubernetes like structure
hey <@U05JENRCF7C> managed to fix connections with my lakehouse (minio and others). But airflow still didn’t work… any idea? Still the same error
what error do you have at this moment after connecting airflow container to kind network?
on the airflow connection I don’t get any error anymore, but on the dag I got thoses:
Only inserting the token url
[2024-10-21, 22:12:32 -03] {taskinstance.py:1225} INFO - Marking task as FAILED. dag_id=***-with-docker, task_id=airbyte_faker, run_id=manual__2024-10-22T01:12:30.448426+00:00, execution_date=20241022T011230, start_date=20241022T011232, end_date=20241022T01123```
With the host:
```airflow.exceptions.AirflowException: HTTPConnectionPool(host='172.21.0.2', port=8000): Max retries exceeded with url: /api/public/v1/applications/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd91b261550>: Failed to establish a new connection: [Errno 111] Connection refused'))```
WIthout host and token url:
```airflow.exceptions.AirflowException: Unexpected status code 500 from token endpoint```
I tried to ping the airbyte `kind` from airflow, but when I set the port, it stop working
what exactly you typed in terminal for pinging?
tried three: curl http://localhost, docker ip, external ip:port
without the port, I got the nginx error, 404