Summary
Airflow container managed by Kubernetes unable to network with Airbyte
Question
Hey all… has anyone tried triggering airbyte data sync from airflow. When I used the docker compose up method for installing airbyte then the sync worked because airflow was also set up using docker compose up. Now that I have started using abctl which is managed by kubernetes, my airflow container is unable to network with airbyte. I request for any guidance here, some directions that I can look into to finding a solution to this.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["triggering-airbyte-data-sync", "airflow", "abctl", "kubernetes", "networking-issue"]
abctl uses kind cluster (https://kind.sigs.k8s.io/) under the hood
you can check this thread as an inspiration how to ensure communication between containers by connecting them to the same network
https://airbytehq.slack.com/archives/C021JANJ6TY/p1723500644587079?thread_ts=1723495994.480679&cid=C021JANJ6TY
Thanks! <@U05JENRCF7C>
I used the docker network command and made my airflow container connect to kind network. When I execute a curl request to airbyte from airflow container it fails with the below error Failed to connect Couldn't connect to server . But the other way round works. I am able to perform a curl request to airflow from Airbyte. I tried performing a curl to Airbyte inside Airbyte container and it fails with the same error.
I ran this command inside my Airbyte container kubectl get nodes -o wide and I notice that the status of the container is ready and with an internal-ip set but no external-ip set.
Additional thing I notice is when I perform a curl with port 80, then I get
404 Not Found
404 Not Found
nginx
Is there something I am missing here?