Summary
When using the Airbyte Operator on Kubernetes, encountering a 404 error when trying to access /api/v1/connections/sync endpoint. Confusion between using localhost and api-server as host. Questioning if OSS installations should use HTTP requests instead of the Operator.
Question
Hi everyone
I’ve deployed Airbyte on Kubernetes and am trying to use Airbyte Operator as described https://docs.airbyte.com/operator-guides/using-the-airflow-airbyte-operator|here.
The docs uses generic localhost as the host, but in the API docs states that Kubernetes users should use api-server, but when using it, I got the following error
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <http://airbyte-airbyte-api-server-svc.airbyte.svc.cluster.local:80/api/v1/connections/sync>
I didn’t find the /connections/sync in the doc, but using this other https://airbyte.com/blog/orchestrating-airbyte-api-airbyte-cloud-airflow|doc, it mentions /v1/jobs with a POST and using shell inside Airflow I managed to trigger Airbyte
Am I missing something using the Operator or are OSS installments supposed to use HTTP requests ?
The first mentions that the Operator should not be used for Airbyte Cloud, but it doesn’t mention its usage for OSS
For historic reasons, the Airbyte operator is designed to work with the internal Config API rather than the newer Airbyte API and is therefore not intended or designed for orchstrating Airbyte Cloud. As an alternative, it is possible to make use of Airflow's HTTP operators with both Airbyte OSS and Airbyte Cloud. This approach is described in Using the new Airbyte API to orchestrate Airbyte Cloud with Airflow.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.