Using Airflow to trigger Airbyte connection

Yeah… I did some trials too but without more success…

Thanks so much for helping here :wink:

I managed to test connectivity between Airflow and Airbyte. Host is different because I installed Airflow on kind cluster created by abctl

To unlock “Test :rocket:” button I needed to set in values.yaml for Airflow helm chart

  - name: "AIRFLOW__CORE__TEST_CONNECTION"
    value: "Enabled"```

https://github.com/apache/airflow/blob/b344cc1474de8f150773bd01e78514788ee88cc7/providers/src/airflow/providers/airbyte/hooks/airbyte.py#L183-L192|airbyte.py#L183-L192 test_connection sends only request to health endpoint

Hmm :thinking_face:
Weird that my previous test didn’t work then…

I will use the test_connection to see if the result is positive at least

I’ve managed to allow test connection but nothing is happening…

In cases like this you can open Network tab in browser’s developer tools to check if request is hanging

https://developer.chrome.com/docs/devtools/open
https://firefox-source-docs.mozilla.org/devtools-user/

yes, nothing is happening

I added the env var in airbyte.config

Actually, I can see the requests to the server if I refresh the page. Were you asked to confirm the test when you request a connect test? When I check the response preview, I see User confirmation needed.

You are on your own here as I don’t have much experience with Airflow. However, you can still configure DAG based on that saved connection

yeah, totally understand. The issue is more related to airflow…

I managed to get a positive response for the test finally… following the same schema you did. However, very difficult to reproduce. I barely understand the logic…

I must understand why the airbyte operators don’t work now when used in a dag.

I tried to bypass the airbyte operator with a SimpleHTTPOperator to airbyte endpoint and it worked perfectly.

I have exactly the same error as this <Issues · apache/airflow · GitHub issue> actually

SimpleHTTPOperator works for you, so it’s good, but for this issue I guess some contribution is necessary

Yes indeed. But I would need to find a way to replace AirbyteJobSensor as well. Didn’t have time to check this part yet.