Yeah… I did some trials too but without more success…
Thanks so much for helping here
I managed to test connectivity between Airflow and Airbyte. Host is different because I installed Airflow on kind cluster created by abctl
To unlock “Test ” button I needed to set in values.yaml for Airflow helm chart
- name: "AIRFLOW__CORE__TEST_CONNECTION"
value: "Enabled"```
https://github.com/apache/airflow/blob/b344cc1474de8f150773bd01e78514788ee88cc7/providers/src/airflow/providers/airbyte/hooks/airbyte.py#L183-L192|airbyte.py#L183-L192 test_connection
sends only request to health endpoint
Hmm
Weird that my previous test didn’t work then…
I will use the test_connection to see if the result is positive at least
I’ve managed to allow test connection but nothing is happening…
In cases like this you can open Network tab in browser’s developer tools to check if request is hanging
https://developer.chrome.com/docs/devtools/open
https://firefox-source-docs.mozilla.org/devtools-user/
yes, nothing is happening
I added the env var in airbyte.config
Actually, I can see the requests to the server if I refresh the page. Were you asked to confirm the test when you request a connect test? When I check the response preview, I see User confirmation needed
.
You are on your own here as I don’t have much experience with Airflow. However, you can still configure DAG based on that saved connection
yeah, totally understand. The issue is more related to airflow…
I managed to get a positive response for the test finally… following the same schema you did. However, very difficult to reproduce. I barely understand the logic…
I must understand why the airbyte operators don’t work now when used in a dag.
I tried to bypass the airbyte operator with a SimpleHTTPOperator to airbyte endpoint and it worked perfectly.
I have exactly the same error as this <Issues · apache/airflow · GitHub issue> actually
SimpleHTTPOperator works for you, so it’s good, but for this issue I guess some contribution is necessary
Yes indeed. But I would need to find a way to replace AirbyteJobSensor
as well. Didn’t have time to check this part yet.