Summary
The user is facing issues with the AirbyteJobSensor in Apache Airflow, receiving a 404 error when trying to retrieve a job from a connection. They are also inquiring about the compatibility between Apache Airflow providers for Airbyte and Airbyte version 0.50.17.
Question
Hi guys, how are you?
I have the following question: what is the path and parameters needed to obtain the job from a connection?
I’m using airbyte version 0.50.17
I’m also using airflow with airbyte operators,
AirbyteTriggerSyncOperator works fine, but AirbyteJobSensor fails:
Traceback (most recent call last):
File “/opt/python3.11/lib/python3.11/site-packages/airflow/providers/http/hooks/http.py”, line 197, in check_response
response.raise_for_status()
File “/opt/python3.11/lib/python3.11/site-packages/requests/models.py”, line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://AIRBYTE_URL443/v1/jobs/417194
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/opt/python3.11/lib/python3.11/site-packages/airflow/providers/airbyte/sensors/airbyte.py”, line 119, in execute
super().execute(context)
File “/opt/python3.11/lib/python3.11/site-packages/airflow/sensors/base.py”, line 257, in execute
raise e
File “/opt/python3.11/lib/python3.11/site-packages/airflow/sensors/base.py”, line 239, in execute
poke_return = self. poke(context)
^^^^^^^^^^^^^^^^^^^
File “/opt/python3.11/lib/python3.11/site-packages/airflow/providers/airbyte/sensors/airbyte.py”, line 89, in poke
job = hook.get_job(job_id=self.airbyte_job_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/python3.11/lib/python3.11/site-packages/airflow/providers/airbyte/hooks/airbyte.py”, line 217, in get_job
return self.run(
^^^^^^^^^
File “/opt/python3.11/lib/python3.11/site-packages/airflow/providers/http/hooks/http.py”, line 187, in run
return self. run_and_check(session, prepped_request, extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/python3.11/lib/python3.11/site-packages/airflow/providers/http/hooks/http.py”, line 238, in run_and_check
self. check_response(response)
File “/opt/python3.11/lib/python3.11/site-packages/airflow/providers/http/hooks/http.py”, line 201, in check_response
raise AirflowException(str(response.status_code) + “:” + response.reason)
airflow.exceptions.AirflowException: 404:Not Found
[2024-06-26, 19:04:55 UTC] {taskinstance.py:1401} INFO - Marking task as FAILED. dag_id=base_test2, task_id=airbyte_sensor_source_dest_example, execution_date=20240625T000000, start_date=20240626T190455, end_date=20240626T190455
[2024-06-26, 19:04:55 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 16482200 for task airbyte_sensor_source_dest_example (404:Not Found; 4521)
[2024-06-26, 19:04:55 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2024-06-26, 19:04:56 UTC] {taskinstance.py:2781} INFO - 0 downstream tasks scheduled from follow-on schedule check
Do you know which version of apache-airflow-providers-airbyte should work with airbyte 0.50.17 ?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.