Configuring Airbyte custom connector to pass args from Airflow Operator

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Configuring Airbyte custom connector to receive arguments from Airflow Operator for fetching data from a specific API endpoint


Question

Hello together, I try to fetch data from https://test.de/api/{{ config[‘product’] }}/{{ config[‘date’] }}/result via Airbyte, where product could be one of 8 values and date is a custom date. I want to trigger the job via Airflow. How do I have to configure the Airbyte custom connector so that I can pass the args product and date via Airflow Operator? Is this possible at all?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["airbyte-custom-connector", "airflow-operator", "api", "data-fetching"]
2

The current Airflow Operator only supports trigger a job based on the connection ID.
If you want to dynamic do this, probably you need to use a REST API operator to change the info /source/update and trigger the sync using the Airbyte Airflow Operator.

3

Thanks. What do you mean by /source/update? Where do I find it?

4

The Airbyte API https://reference.airbyte.com/reference/start