Triggering downstream task based on Airbyte data stream in Airflow and dbt pipeline

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

How to trigger downstream task in Airflow and dbt pipeline based on Airbyte data stream availability


Question

Hello everyone, I need some suggestions to a POC implementation. I am using airbyte, airflow and dbt to build a pipeline. I am new to all these tools. I’m thinking of a case to trigger my downstream task only if my airbyte task retrieves any rows in the defined data stream. If not then not trigger the dbt tasks. Requesting for some suggestions to where I can start looking for and learn about these.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["poc-implementation", "airbyte", "airflow", "dbt", "data-stream", "trigger", "downstream-task"]
2

Probably you need to add an additional task to query your data warehouse after the sync finished.
airbyte >> bigquery_query >> dbt