Airflow for orchestrating Airbyte syncs and DBT transformations

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Exploring the use of Airflow to orchestrate Airbyte syncs and DBT transformations, comparing with Dagster.


Question

Hi all - anyone using airflow to orchestrate airbyte syncs and DBT transformations and have any good or bad experiences? we’re currently using Dagster as it was recommended to us by Airbyte. But there is some push to consolidate to airflow as we use it for other processes. On the surface it looks like airflow supports DBT and Airbyte integrations.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["airflow", "airbyte", "dbt", "orchestration", "dagster"]
2

I should mention we’re using dbt-core through CLI. I haven’t started a POC yet. I wanted to solicit advice from others who might have already implemented this, and understand any possible pain points.

3

<@U02TQLBLDU4> thanks for you input

4

The AirflowOperator is a simple wrapper to trigger your jobs so not big deal there. If you connect a DbtOperator as a following task I don’t see any big issues. Are you having troubles?