How To Set Workspace-Folder/Job-Id In Entrypoint Arguments For Custom Dbt Transformation?

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Linux (Amazon)
  • Memory / Disk: you can use something like 16Gb / 8 Gb
  • Deployment: Docker
  • Airbyte Version: 0.40.10
  • Source name/version: Zuora alpha 0.1.3
  • Destination name/version: Snowflake
  • Step: The issue is happening during sync
  • Description:
    I’d like to run my existing dbt project as the custome dbt transformation. To do that I need to provide my project path to the argument Entrypoint arguments for dbt cli to run the project.
    But because the WORKSPACE-FOLDER/JOB_ID changes at every run I can’t set it correctly. Is there a way to set the WORKSPACE-FOLDER/JOB_ID value dynamically? A variable maybe?
2022-10-05 13:39:54 dbt > Running: dbt deps --project-dir=transform/git_repo/my-dbt-project --profiles-dir=/data/**80/0**/my-dbt-project
2022-10-05 13:39:56 dbt > 13:39:56  Encountered an error:
2022-10-05 13:39:56 dbt > Runtime Error
2022-10-05 13:39:56 dbt >   fatal: Invalid --project-dir flag. Not a dbt project. Missing dbt_project.yml file
2

Hey @murat,

Thanks for the question. Have you followed the tutorial/guide outlined here: https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-dbt/

3

Hi @sajarin . Yes, I followed that documentation but I wasn’t able to run my existing dbt project models and got Not a dbt project. Missing dbt_project.yml file error.

Thanks to Marcelo Pio de Castro, he answered my question in the Slack channel. So to run my models in my existing dbt project I needed to set Entrypoint arguments for dbt cli to run the project as below:

run --select models/my-model.sql --project-dir=my-dbt-project-name

4

Thanks for coming back and posting the solution! Hope this helps other users as well.