How to get dbt transformations on a k8s deployed airbyte installation

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • Deployment: Kubernetes
  • Airbyte Version: What version are you using now?
  • Source name/version:
  • Destination name/version:
  • Description: How can I get the dbt transformations on a kubernetes deployed airbyte and using basic normalization? The normalization pod is create and runs, dynamically the dbt files are created on the pod, and then the pod is ended. Any way to get those dbt files?
2

Not completly sure if works similar to Docker deployment but you can access the dbt files inside the airbyte-server in /tmp/workspace/<job_id>/<attempt>/normalize folder.

3

There is no /tmp/workspace directory on the airbyte-server in k8s. The /tmp/workspace directory is present in the normalization jobs that get kicked off, but those volumes are ephemeral, only there for the life of the job while it is running, and so there is no way to look at the generated files later. We can’t change the volume configuration with config files either because the volumes are configured inside the java code.

4

Oh, looks it’s a limitation of Airbyte running on Kubernetes. Check this feature request: https://github.com/airbytehq/airbyte/issues/9123 which can help you when implemented. Also custom dbt transformation doesn’t work on Kubernetes https://github.com/airbytehq/airbyte/issues/5091