Summary
Inquiring about the possibility of defining a local filesystem source when deploying Airbyte on Kubernetes using a Helm chart.
Question
local filesystem with docker desktop kuberenets. is us possible to get it work?
I managed to use local filesystem when running local version of airbyte (running dockers)
but when i used a helm chart to deploy on k8s I am unable to define a local filesystem source.
so the quesions - is it possible?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["local-filesystem", "docker-desktop", "kubernetes", "helm-chart", "source-definition"]
Yes, it is possible to use a local filesystem with Airbyte when deploying on Kubernetes. When you set up a pipeline using one of the local File based destinations (CSV or JSON), Airbyte writes the resulting files containing the data in the special /local/ directory in the container. By default, this volume is mounted from /tmp/airbyte_local on the host machine. To retrieve the local data files, you need to navigate to this local folder on the filesystem of the machine running the Airbyte deployment. Please make sure that Docker Desktop has access to /tmp (and /private on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with “File sharing” in Settings -> Resources -> File sharing -> add the one or two above folder and hit the “Apply & restart” button. Here is an example of how you can run through docker commands as a proxy: bash #!/usr/bin/env bash echo "In the container:" docker run -it --rm -v /tmp/airbyte_local:/local busybox find /local echo "" echo "On the host:" find /tmp/airbyte_local This will produce an output similar to: bash In the container: /local /local/data /local/data/exchange_rate_raw.csv On the host: /tmp/airbyte_local /tmp/airbyte_local/data /tmp/airbyte_local/data/exchange_rate_raw.csv You can find more details in the Airbyte documentation.
thanks, well actually I am trying to work with a source not destination. my files on the host are in /tmp/airbyte_local
should I mode it to
in the source I am looking for
/local/xx.csv