Summary
This message is asking about how to set the LOCAL_ROOT environment variable in the context of using the Local CSV destination with ‘abctl’ local dev setup.
Question
Hi folks, is anyone using Local CSV destination with ‘abctl’ local dev setup? Docs mention LOCAL_ROOT env var and I’m curious how to set that in this context. https://docs.airbyte.com/integrations/destinations/csv
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["local-csv-destination", "abctl", "local-dev-setup", "local-root-env-var"]
I used Local JSON some time ago, but it stopped working properly with non-root Docker containers for old docker-compose way. You got my attention with your question. I’ll check if it’s going to work with abctl. Tricky part with abctl is that it uses kind cluster which is Kubernetes on Docker and in connector page there is:
> DANGER
> This destination is meant to be used on a local workstation and won’t work on Kubernetes
Thanks <@U05JENRCF7C> - yeah I saw that warning too and wondered if it applied; I tried looking in a couple of the k8s containers in case the file was “local” to one of those (rather than the host) but couldn’t find anything.
But yeah, given that abctl seems to be the recommended way to do local development, it seems odd to even have a “local CSV” destination if it simply isn’t going to work at all. 
I tried to do the same, but I couldn’t find where Local CSV saves files.
Some connectors (like Local JSON, Local CSV) supported by community are broken, and they are waiting for contributions.
Okay, I found that those files are written on destination connector pod file system, but data are gone when synchronization is done
Ah, good find! I wonder if there’s a way to mount that on the host, watch for the file, and then ‘snatch’ it before it disappears or something.
Or maybe this just needs to get escalated as a bug report with Airbyte.
for sure there is a way, because under /var/local-path-provisioner/ in airbyte-abctl-control-plane Docker container there are 2 folders-mounts that match PersistentVolumes and on host file system you can find it under ~/.airbyte/abctl/data/
Someone else beat me to writing up the bug report about local CSV destination & unclear docs w.r.t. abctl - https://github.com/airbytehq/airbyte/issues/45645
So this means, that there is currently no way to get files of the local system to airbyte when running through abctl?
I saw the local folder ~/.airbyte/abctl/data/ aswell (although i am running on windows and its different there), but it looks like there is data in it and i wouldnt know, where i can place data there for the connector
Thanks for the hint. I am currently trying to setup a simple openssh server on the same windows vm as airbyte is running on. So far that worked and i was able to add the sftp server as a source, but i am not able to add it as a destination. I keep getting “permission denied” errors. Have you tried to approach already and faced this issue?
I haven’t tried it yet. Are you able to sftp in and read and write (without using Airbyte, just to confirm the sftp server is working)?
Yes, thats working as expected. Also as a source i have no problem using it, its just the destination. Although i can see, that airbyte creates some json files there, probably to test the connection