Empty tmp/airbyte_local folder when running Airbyte locally with destination set to LOCAL CSV

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

When running Airbyte locally with destination set to LOCAL CSV, the tmp/airbyte_local folder is empty. Similar issue experienced by coworker.


Question

Does anyone knows why when i run airbyte locally and my destination is set to LOCAL CSV, my tmp/airbyte_local folder is empty? My coworker did the same process for another connector he is building and had the same issue


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["empty-folder", "airbyte-local", "local-csv", "connector"]
2

FYI, I tried to trace this back down just now and I don’t see why it’s not writing the files to my host machine’s /tmp/airbyte_local path, which is what the .env file seems to be saying it should do.

Not sure what happened because it did work at one point, but I’ve updated it and messed with custom connectors a lot since then.

3

Nice. I tried again to do local csv today and double checked the .env settings for the local mount and I still don’t see files showing up. I’m going to try local json, I know that worked at one point not long ago.

4

Hi, I’m a new user of Airbyte (open source local deployment) and having trouble figuring out how to map the local filesystem to read CSV files. I followed the documentation to set the .env file (pls see screenshot) but having no luck finding the file when I try to create a source on the GUI.

I’m not sure if I’m missing anything else that needs to be changed. Any help is much appreciated! Thanks!

5

You can see what it’s doing if you look at the docker-compose.yaml. LOCAL_ROOT can be set to some directory on your host machine. LOCAL_DOCKER_MOUNT should probably be left as /tmp/local_airbyte. Then, the comments seem to say that HACK_LOCAL_ROOT_PARENT needs to exist on the host filesystem and be a parent of LOCAL_ROOT. If that makes sense.

With mine as default, the files end up in /tmp/airbyte_local both inside and outside the docker container.

But, it does seem a little flaky, I ran a local export earlier and it didn’t give me a file although it succeeded, but now it’s working and I just used it.

6

I’m using wsl2 as well, makes sense, thanks

7

I’ve build a destination pointing to a dev environment so i actually bypassed the local issue, did nt got time to test the local csv solution

8

I’m setting my destination path to /banana for example, still empty, even looking in the folder with ls -lah

9

You’re looking in the storage for the container running airbyte I presume? I run it on wsl2 so it’s usually in the docker filesystem for that particular linux instantiation.

10

This worked again today. With local csv and local json. I just wiped and re-saved the path as /my-stuff and it shows up in the place that the docs say it should on the docker volume mounted to the airbyte container. But, I also recently did a git pull and re-started manually with the sh file (vs firing up the container), so it may have updated/replaced some bad configuration files.

Worst case, maybe try that, probably not a bad idea to do anyways to make sure stuff is up-to-date.

11

Sure, although I say that and did a test to a local directory myself last night and now can’t find the files so not totally sure. I know in the past I just dug around in the wsl2 linux mounts and found it. And the local csv local json connector even says in the docs where it’s supposed to be in wsl setup, but not there this time.