Not seeing file for import locally for docker

Connector Questions & Issues
,
1

I am just trying to test out airbyte, I am loading up a json, I want to do some data manipulation and export it back out to json. All done locally with while running on docker.

Complete Error:

Failed to load file:///tmp/airbyte_local/data_board.json: FileNotFoundError(2, ‘No such file or directory’) Traceback (most recent call last): File “/airbyte/integration_code/source_file/source.py”, line 101, in check with client.reader.open(): File “/airbyte/integration_code/source_file/client.py”, line 86, in open self._file = self._open() File “/airbyte/integration_code/source_file/client.py”, line 130, in _open return smart_open.open(self.full_url, **self.args) File “/usr/local/lib/python3.9/site-packages/smart_open/smart_open_lib.py”, line 177, in open fobj = _shortcut_open( File “/usr/local/lib/python3.9/site-packages/smart_open/smart_open_lib.py”, line 363, in _shortcut_open return _builtin_open(local_path, mode, buffering=buffering, **open_kwargs) FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/airbyte_local/data_board.json’

A couple of things I’ve noticed/tried:

  • this is the closest similar looking issue I’ve seen but the solution doesn’t seem to be specific and it’s more focused around the export than the import: Local JSON destination folder empty · Issue #1895 · airbytehq/airbyte · GitHub
  • CSV file source - #8 by harshith recommends you use /tmp/airbyte_local. when I did this on Mac it puts it in /private/tmp/airbyte_local. I’ve tried both as source directories and neither work
  • I have added file sharing for /private/tmp/airbyte_local, it will not let me add it for /tmp/airbyte_local. you have to add via the Favorites List on Mac and that always maps to private
  • seems like there are multiple .env files in the repo and I don’t want to just start guessing and changing things

Question:
What additional setup needs to be done to run an file import → file export locally with docker ?
What attributes in what .env files need to be changed ?

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: MacOS… a
  • Memory / Disk: 16 gb
  • Deployment: N/A running locally on docker
  • Airbyte Version: 0.40.17 (I pulled from the .env in the root of the cloned repo)
  • Source name/version: local
  • Destination name/version: local
  • Step: import of data
  • Description: not seeing file for import locally for docker
2

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

3

Hey Don,

Thanks for the post and welcome to the community. Have you tried looking at this guide? https://docs.airbyte.com/operator-guides/browsing-output-logs/#csv-or-json-local-destinations-check-local-data-folder

It seems like Docker on Mac has some extra caveats that might make it a bit awkward in accessing local files.

4

Hey @dmc2015

I had similar issues and tried many things until I found that I had to add /private Docker Desktop “File Sharing”.

I added it to the docs wherever I found it to be helpful, see for example in CSV or JSON Local Destination docs:

image
image1246×794 91.5 KB