Issues with CSV File Access in Airbyte on MacOS

Summary

User is facing difficulties accessing a CSV file in Airbyte after installation using abctl on MacOS. The confusion arises from Docker Desktop’s management of Docker Compose and issues related to configuring paths in a .env file for local file access.


Question

Hello! I am new to Airbyte. I think I’ve installed it correctly using the abctl install on a MacOS. I’m a little confused because I do have Docker Desktop installed and abctl is running as a container there, but I see that Docker Compose is deprecated and isn’t part of the abctl install? In any case, I’m trying to create a very simple connection to a csv file locally. No matter what I try or where I put the file, Airbyte is not finding it. I asked over in the ask-ai channel about this issue and one of the recommendations was to configure a .env file with these paths defined:
LOCAL_ROOT
LOCAL_DOCKER_MOUNT
HACK_LOCAL_ROOT_PARENT
It mentions putting the file in the /tmp/airbyte_local path. This path was not present as part of the install so I manually created it and restarted the container. I added that path to LOCAL_ROOT and LOCAL_DOCKER_MOUNT and its parent directory to HACK_LOCAL_ROOT_PARENT. When I try to use docker cp to copy my file to this location it isn’t recognizing that path. What am I missing here? There has to be an easier way to get this up and running.



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['abctl', 'docker-desktop', 'csv-file', 'local-root', 'local-docker-mount', 'hack-local-root-parent', 'docker-cp']

You can check this thread https://airbytehq.slack.com/archives/C021JANJ6TY/p1726562874434619

Basically after switching from docker compose to abctl those local connectors got broken.

I haven’t tested it yet, but recently (in v1.2.0) this appeared in values.yaml configuration
https://github.com/airbytehq/airbyte-platform/blob/86430ee8e57375497d5295ab9dc52f84b81f1699/charts/airbyte/values.yaml#L159-L161

I found a way to make it work :slightly_smiling_face:

Create values.yaml with a configuration like this:

  jobs:
    local:
      volume:
        enabled: true```
Run `abctl local install --values values.yaml`

Restart deployment `kubectl rollout restart deployment airbyte-abctl-workload-launcher`

Create `pv.yaml` with a definition:

```apiVersion: v1
kind: PersistentVolume
metadata:
  name: airbyte-local
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 500Mi
  hostPath:
    path: /var/local-path-provisioner/airbyte-local
    type: DirectoryOrCreate
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  volumeMode: Filesystem ```
And create `pvc.yaml` with a definition

```apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: airbyte-local-pvc
  namespace: airbyte-abctl
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
  storageClassName: standard
  volumeMode: Filesystem
  volumeName: airbyte-local```
Apply changes `kubectl apply -f pv.yaml -f pvc.yaml`

In Local JSON destination connector I set `Destination Path` to `/data` (for source connector I used Faker)

After running synchronization I found data in `~/.airbyte/abctl/data/airbyte-local/`

Thank you! I will give these a try and let you know. :heart:

This command gives me an error:
kubectl rollout restart deployment airbyte-abctl-workload-launcher
E1208 15:53:57.240806 72883 memcache.go:265] couldn’t get current server API group list: Get “http://localhost:8080/api?timeout=32s”: dial tcp [::1]:8080: connect: connection refused

Similar error for this command as well:
kubectl apply -f pv.yaml -f pvc.yaml
error validating “pv.yaml”: error validating data: failed to download openapi: Get “http://localhost:8080/openapi/v2?timeout=32s”: dial tcp [::1]:8080: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
error validating “pvc.yaml”: error validating data: failed to download openapi: Get “http://localhost:8080/openapi/v2?timeout=32s”: dial tcp [::1]:8080: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false

But I restarted through Docker Desktop. When I tried to connect to my source, it took a lot longer this time, which I initially thought was promising, but then I got an error about Airflow being unavailable. I get the same error when I try to change the JSON destination path. Also, I noticed in the documentation, it makes a mention that this doesn’t work with kubernetes. But isn’t the abctl installation using kubernetes?

You need to set context for kubectl first

Install kubectx + kubens (https://github.com/ahmetb/kubectx)
then execute commands:

KUBECONFIG=~/.airbyte/abctl/abctl.kubeconfig kubectl config view --flatten > ~/.kube/config
kubectx kind-airbyte-abctl
kubens airbyte-abctl```
After setting context, you should be able run steps above.
On the diagram you can see architecture of Airbyte deployment with `abctl`