Summary
User is facing difficulties accessing a CSV file in Airbyte after installation using abctl on MacOS. The confusion arises from Docker Desktop’s management of Docker Compose and issues related to configuring paths in a .env file for local file access.
Question
Hello! I am new to Airbyte. I think I’ve installed it correctly using the abctl install on a MacOS. I’m a little confused because I do have Docker Desktop installed and abctl is running as a container there, but I see that Docker Compose is deprecated and isn’t part of the abctl install? In any case, I’m trying to create a very simple connection to a csv file locally. No matter what I try or where I put the file, Airbyte is not finding it. I asked over in the ask-ai channel about this issue and one of the recommendations was to configure a .env file with these paths defined:
• LOCAL_ROOT
• LOCAL_DOCKER_MOUNT
• HACK_LOCAL_ROOT_PARENT
It mentions putting the file in the /tmp/airbyte_local path. This path was not present as part of the install so I manually created it and restarted the container. I added that path to LOCAL_ROOT and LOCAL_DOCKER_MOUNT and its parent directory to HACK_LOCAL_ROOT_PARENT. When I try to use docker cp to copy my file to this location it isn’t recognizing that path. What am I missing here? There has to be an easier way to get this up and running.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
['abctl', 'docker-desktop', 'csv-file', 'local-root', 'local-docker-mount', 'hack-local-root-parent', 'docker-cp']
You can check this thread https://airbytehq.slack.com/archives/C021JANJ6TY/p1726562874434619
Basically after switching from docker compose to abctl those local connectors got broken.
I haven’t tested it yet, but recently (in v1.2.0) this appeared in values.yaml configuration
https://github.com/airbytehq/airbyte-platform/blob/86430ee8e57375497d5295ab9dc52f84b81f1699/charts/airbyte/values.yaml#L159-L161
I found a way to make it work
Create values.yaml
with a configuration like this:
jobs:
local:
volume:
enabled: true```
Run `abctl local install --values values.yaml`
Restart deployment `kubectl rollout restart deployment airbyte-abctl-workload-launcher`
Create `pv.yaml` with a definition:
```apiVersion: v1
kind: PersistentVolume
metadata:
name: airbyte-local
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 500Mi
hostPath:
path: /var/local-path-provisioner/airbyte-local
type: DirectoryOrCreate
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
volumeMode: Filesystem ```
And create `pvc.yaml` with a definition
```apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: airbyte-local-pvc
namespace: airbyte-abctl
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
storageClassName: standard
volumeMode: Filesystem
volumeName: airbyte-local```
Apply changes `kubectl apply -f pv.yaml -f pvc.yaml`
In Local JSON destination connector I set `Destination Path` to `/data` (for source connector I used Faker)
After running synchronization I found data in `~/.airbyte/abctl/data/airbyte-local/`
Thank you! I will give these a try and let you know.
This command gives me an error:
kubectl rollout restart deployment airbyte-abctl-workload-launcher
E1208 15:53:57.240806 72883 memcache.go:265] couldn’t get current server API group list: Get “http://localhost:8080/api?timeout=32s”: dial tcp [::1]:8080: connect: connection refused
Similar error for this command as well:
kubectl apply -f pv.yaml -f pvc.yaml
error validating “pv.yaml”: error validating data: failed to download openapi: Get “http://localhost:8080/openapi/v2?timeout=32s”: dial tcp [::1]:8080: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
error validating “pvc.yaml”: error validating data: failed to download openapi: Get “http://localhost:8080/openapi/v2?timeout=32s”: dial tcp [::1]:8080: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
But I restarted through Docker Desktop. When I tried to connect to my source, it took a lot longer this time, which I initially thought was promising, but then I got an error about Airflow being unavailable. I get the same error when I try to change the JSON destination path. Also, I noticed in the documentation, it makes a mention that this doesn’t work with kubernetes. But isn’t the abctl installation using kubernetes?
You need to set context for kubectl
first
Install kubectx + kubens (https://github.com/ahmetb/kubectx)
then execute commands:
KUBECONFIG=~/.airbyte/abctl/abctl.kubeconfig kubectl config view --flatten > ~/.kube/config
kubectx kind-airbyte-abctl
kubens airbyte-abctl```
After setting context, you should be able run steps above.
On the diagram you can see architecture of Airbyte deployment with `abctl`