Mounting local folder in Airbyte connector running in a pod

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

How to mount a local folder in an Airbyte connector running in a pod using abctl and referencing them in workers and servers using values.yaml


Question

Hi again,
So I developed a connector that access local csv.
I works well when I used docker compose deployed airbyte.
Now with abctl and connectors running in a pod how to mount the folder ?
There is a way with abctl to add additional volumes, but how to reference them in the running connector, in the workers and in the servers using values.yaml


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["mounting-local-folder", "airbyte-connector", "abctl", "pod", "values.yaml"]
2

Local file access in abctl (or generally with airbyte in k8s) doesn’t work at the moment. There’s no way to configure airbyte currently to support that mode (that I’m aware of anyway).

3

Thanks for you reply, so what about abctl --volume ?

4

That would be part of the solution maybe, but the trouble is that the system doesn’t create the appropriate volume mounts on the connector containers

5

arrgh so i should use cloud storage instead…

do you have any idea if this will be implemented, and when?

6

We’ve talked about implementing support for local access, yes. I can’t say when exactly that will happen.

7

<@U07FH2Y34A1> for abctl there is local-path-provisioner
https://airbytehq.slack.com/archives/C021JANJ6TY/p1726585982839499?thread_ts=1726562874.434619&amp;cid=C021JANJ6TY
that stores data for minio and db

I think creating another volume and mounting it, should work. For me the most tricky part is dynamic nature of jobs pods. Maybe passing something like extraVolumes in values.yaml?

8

Ya, last I looked, the problem was that the component that dynamically launches job pods doesn’t understand/support the idea of extra, local volumes + mounts.

9

last I looked was a few weeks ago, and the megapod (single job pod) project was finished since then, so it’s time for a fresh look

10

Do you remember which file/class is responsible for spawning pods for jobs?

11

I’m not too familiar with the new code, but these look related:
• https://github.com/airbytehq/airbyte-platform/blob/main/airbyte-workload-launcher/src/main/kotlin/pods/factories/ReplicationPodFactory.kt
• https://github.com/airbytehq/airbyte-platform/blob/main/airbyte-workload-launcher/src/main/kotlin/pods/factories/VolumeFactory.kt