Using File Source in Kubernetes Cluster for Reading JSON Files from Mounted Volume

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is looking for information on how to use the File source in a Kubernetes cluster to read JSON files from a mounted volume. They are specifically asking about the configuration in the values.yaml file that manages sync jobs.


Question

I want to use the File source in my kubernetes cluster to read json files from a mounted volume. I can’t find any definitive answers to this. It seems like in 2022 this was not supported in Kubernetes. Hopefully this is no longer the case, but I can’t really see how to do it… Maybe in the worker.extraVolumes? Which section in the kubernetes values.yaml exactly manages the sync jobs?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["file-source", "kubernetes-cluster", "json-files", "mounted-volume", "values.yaml", "sync-jobs"]
2

Like others, I see that setting worker.extraVolumes does not impact the mounts for the sync job.

I see https://github.com/airbytehq/airbyte/blob/bb1adbe06654850e279334ed419909ad89620474/docs/operator-guides/configuring-connector-resources.md?plain=1#L5|notes on how to set CPU and other resource settings

So it looks like the OSS AirByte doesn’t allow me to work with mounted files, so it is completely useless for my use cases.

I’ve sunk too much time into AirByte and guess I’ll move on to some other solution.