Increasing ephemeral storage limits for source and destination pods in Airbyte helm chart on Kubernetes

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is looking to increase ephemeral storage limits for source and destination pods in Airbyte deployed on Kubernetes using the Airbyte helm chart. They are also seeking information on where to find the code that generates templates for creating source and connector pods.


Question

Hey guys :wave:
We’ve deployed airbyte on k8s using the airbyte helm chart (from <Deploy Airbyte on Kubernetes using Helm | Airbyte Documentation guide>), we’d like to increase the ephemeral storage limits for all source and destination pods. Is this possible using the helm chart? The only references i’ve been able to find relating to resources is https://github.com/airbytehq/airbyte-platform/blob/156f8f998b56ca9a2910f2c16229807e6bc8af82/charts/airbyte/templates/env-configmap.yaml#L101-L104|here
Also could someone please send a link to a line in the code which generates the templates for creating the source and connector pods ? (it would be really nice to know how this works)


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["increasing-ephemeral-storage-limits", "source-pods", "destination-pods", "airbyte-helm-chart", "kubernetes", "code-templates"]
2

https://airbytehq.slack.com/archives/C021JANJ6TY/p1727299403946419?thread_ts=1727295395.747489&amp;cid=C021JANJ6TY

3

Thanks, and it looks like <airbyte-platform/airbyte-workload-launcher/src/main/kotlin/pods/KubePodLauncher.kt at 156f8f998b56ca9a2910f2c16229807e6bc8af82 · airbytehq/airbyte-platform · GitHub sends> a request to k8s to make the pods?

4

also is there a way to configure the ephemeral storage requests for the pods from the helm chart?
To somehow supply custom config from the helm chart <airbyte-platform/airbyte-workload-launcher/src/main/kotlin/pods/factories/ConnectorPodFactory.kt at main · airbytehq/airbyte-platform · GitHub this >

5

Yes, it sends request to k8s that creates pods.

I don’t think so. It will require extending existing code.

6

By the way, why do you need more ephemeral storage?

7

Our source and destination pods are using more than what’s currently allocated by default

8

Is it possible to request for this feature to be added in to a later release?

9

You can always add issue here: https://github.com/airbytehq/airbyte/issues/new/choose

I can see that something is going on related to volumes https://github.com/airbytehq/airbyte-platform/commit/140e45ae0b2dca3440645a25ccd66583c5f43749