Volume size increasing on airbyte-abctl-control-plane container

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Volume size of airbyte-abctl-control-plane container keeps increasing, despite using S3 storage setup. Seeking insights on the reason behind this.


Question

When using abctl, I see that there is a volume attached to airbyte-abctl-control-plane container and this container’s volume size keeps increasing. I have storage set-up to use S3, so does anyone know why the volume size keeps increasing over time on this volume? Thank you.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["volume-size", "airbyte-abctl-control-plane", "S3-storage", "container"]
2

During default installation 2 persistent volumes are created

  INFO    Persistent volume 'airbyte-volume-db' created
  INFO    Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' created
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' created```
`abctl` uses kind cluster (<https://kind.sigs.k8s.io/>) under the hood

Install kubectl (<https://kubernetes.io/docs/tasks/tools/#kubectl>) and kubectx + kubens (<https://github.com/ahmetb/kubectx>)

then execute commands:
```mkdir -p ~/.kube
KUBECONFIG=~/.airbyte/abctl/abctl.kubeconfig kubectl config view --flatten > ~/.kube/config
kubectx kind-airbyte-abctl
kubens airbyte-abctl```
What results do you get for
`kubectl get persistentvolumeclaim`
and
`kubectl get persistentvolume`?
3

Thank you for the response. I get

NAME                                     STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
airbyte-minio-pv-claim-airbyte-minio-0   Bound    airbyte-minio-pv    500Mi      RWO            standard       <unset>                 2d13h
airbyte-volume-db-airbyte-db-0           Bound    airbyte-volume-db   500Mi      RWO            standard       <unset>                 2d13h
[ec2-user ~]$ kubectl get persistentvolume
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                  STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
airbyte-minio-pv    500Mi      RWO            Retain           Bound    airbyte-abctl/airbyte-minio-pv-claim-airbyte-minio-0   standard       <unset>                          2d13h
airbyte-volume-db   500Mi      RWO            Retain           Bound    airbyte-abctl/airbyte-volume-db-airbyte-db-0           standard       <unset>                          2d13h```
4

I have both of those set-up and see that it is using that external database and storage.

5

I checked abctl code and it looks like this part is always executed during installation https://github.com/airbytehq/abctl/blob/1927aa3de98abae7bcdf5d6b3e8b6c9da3712de4/internal/cmd/local/local/install.go#L154-L159|install.go#L154-L159, https://github.com/airbytehq/abctl/blob/1927aa3de98abae7bcdf5d6b3e8b6c9da3712de4/internal/cmd/local/local/install.go#L169-L174|install.go#L169-L174 :shrug::skin-tone-2:

6

What do you have in custom values.yaml? You might still saving logs in minio. If you don’t use external database, then for sure some new data will be saved on volume as well.

https://docs.airbyte.com/deploying-airbyte/integrations/storage
https://docs.airbyte.com/deploying-airbyte/integrations/database