Deploy Kubernetes: Unable to launch airbyte using helm charts

Platform, Deploy & Infra Issues
, ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: MacOS.
  • Memory / Disk: 16Gb / 0.5 Tb
  • Deployment: Kubernetes
  • Airbyte Version: v0.39.28-alpha
  • Source name/version: -
  • Destination name/version: -
  • Step: The issue is happening while setting up airbyte using helm chart
  • Description:

Reason for failure : pod airbyte-bootloader failed

Followed following steps to setup airbyte :

  1. Created namespace airbyte : kubectl create ns airbyte

  2. Build helm dependencies : helm dependency build --debug -n airbyte
    LOGS :

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 3 charts
Downloading common from repo https://charts.bitnami.com/bitnami
Downloading postgresql from repo https://charts.bitnami.com/bitnami
Downloading minio from repo https://charts.bitnami.com/bitnami
Deleting outdated charts
  1. Install airbyte : helm install airbyte -n airbyte . --debug
    LOGS :
install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: /Users/somitragupta/factly/airbyte/charts/airbyte

client.go:310: [debug] Starting delete for "airbyte-admin" ServiceAccount
client.go:339: [debug] serviceaccounts "airbyte-admin" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "airbyte-env" ConfigMap
client.go:339: [debug] configmaps "airbyte-env" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "airbyte-postgresql" Secret
client.go:339: [debug] secrets "airbyte-postgresql" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "airbyte-postgresql" Service
client.go:339: [debug] services "airbyte-postgresql" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "airbyte-postgresql" StatefulSet
client.go:339: [debug] statefulsets.apps "airbyte-postgresql" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "airbyte-postgresql-headless" Service
client.go:339: [debug] services "airbyte-postgresql-headless" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "airbyte-secrets" Secret
client.go:339: [debug] secrets "airbyte-secrets" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "airbyte-bootloader" Pod
client.go:339: [debug] pods "airbyte-bootloader" not found
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Pod airbyte-bootloader with timeout of 5m0s
client.go:568: [debug] Add/Modify event for airbyte-bootloader: ADDED
client.go:627: [debug] Pod airbyte-bootloader pending
client.go:568: [debug] Add/Modify event for airbyte-bootloader: MODIFIED
client.go:627: [debug] Pod airbyte-bootloader pending
client.go:568: [debug] Add/Modify event for airbyte-bootloader: MODIFIED
client.go:627: [debug] Pod airbyte-bootloader pending
client.go:568: [debug] Add/Modify event for airbyte-bootloader: MODIFIED
client.go:629: [debug] Pod airbyte-bootloader running
client.go:568: [debug] Add/Modify event for airbyte-bootloader: MODIFIED
Error: INSTALLATION FAILED: failed pre-install: pod airbyte-bootloader failed
helm.go:84: [debug] failed pre-install: pod airbyte-bootloader failed
INSTALLATION FAILED
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.4.0/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.4.0/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.4.0/command.go:902
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1571
  1. This leads to pod status :
NAME                       READY   STATUS    RESTARTS   AGE
airbyte-bootloader      0/1     Error         0               18m
airbyte-postgresql-0   0/1     Pending   0               18m

Could someone guide me where exactly I am missing context ?
Thanks in advance.

2

The bootloader will fail without connectivity to a database - why is the postgresql pod stuck pending? You may want to check events for that pod:
kubectl describe pod -n airbyte airbyte-postgresql-0

3 
➜  airbyte git:(master) kubectl describe pvc data-airbyte-postgresql-0 -n airbyte
Name:          data-airbyte-postgresql-0
Namespace:     airbyte
StorageClass:  standard
Status:        Pending
Volume:
Labels:        app.kubernetes.io/instance=airbyte
               app.kubernetes.io/name=postgresql
               role=primary
Annotations:   volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       airbyte-postgresql-0
Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  ExternalProvisioning  10h (x162 over 10h)   persistentvolume-controller  waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
  Normal  ExternalProvisioning  2m20s (x41 over 12m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
4

Just FYI , I could not resolve the error , Thus I tried to reinstall minikube with virtual-box , and this worked for me.

minikube start --vm-driver=virtualbox --memory=8192 --cpus=4

Then I added the chart and it worked for me.

NAME                                   READY   STATUS      RESTARTS   AGE
airbyte-bootloader                     0/1     Completed   0          11m
airbyte-minio-ff5694467-svbtk          1/1     Running     0          11m
airbyte-pod-sweeper-566b4b7bbd-24shw   1/1     Running     0          11m
airbyte-postgresql-0                   1/1     Running     0          11m
airbyte-server-79796d8d5c-cwt76        1/1     Running     0          11m
airbyte-temporal-57c4bfcd79-4pwjk      1/1     Running     0          11m
airbyte-webapp-86cdbdb646-wk5xs        1/1     Running     0          11m
airbyte-worker-7bc4c67bb8-nsjpg        1/1     Running     0          11m

Although I still don’t know how to fix the previous condition - **waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator** .

5

The postgres bitnami chart that airbyte uses for it’s internal database does not automatically create a persistent volume - so it fails to boot, which means that the airbyte-bootloader also fails to boot - this is a bad design for a helm chart as IMHO the default values for the helm chart should create a minimal working installation

6

Thanks for the post and the great discussion. I created a Github issue here: https://github.com/airbytehq/airbyte/issues/14463

Feel free to follow the issue and contribute to the discussion there.

7

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.