Issue with configuring GCS bucket in latest Helm chart for Airbyte

Summary

The user is facing an error while configuring a GCS bucket in the latest Helm chart for Airbyte. The error message indicates a problem with the deployment template and a nil pointer evaluation. The values.yaml file contains configuration settings for the Airbyte deployment, including GCS storage configuration.


Question

Hello im trying to migrate to the latest helm chart but facing issue while configuring GCS bucket im getting below error

Unable to create application: application spec for airbyte is invalid: InvalidSpecError: Unable to generate manifests in airbyte: rpc error: code = Unknown desc = helm template . --name-template airbyte --namespace airbyte --kube-version 1.28 --values &lt;path to cached source&gt;/airbyte/values.yaml &lt;api versions removed&gt; --include-crds failed exit status 1: Error: template: airbyte/charts/worker/templates/deployment.yaml:74:44: executing “airbyte/charts/worker/templates/deployment.yaml” at <.Values.global.storage.type>: nil pointer evaluating interface {}.type Use --debug flag to render out invalid YAML

values.yaml

global:
# – Service Account name override
serviceAccountName: &service-account-name “airbyte-admin”
# – Deployment mode, whether or not render the default env vars and volumes in deployment spec
deploymentMode: “oss”
# – Edition; “community” or “pro”
edition: “community”

airbyteYml: “http://airbyte.test.com|airbyte.test.com

enterprise:
# – Secret name where an Airbyte license key is stored
secretName: “airbyte-config-secrets”
# – The key within licenseKeySecretName where the Airbyte license key is stored
licenseKeySecretKey: “license-key”

# – The URL where Airbyte will be reached; This should match your Ingress host
airbyteUrl: “”

# – Auth configuration
auth:
# – Admin user configuration
instanceAdmin:
# – Secret name where the instanceAdmin configuration is stored
secretName: “airbyte-config-secrets”
# – The first name of the initial user
firstName: “”
# – The last name of the initial user
lastName: “”
# – The key within emailSecretName where the initial user’s email is stored
emailSecretKey: “instance-admin-email”
# – The key within passwordSecretName where the initial user’s password is stored
passwordSecretKey: “instance-admin-password”

_# -- SSO Identify Provider configuration; (requires Enterprise)_
_#identityProvider:_
_#  # -- Secret name where the OIDC configuration is stored_
_#  secretName: "airbyte-config-secrets"_
_#  # -- The identity provider type (e.g. oidc)_
_#  type: ""_
_#  # -- OIDC configuration (required if `auth.identityProvider.type` is "oidc")_
_#  oidc:_
_#    # -- OIDC application domain_
_#    domain: ""_
_#    # -- OIDC application name_
_#    appName: ""_
_#    # -- The key within `clientIdSecretName` where the OIDC client id is stored_
_#    clientIdSecretKey: "client-id"_
_#    # -- The key within `clientSecretSecretName` where the OIDC client secret is stored_
_#    clientSecretSecretKey: "client-secret"_

# – Environment variables
env_vars: {}

# – Database configuration
database:
type: “external” # “external”

_# -- Secret name where database credentials are stored_
secretName: "my-dbsecret" _# e.g. "airbyte-config-secrets"_

_# -- The database host_
host: ""

_# -- The database port_
port: "5432"

_# -- The database name_
database: "airbyte-db"

_# -- The database user_
user: "airbyte"
_# -- The key within `secretName` where the user is stored_
_#userSecretKey: "" # e.g. "database-user"_

_# -- The database password_
_#password: ""_
_# -- The key within `secretName` where the password is stored_
passwordSecretKey: "password" _# e.g."database-password"_

state:
storage:
type: GCS # default storage used
# TODO: Add full configuration here

logs:
storage:
type: “GCS”
gcs:
bucket: “gcs-airbyte”
credentials: “/secrets/gcs-log-creds/gcp.json”
credentialsJson: “”

can someone please help me here ?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["issue", "configuring-gcs-bucket", "latest-helm-chart", "airbyte", "values.yaml", "deployment-template", "nil-pointer", "gcs-storage"]

Just a tip, when you’re pasting configs you can use the Code Block option (or the GitHub-style `````) to make one multi-line code block.

As for the issue, they recently consolidated the storage config, which should look something like this:

  storage:
    type: GCS
    storageSecretName: gcp-cred-secrets
    bucket:
      log: &amp;storage-bucket TODO-bucket-name
      state: *storage-bucket
      workloadOutput: *storage-bucket
      activityPayload: *storage-bucket
    gcs:
      projectId: TODO-project-id
      bucket: *storage-bucket
      credentialsJson: &lt;CREDENTIALS_JSON_BLOB&gt;```
Since this change is recent, I've found that the <https://docs.airbyte.com/enterprise-setup/implementation-guide|deployment instructions for the Self-Managed Enterprise> (SME) version are a little more accurate to the current `values.yaml` spec (although I know the Airbyte team is actively working on this)—so I would check those first. Just expand the sections on prerequisites and storage config and choose the GCP option and you'll have a good start.

Some of the values I have listed above (like `globals.storage.bucket.activityPayload` and `globals.storage.gcs.bucket`) were from warnings that I got that they were missing, but these may have been resolved in the latest build (I just haven't tried removing them yet).

(Also note that I'm using YAML anchors in the above example (basically defining a variable of sorts using `&amp;storage-bucket` and then referencing it with an alias using `*storage-bucket` so I don't have to repeat the name so much. You can just as easily hard-code all of these, I just like keeping things DRY where possible.)

<@U035912NS77> thank you,the provided solution worked, however after adding this i was facing another issue on json file after adding following line "credentialsJson" i was able to deploy

storage:
type: “GCS”
storageSecretName: airbyte-config-secrets
bucket: ## GCS bucket names that you’ve created. We recommend storing the following all in one bucket.
log: gcs-airbyte
state: gcs-airbyte
workloadOutput: gcs-airbyte
gcs:
projectId: test-project
credentialsPath: /secrets/gcs-log-creds/gcp.json
credentialsJson: “encoded json file”

Thank you for the support