Issue with Airbyte Server and Worker Installation on GKE

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User encountered an error starting the Micronaut server during Airbyte installation on GKE using Helm, specifically related to GCS storage client bean instantiation due to missing credentials at the specified path.


Question

Hi all, I am trying to install airbyte with helm on GKE, all but 2 containers did install, the airbyte-server and the airbyte-worker. The logs I am getting:

2024-12-06 09:21:52,142 [main] ERROR i.m.r.Micronaut(handleStartupException):349 - Error starting Micronaut server: Error instantiating bean of type [io.airbyte.commons.storage.GcsStorageClient]: /secrets/gcs-log-creds/gcp.json
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.airbyte.commons.storage.GcsStorageClient]: /secrets/gcs-log-creds/gcp.json

I am using a GSC bucket and Workload Identity
I saw in a github section from a user (https://github.com/airbytehq/airbyte/issues/31988), that the container might be deprecated, would that be true? Or how can we get it to run?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['helm-installation', 'gke', 'airbyte-server', 'airbyte-worker', 'gcs-storage-client', 'workload-identity']
2

What is the version of Airbyte platform/helm you’re using?

3

hi, it is 1.2.0., however I resolved the issue by specifying my gcp.json service account in the following format:

apiVersion: v1
kind: Secret
metadata:
name: airbyte-config-secrets
type: Opaque
stringData:
gcp.json: |
{
"type": "service_account",
"project_id": "cloud-proj",
"private_key_id": "2f3b9c8e7d5a1b4f23e697c0d84af6e1",
"private_key": "-----BEGIN PRIVATE KEY-----<REDACTED>\n-----END PRIVATE KEY-----\n",
"client_email": "<http://cloud-proj.iam.gserviceaccount.com|cloud-proj.iam.gserviceaccount.com>",
"client_id": "9876543210987654321",
"auth_uri": "<https://accounts.google.com/o/oauth2/auth>",
"token_uri": "<https://oauth2.googleapis.com/token>",
"auth_provider_x509_cert_url": "<https://www.googleapis.com/oauth2/v1/certs>",
"client_x509_cert_url": "<https://www.googleapis.com/robot/v1/metadata/x509/cloud-proj.iam.gserviceaccount.com>"
}

As the issue was related to the secret of the gcs-log-creds