Logging to GCS with Workload Identity in Airbyte on GKE

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Airbyte user is looking to switch application logging from Minio to GCS on GKE while using Workload Identity to avoid providing GCP service account JSON key. They are concerned about the interaction between setting credentialJson in values.yaml and the existing workload identity setup.


Question

Hi folks,

I’ve deployed Airbyte on GKE, helm chart version 0.55.40.

The pods are set up to use Workload Identity, the reason being that we wanted to avoid the user having to provide a GCP service account JSON key each time they set up a BigQuery connector. This is working well.

I’m looking to switch the application logging from Minio to GCS, following instructions here: https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#external-logs-with-gcs

This requires that I provide encoded credentialJson in the values.yaml, which I’d like to avoid as I’ve already got a workload identity set up. By setting this, GOOGLE_APPLICATION_CREDENTIALS would get set, which I expect would render my set up of workload identity redundant.

Can GCS logging work based on workload identity?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["logging", "gcs", "workload-identity", "airbyte", "gke", "helm-chart"]
2

In what I’ve read of the source, this is all pretty hard-coded to use credentials.

Would be a good feature request to put in on GitHub though. I think AWS has a similar functionality to Workload Identity that may be worth mentioning as well.

3

I ended up providing the credentialsJson which is working OK. Good idea on the feature request though, I’ll make a note to add it.