Setting up connector secrets in values.yaml with GCP Secret Manager

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Struggling to map connector secrets stored in GCP Secret Manager with Airbyte keys in values.yaml file


Question

Hello :wave: ,

I am struggling on how to set up connector secrets (i.e. tokens) in my values.yaml . I plan to store these secrets in GCP Secret Manager. I have already referred gcp.json in values.yaml as described in the doc but I don’t get how to map connector secrets (stored in GCP secret managers) with the airbyte keys specific to connector tokens.

Can anyone clarify this please? :pray:


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["connector-secrets", "values.yaml", "gcp-secret-manager", "airbyte-keys", "connector-tokens"]
2

I don’t think you can directly link GCP Secret Manager secrets into Airbyte deployment. The secrets names and keys you refer to in the values.yaml are Kubernetes secrets. The way I’ve done it is to store the secrets in GCP secret manager and have Terraform pull them and create Kubernetes secrets from them.

The code below is a snippet of the relevant file. It pulls down a secret named “airbyte-admin-password” from GCP Secret Manager and uses that data to populate “airbyte-auth-secrets” kubernetes secret.

  secret = "airbyte-admin-password"
}


resource "kubernetes_secret" "airbyte_auth_secret" {
  metadata {
    name      = "airbyte-auth-secrets"
    namespace = kubernetes_namespace.clickbuddy.metadata[0].name
  }

  data = {
    "instance-admin-email"    = ""
    "instance-admin-password" = data.google_secret_manager_secret_version.airbyte_instance_admin_password.secret_data
  }

  type       = "Opaque"
  depends_on = [google_service_account_key.airbyte_service_account_key]
}```
3

You could use secret management described in docs
https://docs.airbyte.com/deploying-airbyte/integrations/secrets

4

<@U05JENRCF7C> I may miss a few examples then :sweat_smile:.

I already created a Kubernetes secret for my service account. I granted the service account required role to interact with GCP Secret Manager.

I also modified values.yaml file as per the documentation:

secretsManager:
    type: googleSecretManager
    secretManagerSecretName: airbyte-config-secrets
    googleSecretManager:
      projectId: <project-id>
      credentialsSecretKey: gcp.json```
But, the next steps are not that clear to me...

Can you maybe clarify? :pray:
5

<@U07SPT1BWAG> is the setup not working? Mine is more or less identical. Airbyte uses the service account credentials to create and manage connector secrets in GCP secret manager.

6

Your setup makes sense <@U07LLEW2M7E>. Thanks for sharing it :pray:
From Przemyslaw’s message, I was wondering if there was any other solution. But, I will follow your way if this how to do it.

7

That’s all I had to do. Create a service account, give it the necessary permissions, store the credentials in a k8 secret and pass the secret name and key in values.yaml. After that it just worked.

8

Thanks a lot! Makes perfect sense :wink: