Error running abctl local install with secrets.yaml on Google Compute Engine VM

Summary

The user is encountering an error when trying to run ‘abctl local install’ with a ‘secrets.yaml’ file on a Google Compute Engine VM. The error message is not provided, but the user has shared the format of the ‘secrets.yaml’ file containing GCP service account information.


Question

Hello guys, I’m using Google Compute Engine VM to deploy airbyte 1.0 using ‘abctl’ tool. However when I try to run this abctl local install --values ./values.yaml --secret ./secrets.yaml . I get following error (see attached image). This is how my secrets.yaml looks like in format:

kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  gcp.json: {
    "type": "service_account",
    "project_id": "cloud-proj",
    "private_key_id": "2f3b9c8e7d5a1b4f23e697c0d84af6e1",
    "private_key": "-----BEGIN PRIVATE KEY-----<REDACTED>\n-----END PRIVATE KEY-----\n",
    "client_email": "<http://cloud-proj.iam.gserviceaccount.com|cloud-proj.iam.gserviceaccount.com>",
    "client_id": "9876543210987654321",
    "auth_uri": "<https://accounts.google.com/o/oauth2/auth>",
    "token_uri": "<https://oauth2.googleapis.com/token>",
    "auth_provider_x509_cert_url": "<https://www.googleapis.com/oauth2/v1/certs>",
    "client_x509_cert_url": "<https://www.googleapis.com/robot/v1/metadata/x509/cloud-proj.iam.gserviceaccount.com>"
  }```
Can someone please tell me what I'm doing wrong? Thanks!

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1728400723478499) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["google-compute-engine-vm", "abctl", "secrets.yaml", "gcp-service-account", "error"]
</sub>

I think you have the wrong format of secret data. You appear to have a JSON object there, but it’s expecting a string

Does this help? https://stackoverflow.com/questions/59882986/how-to-embed-json-string-as-the-value-in-a-kubernetes-secret

Thank you! I will try that. I was following steps in the documentation: https://docs.airbyte.com/deploying-airbyte/integrations/storage

Hm, ya, those docs could be wrong.

fix the docs: https://github.com/airbytehq/airbyte/pull/46580

Let me know if that | fix worked for you, so I know if the docs change I proposed is correct

Hey so the | fix worked perfectly. Thank you again for your help.

Hi <@U07FH2Y34A1>, I also ran into this problem today, but deploying to k8s on GCP via helm, with kubectl pulling the yaml secret.
The suggested fix doesn’t seem to work in this case:

kubectl apply -f secrets/airbyte_service_account.yaml -n airbyte
error: error parsing secrets/airbyte_service_account.yaml: error converting YAML to JSON: yaml: line 9: could not find expected ‘:’

Supplied yaml:

kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  gcp.json: |
  {
    "type": "service_account",
    "project_id": "f1toolbox-core",
    "private_key_id": "1b47257ad01f07dc6c53a06da103437b86170d8f",
    "private_key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
    "client_email": "airbyte@f1toolbox-core.iam.gserviceaccount.com",
    "client_id": "103040727631725486249",
    "auth_uri": "<https://accounts.google.com/o/oauth2/auth>",
    "token_uri": "<https://oauth2.googleapis.com/token>",
    "auth_provider_x509_cert_url": "<https://www.googleapis.com/oauth2/v1/certs>",
    "client_x509_cert_url": "<https://www.googleapis.com/robot/v1/metadata/x509/airbyte%40f1toolbox-core.iam.gserviceaccount.com>",
    "universe_domain": "googleapis.com"
  }```

Hm, I can’t see anything wrong by looking at it. YAML can be pretty finicky. You could try the base64 encoded version, that might avoid some yaml woes.

Found the issue. It turns out the indentation of the json needed to be inline with the brackets, not with one extra indent.