Error running abctl local install with secrets.yaml on Google Compute Engine VM

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is encountering an error when trying to run ‘abctl local install’ with a ‘secrets.yaml’ file on a Google Compute Engine VM. The error message is not provided, but the user has shared the format of the ‘secrets.yaml’ file containing GCP service account information.


Question

Hello guys, I’m using Google Compute Engine VM to deploy airbyte 1.0 using ‘abctl’ tool. However when I try to run this abctl local install --values ./values.yaml --secret ./secrets.yaml . I get following error (see attached image). This is how my secrets.yaml looks like in format:

kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  gcp.json: {
    "type": "service_account",
    "project_id": "cloud-proj",
    "private_key_id": "2f3b9c8e7d5a1b4f23e697c0d84af6e1",
    "private_key": "-----BEGIN PRIVATE KEY-----<REDACTED>\n-----END PRIVATE KEY-----\n",
    "client_email": "<http://cloud-proj.iam.gserviceaccount.com|cloud-proj.iam.gserviceaccount.com>",
    "client_id": "9876543210987654321",
    "auth_uri": "<https://accounts.google.com/o/oauth2/auth>",
    "token_uri": "<https://oauth2.googleapis.com/token>",
    "auth_provider_x509_cert_url": "<https://www.googleapis.com/oauth2/v1/certs>",
    "client_x509_cert_url": "<https://www.googleapis.com/robot/v1/metadata/x509/cloud-proj.iam.gserviceaccount.com>"
  }```
Can someone please tell me what I'm doing wrong? Thanks!

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1728400723478499) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["google-compute-engine-vm", "abctl", "secrets.yaml", "gcp-service-account", "error"]
</sub>
2

I think you have the wrong format of secret data. You appear to have a JSON object there, but it’s expecting a string

3

Does this help? https://stackoverflow.com/questions/59882986/how-to-embed-json-string-as-the-value-in-a-kubernetes-secret

4

Thank you! I will try that. I was following steps in the documentation: https://docs.airbyte.com/deploying-airbyte/integrations/storage

5

Hm, ya, those docs could be wrong.

6

fix the docs: https://github.com/airbytehq/airbyte/pull/46580

7

Let me know if that | fix worked for you, so I know if the docs change I proposed is correct

8

Hey so the | fix worked perfectly. Thank you again for your help.

9

Hi <@U07FH2Y34A1>, I also ran into this problem today, but deploying to k8s on GCP via helm, with kubectl pulling the yaml secret.
The suggested fix doesn’t seem to work in this case:

kubectl apply -f secrets/airbyte_service_account.yaml -n airbyte
error: error parsing secrets/airbyte_service_account.yaml: error converting YAML to JSON: yaml: line 9: could not find expected ‘:’

Supplied yaml:

kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  gcp.json: |
  {
    "type": "service_account",
    "project_id": "f1toolbox-core",
    "private_key_id": "1b47257ad01f07dc6c53a06da103437b86170d8f",
    "private_key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
    "client_email": "airbyte@f1toolbox-core.iam.gserviceaccount.com",
    "client_id": "103040727631725486249",
    "auth_uri": "<https://accounts.google.com/o/oauth2/auth>",
    "token_uri": "<https://oauth2.googleapis.com/token>",
    "auth_provider_x509_cert_url": "<https://www.googleapis.com/oauth2/v1/certs>",
    "client_x509_cert_url": "<https://www.googleapis.com/robot/v1/metadata/x509/airbyte%40f1toolbox-core.iam.gserviceaccount.com>",
    "universe_domain": "googleapis.com"
  }```
10

Hm, I can’t see anything wrong by looking at it. YAML can be pretty finicky. You could try the base64 encoded version, that might avoid some yaml woes.

11

Found the issue. It turns out the indentation of the json needed to be inline with the brackets, not with one extra indent.