What is the right format to specify the JSON credentials for the Google secret manager?

  • Is this your first time deploying Airbyte: No
  • OS Version / Instance: COS-5.4.171 Kubernetes: 1.18.20 Docker: 19.03.15 Family: cos-85-lts, supports Shielded VM features, supports Confidential VM features on N2D, GCP e2-standard-2
  • Memory / Disk: 8Gb / 20GB SSD
  • Deployment: Docker
  • Airbyte Version: 0.35.62-alpha
  • Source name/version: Postgres/I don’t know.
  • Destination name/version: N/A
  • Step: Setting new connection
  • Description: I’m trying to use the GCP secret manager according to the airbyte doc to save connection credentials but keeps getting an error about “credentialJson is null” as shown in the image below.

    I suspect this is related to the GCP secret manager configuration in the .env file. I was pasting the whole json file to the SECRET_STORE_GCP_CREDENTIALS variable as
SECRET_STORE_GCP_CREDENTIALS={"type": "service_account","project_id":"my_project","private_key_id": "acd90445....","private_key":"...",...}

I also tried escaping the quotations and provide a JSON string instead but got the same error.

SECRET_STORE_GCP_CREDENTIALS="{\"type\": \"service_account\",\"project_id\":\"my_project\",\"private_key_id\": \"acd90445....\",\"private_key\":\"...\",...}

Lastly I tried specifying the location to the credential file but still got no luck.

Could you share some insights on how to configure the credentials?

Hi @Jing, do you mind sharing your server logs? We might get more details about the error. You can download these from the UI in the settings page.

According to your error, it looks like the SECRET_STORE_GCP_CREDENTIALS is not set on your containers. It’s probably because this variable, from the .env file, is not mapped to the services declared in the docker-compose.yaml file. Do you mind adding this variable to the server, scheduler and worker services in docker-compose.yaml:

    environment:
      - SECRET_PERSISTENCE=${SECRET_PERSISTENCE}
      - SECRET_STORE_GCP_CREDENTIALS=${SECRET_STORE_GCP_CREDENTIALS}
      ...

I think you don’t need to escape the json quotes in the env var, I doubled check on our Cloud deployment, this value is stored as your first example, the original JSON.

2 Likes

Are you running on Kubernetes or with docker-compose?

Hi @alafanechere ! Just tried your solution and it worked :grinning: . One minor correction - SECRET_PERSISTENCE is already in docker-compose.yaml, so these had to be added instead

    environment:
      - SECRET_STORE_GCP_PROJECT_ID=${SECRET_STORE_GCP_PROJECT_ID}
      - SECRET_STORE_GCP_CREDENTIALS=${SECRET_STORE_GCP_CREDENTIALS}
1 Like

Hi @alafanechere @coder46 it is now working for my docker deployment! Thank you for the help!

1 Like

I’m using docker-compose :slight_smile:

I opened a PR to make this fix part of our repo: docker-compose: map secret related env var to containers by alafanechere · Pull Request #11852 · airbytehq/airbyte · GitHub