What is the right format to specify the JSON credentials for the Google secret manager?

Platform, Deploy & Infra Issues
, ,
1
  • Is this your first time deploying Airbyte: No
  • OS Version / Instance: COS-5.4.171 Kubernetes: 1.18.20 Docker: 19.03.15 Family: cos-85-lts, supports Shielded VM features, supports Confidential VM features on N2D, GCP e2-standard-2
  • Memory / Disk: 8Gb / 20GB SSD
  • Deployment: Docker
  • Airbyte Version: 0.35.62-alpha
  • Source name/version: Postgres/I don’t know.
  • Destination name/version: N/A
  • Step: Setting new connection
  • Description: I’m trying to use the GCP secret manager according to the airbyte doc to save connection credentials but keeps getting an error about “credentialJson is null” as shown in the image below.
    image (22)
    image (22)1682×452 42.4 KB

    I suspect this is related to the GCP secret manager configuration in the .env file. I was pasting the whole json file to the SECRET_STORE_GCP_CREDENTIALS variable as
SECRET_STORE_GCP_CREDENTIALS={"type": "service_account","project_id":"my_project","private_key_id": "acd90445....","private_key":"...",...}

I also tried escaping the quotations and provide a JSON string instead but got the same error.

SECRET_STORE_GCP_CREDENTIALS="{\"type\": \"service_account\",\"project_id\":\"my_project\",\"private_key_id\": \"acd90445....\",\"private_key\":\"...\",...}

Lastly I tried specifying the location to the credential file but still got no luck.

Could you share some insights on how to configure the credentials?

2

Hi @Jing, do you mind sharing your server logs? We might get more details about the error. You can download these from the UI in the settings page.

3

According to your error, it looks like the SECRET_STORE_GCP_CREDENTIALS is not set on your containers. It’s probably because this variable, from the .env file, is not mapped to the services declared in the docker-compose.yaml file. Do you mind adding this variable to the server, scheduler and worker services in docker-compose.yaml:

    environment:
      - SECRET_PERSISTENCE=${SECRET_PERSISTENCE}
      - SECRET_STORE_GCP_CREDENTIALS=${SECRET_STORE_GCP_CREDENTIALS}
      ...

I think you don’t need to escape the json quotes in the env var, I doubled check on our Cloud deployment, this value is stored as your first example, the original JSON.

2 Likes
4

Are you running on Kubernetes or with docker-compose?

5

Hi @alafanechere ! Just tried your solution and it worked :grinning: . One minor correction - SECRET_PERSISTENCE is already in docker-compose.yaml, so these had to be added instead

    environment:
      - SECRET_STORE_GCP_PROJECT_ID=${SECRET_STORE_GCP_PROJECT_ID}
      - SECRET_STORE_GCP_CREDENTIALS=${SECRET_STORE_GCP_CREDENTIALS}
1 Like
6

Hi @alafanechere @coder46 it is now working for my docker deployment! Thank you for the help!

1 Like
7

I’m using docker-compose :slight_smile:

8

I opened a PR to make this fix part of our repo: docker-compose: map secret related env var to containers by alafanechere · Pull Request #11852 · airbytehq/airbyte · GitHub

9

Hi I was able to set up the secret manager env vars, but how do I access the secrets in the UI? and how do I set up secrets in GSM?

10

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.