Container orchestrator pod access to GCP credentials

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Container orchestrator pod is unable to access GCP credentials for authentication.


Question

:thread: Container orchestrator pod does not have access to GCP credentials


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["container-orchestrator", "gcp-credentials", "authentication"]
2

I have configured my worker with this config:

    - name: STATE_STORAGE_GCS_BUCKET_NAME
      valueFrom:
        configMapKeyRef:
          key: GCS_LOG_BUCKET
          name: airbyte-airbyte-env
    - name: STATE_STORAGE_GCS_APPLICATION_CREDENTIALS
      valueFrom:
        configMapKeyRef:
          key: GOOGLE_APPLICATION_CREDENTIALS
          name: airbyte-airbyte-env```
When the worker spawns the orchestrator pod this log appers in the orchestrator:

```│ main Caused by: java.nio.file.NoSuchFileException: /secrets/gcs-log-creds/gcp.json```
Using GKE with Helm deployment.

Ask AI thread: <https://airbytehq.slack.com/archives/C01AHCD885S/p1707482540874689>
3

<@U0697SLH4TS> can you help here?

4

Sorry about the issue Damjan, at the moment we’re trying to improve the documentation to use external logs and database in Helm

5

Appreciate your efforts Marcos!

6

Sorry for bothering, any news on this ?

7

Hello Damjan, team still working on. You probably need to create an externalVolumeMount and Volume in the Helm Chart to map your external variables

8

Hello All, I’m stuck with the same issue for a while , appreciate your help if there is any update regarding this issue.

9

<@U05U3906VTP> hello , I have found a solution for this problem you just need to define the following extraEnv variables in the worker

  • name: CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH
    value: /secrets/gcs-log-creds/
  • name: CONTAINER_ORCHESTRATOR_SECRET_NAME
    value: airbyte-gcs-log-creds
10

Well aren’t you the best
Thanks a lot mate, it works!

11

Docs are updated https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#external-logs-with-gcs