Using container orchestrator in kubernetes

Platform, Deploy & Infra Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: GKE Container Optimized Linux
  • Memory / Disk: you can use something like 16Gb / 1 Tb
  • Deployment: Kubernetes on GKE
  • Airbyte Version: 0.39.42-alpha
  • Step: How to use new container orchestrator properly
  • Description:
    Set this to true CONTAINER_ORCHESTRATOR_ENABLED. Since we are on GCP created a bucket for state storage and set the value STATE_STORAGE_GCS_BUCKET_NAME. Since the credentials for GCS access is similar re-used what we have for that, so STATE_STORAGE_GCS_APPLICATION_CREDENTIALS points to same mounted secret location as GOOGLE_APPLICATION_CREDENTIALS.
    But it looks like the secret is not mounted to orchestrator container. Is there some step missing here? Any other way to pass on secrets? We have disabled this for now but would love to get this working if possible. (happy to implement for OSS if there is any open ticket to address this)
2022-08-10 16:34:58 ERROR i.a.c.ContainerOrchestratorApp(main):187 - Orchestrator failed...
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /secrets/gcs-log-creds/gcp.json

cc @lgomezm

2

Hey did you add that to .secrets file?

3

@harshith Yes, we added it to .secrets like
STATE_STORAGE_GCS_APPLICATION_CREDENTIALS=/secrets/gcs-log-creds/gcp.json

4

@harshith Do you have any other idea on what else we should try?

5

Hey didn’t get much support from the team. Kindly create a ticket on github so that team can get back to you there