Worker process of Airbyte > v0.40.9 fails to start on custom S3 config

Platform, Deploy & Infra Issues
19

Yes; we encountered a similar problem with GCS.

These configuration changes solved the issue for us (note that we are using the k8s manifests directly, not the helm chart):

  1. In .env, the env var GCS_LOG_BUCKET needs to be set to the log bucket and the additional variable called STATE_STORAGE_GCS_BUCKET_NAME needs to be set to the state storage bucket. As far as I can tell, STATE_STORAGE_GCS_BUCKET_NAME isn’t documented, but you can see that it is part of the GCS configuration block for the workers: airbyte/application.yml at 7676af5f5fb53542ebaff18a415f9c89db417055 · airbytehq/airbyte · GitHub . The Minio/S3 variables for us are mostly nulled out, so the config variables for logs and storage largely look like so:
# S3/Minio Log Configuration
S3_LOG_BUCKET=
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=
S3_PATH_STYLE_ACCESS=

# GCS Log Configuration
GCS_LOG_BUCKET=<log bucket>
STATE_STORAGE_GCS_BUCKET_NAME=<state bucket>

# State Storage Configuration
STATE_STORAGE_MINIO_BUCKET_NAME=
STATE_STORAGE_MINIO_ENDPOINT=

# Cloud Storage Configuration
WORKER_LOGS_STORAGE_TYPE=gcs
WORKER_STATE_STORAGE_TYPE=gcs
  1. Secondly, the manifests for the workers need to be modified to actually pass the GCS state bucket variables as they currently do not. In the airbyte-worker deployment (airbyte/worker.yaml at master · airbytehq/airbyte · GitHub), we added the following vars (note that GOOGLE_APPLICATION_CREDENTIALS are reused here, but it is probably better to have a separate SA credentials for writing state):
- name: STATE_STORAGE_GCS_BUCKET_NAME
  valueFrom:
    configMapKeyRef:
      name: airbyte-env
      key: STATE_STORAGE_GCS_BUCKET_NAME
- name: STATE_STORAGE_GCS_APPLICATION_CREDENTIALS
  valueFrom:
    secretKeyRef:
      name: airbyte-secrets
      key: GOOGLE_APPLICATION_CREDENTIALS

Hope this helps.