Worker process of Airbyte > v0.40.9 fails to start on custom S3 config

Platform, Deploy & Infra Issues
8

Hey all!

I was having the same issue. I’m using helm and am not super familiar with kustomize, but hopefully this helps. I had to set a couple more values in my values.yaml file to get it to work.

global:
  # ...
  logs:
    accessKey:
      password: <access_key_id>
      # Downstream charts don't use the secret created by the password above, so we need to pass in the secret info ourselves
      existingSecret: <helm_release_name>-airbyte-secrets
      existingSecretKey: AWS_ACCESS_KEY_ID
    secretKey:
      password: <secret_access_key>
      # Downstream charts don't use the secret created by the password above, so we need to pass in the secret info ourselves
      existingSecret: <helm_release_name>-airbyte-secrets
      existingSecretKey: AWS_SECRET_ACCESS_KEY

Dug in to the code and found out that basically the airbyte-worker and airbyte-server deployment.yaml files only set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables if the existingSecret and existingSecretKey are set, or if minio or externalMinio is enabled. There’s nothing there if I’m just passing in the password myself.

For your situation, I assume the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY aren’t being set properly on the worker/server for some reason. Hope that helps!

1 Like