Issue with Airbyte orchestrator pods not utilizing Kubernetes service account

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is facing an issue where Airbyte orchestrator pods are not correctly utilizing the Kubernetes service account test-sa to access AWS resources like S3 buckets. Other components of the Airbyte setup are functioning as expected with the same service account.


Question

Hi Team,
I am experiencing an issue with the Airbyte orchestrator pods where they are not correctly utilizing the Kubernetes service account test-sa. The same service account works fine for other pods like server and worker. Here are the details:

Version: 0.63.6

Problem Description: The orchestrator pods are unable to access AWS resources (e.g., S3 buckets) using the service account, while other components of the Airbyte setup are functioning as expected with the same service account.

values.yaml

  serviceAccountName: "test-sa"
  env_vars:
    AWS_DEFAULT_REGION: us-east-2
    AWS_REGION: us-east-2
    S3_REGION: us-east-2

  storage:
    type: s3
    bucket:
      log: test-s3
      state: test-s3
      workloadOutput: test-s3
    s3:
      enabled: true
      bucketRegion: us-east-2
      region: us-east-2

  jobs: 
    resources:
      limits:
        cpu: 200m
        memory: 800Mi


# Add these environment variables to all components
extraEnv: &common_env_vars
  - name: AWS_REGION
    value: us-east-2
  - name: STATE_STORAGE_S3_BUCKET_NAME
    value: test-s3
  - name: STATE_STORAGE_S3_REGION
    value: us-east-2
  - name: AWS_DEFAULT_REGION
    value: us-east-2
  - name: STATE_STORAGE_S3_PATH_PREFIX
    value: logs/airbyte-logs/
  - name: S3_REGION
    value: us-east-2```
Error:
```WARN c.a.u.EC2MetadataUtils(getItems):414 - Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)
com.amazonaws.AmazonServiceException: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)

ERROR c.v.l.BufferPublisher(startPublish):38 - Cannot start publish with com.van.logging.aws.S3PublishHelper@64f65ad1 due to error
java.lang.RuntimeException: Cannot start publishing: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region., com.amazonaws.auth.profile.ProfileCredentialsProvider@3113ce5f: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@5f8d5788: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)]```
Are there any specific configurations or additional settings required for the orchestrator pods to use the service account properly?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1726663605152749) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["airbyte-orchestrator-pods", "kubernetes-service-account", "aws-resources", "s3-buckets", "configuration", "settings"]
</sub>