Struggling with S3 configuration for state & logging using instanceProfile authentication

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is facing issues with S3 configuration for state & logging using instanceProfile authentication in Airbyte platform. The error message indicates a problem with loading AWS credentials from the provider chain.


Question

Hi all,
I’m struggling with a configuration of S3 for state & logging. I would like to use the instanceProfile
• installation on EC2 with abctl
• EC2 has the role attached with access to S3
• airbyte is installed with abctl local install --values values.yaml
• values.yml:
• using the aws s3 ls commands succeeds, so role is attached correctly to EC2 instance

  auth:
    enabled: false
  env_vars:
    JOB_MAIN_CONTAINER_CPU_REQUEST: "250m"
    JOB_MAIN_CONTAINER_CPU_LIMIT: "500m"
    HTTP_IDLE_TIMEOUT: 1800s
  storage:
    type: "S3"
    bucket:
      log: my-airbyte-bucket-test
      state: my-airbyte-bucket-test
      workloadOutput: my-airbyte-bucket-test
    s3:
      region: us-east-1
      authenticationType: "instanceProfile"```
The result is:
```[airbyte-abctl-workload-launcher-bcc88976b-zdds9] java.lang.RuntimeException: Cannot start publishing: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@5843fca0: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@568419d8: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at com.van.logging.aws.S3PublishHelper.start(S3PublishHelper.java:80) ~[appender-core-5.3.2.jar:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at com.van.logging.BufferPublisher.startPublish(BufferPublisher.java:36) ~[appender-core-5.3.2.jar:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at com.van.logging.LoggingEventCache.publishEventsFromFile(LoggingEventCache.java:187) ~[appender-core-5.3.2.jar:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at com.van.logging.LoggingEventCache.lambda$publishCache$0(LoggingEventCache.java:243) ~[appender-core-5.3.2.jar:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[airbyte-abctl-workload-launcher-bcc88976b-zdds9] Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@5843fca0: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@568419d8: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)]```

Any ideas what can be missing?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1723200907998209) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["s3-configuration", "state-logging", "instanceprofile-authentication", "aws-credentials", "error-message"]
</sub>
2

Yeah, I’ve been struggling with this exact issue for a while now. The only thing that worked is to disable the Orchestrator, workload launcher, etc.
It seems like those pods/workloads still expect AWS keys defined and won’t use an AWS Role instead.
Without orchestrator and workload pods worker will do all the job. And you will get logs in S3 and Airbyte UI with no issues.
If you figure out the way to use Role with these pods let me know :slightly_smiling_face:

3

Though I’m using Helm. Not sure if there are any differences here

4

For now I finished with creating dedicated user for Airbyte just with access to specific s3 and used the credentials as authentication type :cry: