Enabling external log storage to S3 using instanceProfile in Airbyte v1.0.0 on AWS EC2

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is trying to enable external log storage to S3 using instanceProfile in Airbyte v1.0.0 on AWS EC2, but encountering an error related to AWS credentials. They are seeking confirmation if it’s possible to use instanceProfile for this purpose.


Question

Hello! I am installing Airbye v1.0.0 using abctl on AWS Linux 2023 on EC2. Does anyone know if it’s possible to enable external log storage to S3 using instanceProfile only (not user)? I tried the documentation https://docs.airbyte.com/deploying-airbyte/integrations/storage#secrets|here, but I am getting the following error on the server pod. It seems Airbyte is still seeking an access key id and secret. Can anyone help confirm if it’s possible to use instanceProfile?

Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@1de72018: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@1ac39d2f: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)]


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["enabling-external-log-storage", "s3", "instanceprofile", "airbyte-v1.0.0", "aws-ec2", "aws-credentials"]
2

You can check this thread, maybe you will have ideas what to try
https://airbytehq.slack.com/archives/C021JANJ6TY/p1726240154984179?thread_ts=1726165760.065099&cid=C021JANJ6TY

3

Thank you for the suggestion. I tried the values.yaml file suggested in the thread, but still getting the same error. I find it strange that the last part of the credentials chain has the following error: com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@1ac39d2f: Unauthorized . Anyway, I’m starting to get the impression that the fix for this is beyond my control, and maybe i’ll report it as an issue on the github discussions. :laughing:

4

Hi Mark! Confirmed on my end that setting http_put_response_hop_limit to 3 solves this issue. Thanks so much for your help. :grinning:

5

I encountered the same issue, and I also was able to resolve it by changing the value of http_put_response_hop_limit.
Thank you!

6

I’ve just been running in to the same issue and have figured out what’s going on. If you require AWS IMDSv2 on the EC2 instance that Airbyte is running on, and you’ve used abctl to launch Airbyte, you need to set the http_put_response_hop_limit metadata option for the EC2 instance to 3. The <Access instance metadata for an EC2 instance - Amazon Elastic Compute Cloud documentation> suggests a value of 2 for container environments, but I think running Kubernetes in Docker through kind adds another network that the packets must traverse and so 3 seems to be required. Having is set to 1 or 2 will fail because the containers can’t access the IMDS token endpoint at all.

7

<@U07BQ124R0D> wow, it makes totally sense, but it’s not easy to figure it out
I learned something new today :slightly_smiling_face:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-new-instances.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-existing-instances.html