Summary
Error occurs when trying to access S3 bucket from Airbyte server, worker, or job despite correct AWS region settings and credentials
Question
Trying to install OSS Community Airbyte on EKS, Helm chart version 0.64.205. We are using an annotated Service Account to grant AWS role credentials per AWS documentation for IAM roles for service accounts, and we have validated that non-Airbyte resources can access a given S3 bucket. All of our resources are in the AWS region us-east-1 - S3 bucket, IAM role, etc. All pods have us-east-1 set as the environment variables AWS_REGION and AWS_DEFAULT_REGION and we also have global.storage.s3.bucketRegion set to us-east-1. Whenever we try to access that S3 bucket or any other S3 bucket from Airbyte server, Airbyte Worker, or any source-check or destination-check job that gets spun up we get the same error:
software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'
This error comes up regardless if we specify the AWS region in the source or destination config, if we specify an AWS Access Key ID and Secret Access Key, if we specify an endpoint, etc.
Any assistance would be very welcome.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["airbyte", "eks", "s3-bucket", "aws-region", "iam-role"]
we’ve got global.storage.s3.region configured, not global.storage.s3.bucketRegion and don’t have this issue. maybe try that?
I was able to figure this out (after dealing with it for the better part of a week, putting the resolution here for anyone else)
I turned on debug logging and saw airbyte trying to reach out to this bucket:
2024-04-26 22:29:16 DEBUG c.a.h.AmazonHttpClient$RequestExecutor(executeOneRequest):1288 - Sending Request: PUT <https://airbyte-storage.s3.amazonaws.com> /app-logging/workspace/scheduler/logs/20240426222916_dev-dal-airbytetwo-worker-7dc4c87779-vz4fj_e28a26294d3b4e98834b680fb8ae408a```
I assume that the bucket in question lives in `eu-west-1` so when our role tries to reach out to that bucket it gets the `MalformedHeaderException` . When we got the error there was no indication it was erroring out on the logging step.
So then we have a new question: why is Airbyte trying to reach out to that bucket? I did some poking around in the helm charts and there was a default value for `airbyte-storage` set <https://github.com/airbytehq/airbyte-platform/blob/206738cd3438876ed57946c9541d69073fd3b87e/charts/airbyte/templates/env-configmap.yaml#L91|in the configmap>. (the default setup is to configure a bucket in minio named `airbyte-storage`. If you're just using minio you can reuse a bucket name, but not in S3 where buckets have to be globally unique.) Set the values below to my bucket and we stopped getting the error.
• `.Values.global.storage.bucket.workloadOutput`
• `.Values.global.storage.bucket.activityPayload`
• `.Values.global.storage.bucket.log`
• `.Values.global.storage.bucket.state`