Error accessing S3 bucket in Airbyte server logs

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The error in the Airbyte server logs indicates a region mismatch while trying to access an S3 bucket. The logs show a PermanentRedirect error due to the bucket being in the eu-west-1 region while the request is made from us-east-2 region.


Question

I am still getting some errors in airbyte server logs, as it is not able to accesss right S3 bucket. If someone can help me with it?

java.lang.RuntimeException: Cannot end publishing: Cannot publish to S3: The bucket is in this region: eu-west-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: RBBZHF447NKK8FSH; S3 Extended Request ID: NhvCwGZRVVylHOAgcalBEfoLt+dd/Kl+T/d/nSuI=; Proxy: null)
	at com.van.logging.AbstractFilePublishHelper.end(AbstractFilePublishHelper.java:66) ~[appender-core-5.3.2.jar:?]
	at com.van.logging.BufferPublisher.endPublish(BufferPublisher.java:67) ~[appender-core-5.3.2.jar:?]
	at com.van.logging.LoggingEventCache.publishEventsFromFile(LoggingEventCache.java:198) ~[appender-core-5.3.2.jar:?]
	at com.van.logging.LoggingEventCache.lambda$publishCache$0(LoggingEventCache.java:243) ~[appender-core-5.3.2.jar:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.RuntimeException: Cannot publish to S3: The bucket is in this region: eu-west-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: RBBZHF447NKK8FSH; S3 Extended Request ID: +bMnkdRkZDZlIdddw5K8sSE/K/nSuI=; Proxy: null)
	at com.van.logging.aws.S3PublishHelper.publishFile(S3PublishHelper.java:131) ~[appender-core-5.3.2.jar:?]
	at com.van.logging.AbstractFilePublishHelper.end(AbstractFilePublishHelper.java:61) ~[appender-core-5.3.2.jar:?]
	... 8 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: eu-west-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: RBBZHF447NKK8FSH; S3 Extended Request ID: NhvCwGZRVVylHOAdddgcalBEfoLt+ddd/Kl+T//nSuI=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1880) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[aws-java-sdk-core-1.12.770.jar:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5575) ~[aws-java-sdk-s3-1.12.770.jar:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5522) ~[aws-java-sdk-s3-1.12.770.jar:?]```


here is my configs in values file:
```global:
  storage:
    daps-dev-airbyte-logs: ## S3 bucket name
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
    s3:
      secretAccessKeySecretKey: "AWS_SECRET_ACCESS_KEY"
      accessKeyIdSecretKey: "AWS_ACCESS_KEY_ID" 
      secretAccessKey: "<>"
      accessKeyId: "<>"
      region: "us-east-2" ## e.g. us-east-1
      authenticationType: credentials ## Use "credentials" or "instanceProfile"```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1730375960884469) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["error", "s3-bucket", "airbyte-server-logs", "region-mismatch", "config-values"]
</sub>