NoSuchBucket exception in S3DocumentStoreClient

Platform, Deploy & Infra Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 5 servers t2.small
  • Deployment: Are you using Docker or Kubernetes deployment? k8s
  • Airbyte Version: What version are you using now? 0.39.36-alpha
  • Source name/version: stripe
  • Destination name/version: snowflake
  • Step: happens during syncs
  • Description:

hi! I get errors syncing data using stripe to snowflake (although I assume its a global cluster issue).
this started to happen after cloning my k8s cluster to a new eks cluster (along with upgrading from 0.35.64-alpha to 0.39.36-alpha
the clone I did was to just to basically to take a postgres dump → redeploy the cluster on a new eks cluster → postgres restore
all configurations kept the same.
the error I get is:
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.8.1.jar:?]
at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:448) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.8.1.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:138) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:158) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:284) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
… 14 more
Caused by: software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 1702D3276AA70C1C, Extended Request ID: null)
at io.airbyte.workers.storage.S3DocumentStoreClient.read(S3DocumentStoreClient.java:75) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocument(AsyncOrchestratorPodProcess.java:216) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.checkStatus(AsyncOrchestratorPodProcess.java:220) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocStoreStatus(AsyncOrchestratorPodProcess.java:229) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:137) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:315) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.36-alpha.jar:?]

I am not sure which bucket it tries to find. but I use S3 logging and the bucket exists with all the secrets configured

Would appreciate any help
thanks!
Lior

2

Hey could you check if there are any changes in the .env files in the latest version

3

from what I’ve seen someone added this:
USE_STREAM_CAPABLE_STATE=false

but I’ve already tried that against
AIRBYTE_VERSION=0.39.37-alpha and AIRBYTE_VERSION=0.39.36-alpha
and the issue still persists

4

looks like its not a version issue but more of an environment issue, because I’ve downgraded back to 0.35.64 (disabled database migrations) and this still happens. can you please specify which bucket is it looking for? what configuration is missing?

5

ok found the solution:
needed to run:

kubectl port-forward svc/airbyte-minio-svc 9303:9000

AWS_ACCESS_KEY_ID=minio AWS_SECRET_ACCESS_KEY=minio123  aws --endpoint-url http://localhost:9303 s3 mb airbyte-dev-logs
6

@liorchen Where and how do you ran these commands?

7

on your terminal (assuming aws cli installed)

8

Thanks @liorchen, it solved my issue!
Airbyte just can’t create MinIO bucket for itself, it’s strange.