Kubernetes - Bigquery sync failed

  • Is this your first time deploying Airbyte?: / Yes
  • Deployment: Are you using Docker or Kubernetes deployment? Kubernetes
  • Airbyte Version: What version are you using now?
  • Source name/version: 0.39.1-alpha
  • Destination name/version: Big Query
  • Step: BigQuery sync failed
  • Description:

I’m new to airbyte and having trouble setting up bigquery. i’m using S3 to keep logs and already able to write to it , problem appears while doing bigquery sync.
please advise

Log4j2Appender says: Completing future exceptionally...
2022-05-25 09:34:38 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
Log4j2Appender says: Stopping cancellation check scheduling...
2022-05-25 09:34:38 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):236 - Stopping temporal heartbeating...
Log4j2Appender says: Stopping temporal heartbeating...
2022-05-25 09:34:38 WARN i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=9b3a92ff-05ab-3f9a-a118-fc45f5360cef, activityType=Replicate, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:234) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.replicate(ReplicationActivityImpl.java:111) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.8.1.jar:?]
    at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:448) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.8.1.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:138) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:154) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:229) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 14 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:154) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:229) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 14 more
Caused by: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:265) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 1 more
Caused by: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:177) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:260) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 1 more
Caused by: software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 16F24F5DEDF251C3, Extended Request ID: null)
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:135) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:161) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:84) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:62) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:62) ~[aws-core-2.16.84.jar:?]
    at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:4371) ~[s3-2.16.84.jar:?]
    at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:8123) ~[s3-2.16.84.jar:?]
    at io.airbyte.workers.storage.S3DocumentStoreClient.read(S3DocumentStoreClient.java:75) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocument(AsyncOrchestratorPodProcess.java:216) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.process.AsyncOrchestratorPodProcess.checkStatus(AsyncOrchestratorPodProcess.java:220) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocStoreStatus(AsyncOrchestratorPodProcess.java:229) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:137) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:260) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 1 more
Log4j2Appender says: Activity failure. ActivityId=9b3a92ff-05ab-3f9a-a118-fc45f5360cef, activityType=Replicate, attempt=1
2022-05-25 09:34:38 ERROR i.t.i.s.WorkflowExecuteRunnable(logWorkflowExecutionException):125 - Workflow execution failure WorkflowId=sync_32, RunId=9963237c-9842-49ed-897d-719cfa04ecd5, WorkflowType=SyncWorkflow
io.temporal.failure.ActivityFailure: scheduledEventId=5, startedEventId=6, activityType='Replicate', activityId='9b3a92ff-05ab-3f9a-a118-fc45f5360cef', identity='1@airbyte-worker-788fcc8dd6-z556g', retryState=RETRY_STATE_NON_RETRYABLE_FAILURE
    at java.lang.Thread.getStackTrace(Thread.java:1610) ~[?:?]
    at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:48) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:77) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:70) ~[temporal-sdk-1.8.1.jar:?]
    at jdk.proxy2.$Proxy74.replicate(Unknown Source) ~[?:?]
    at io.airbyte.workers.temporal.sync.SyncWorkflowImpl.run(SyncWorkflowImpl.java:43) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:317) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:292) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.WorkflowExecuteRunnable.run(WorkflowExecuteRunnable.java:72) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:137) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:101) [temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:111) [temporal-sdk-1.8.1.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.lang.RuntimeException', nonRetryable=false
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:234) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.replicate(ReplicationActivityImpl.java:111) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method:0) ~[?:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.util.concurrent.ExecutionException', nonRetryable=false
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:154) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:229) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.replicate(ReplicationActivityImpl.java:111) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method:0) ~[?:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.lang.RuntimeException', nonRetryable=false
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:265) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='Running the launcher replication-orchestrator failed', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false
    at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:177) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:260) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 16F24F5DEDF251C3, Extended Request ID: null)', type='software.amazon.awssdk.services.s3.model.NoSuchBucketException', nonRetryable=false
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.16.84.jar:?]

Hey I could see this error The specified bucket does not exist (Service: S3, Status Code: 404 does this make sense? As can you check if this is true and fix this?

hi @harshith, airbyte write successfully in S3 bucket

s3

Collecting content into /tmp/toBePublished9902462365415211188.tmp before uploading.
Publishing to S3 (bucket=airbyte-datatest-bucket; key=job-logging/workspace/37/0/logs.log/20220526123242_airbyte-worker-85dd69bff7-xrlpt_9c411a4e2f7b470eaa1b0591310e67a1):

is there something missing from the configuration?

Airbyte Version: 0.39.1-alpha
Source: MySQL (0.5.11)
Destination: BigQuery (1.1.6)

Hey do you mind down-up of the whole instance and sharing the server/scheduler logs here?

@harshith this is the server and scheduler logs

scheduler logs

2022-05-26 16:07:23 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2022-05-26 16:07:23 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2022-05-26 16:07:23 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting...
2022-05-26 16:07:23 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed.
2022-05-26 16:07:27 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:29 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:31 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:33 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:35 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:37 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:39 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:41 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:43 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:45 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:47 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:49 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:51 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:53 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:55 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:57 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:59 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:01 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:03 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:05 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:07 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:09 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:16 INFO i.a.s.a.SchedulerApp(main):260 - Creating Job DB connection pool...
2022-05-26 16:08:16 WARN i.a.d.Database(createWithRetryTimeout):63 - Waiting for database to become available...
2022-05-26 16:08:16 INFO i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):26 - Testing if jobs database is ready...
2022-05-26 16:08:20 INFO i.a.d.Database(createWithRetry):52 - Database available!
2022-05-26 16:08:20 WARN i.a.d.Database(createWithRetryTimeout):63 - Waiting for database to become available...
2022-05-26 16:08:20 INFO i.a.d.i.c.ConfigsDatabaseInstance(lambda$static$3):28 - Testing if airbyte_configs has been created and seeded...
2022-05-26 16:08:20 INFO i.a.d.Database(createWithRetry):52 - Database available!
2022-05-26 16:08:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable MINIMUM_WORKSPACE_RETENTION_DAYS: '1'
2022-05-26 16:08:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable MAXIMUM_WORKSPACE_RETENTION_DAYS: '60'
2022-05-26 16:08:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable MAXIMUM_WORKSPACE_SIZE_MB: '5000'
2022-05-26 16:08:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-05-26 16:08:21 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '10'
2022-05-26 16:08:21 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable ACTIVITY_MAX_TIMEOUT_SECOND: '30'
2022-05-26 16:08:21 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable TEMPORAL_HISTORY_RETENTION_IN_DAYS: '30'
2022-05-26 16:08:21 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):174 - Waiting for temporal server...
2022-05-26 16:08:21 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):185 - Waiting for default namespace to be initialized in temporal...
2022-05-26 16:08:25 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-05-26 16:08:31 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):201 - Found temporal default namespace!
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable DD_AGENT_HOST: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable DD_DOGSTATSD_PORT: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable PUBLISH_METRICS: 'false'
2022-05-26 16:08:31 INFO i.a.s.a.SchedulerApp(main):296 - Launching scheduler...
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable SYNC_JOB_MAX_ATTEMPTS: '3'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_MAIN_CONTAINER_CPU_REQUEST: 'null'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_MAIN_CONTAINER_CPU_LIMIT: 'null'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_MAIN_CONTAINER_MEMORY_REQUEST: 'null'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_MAIN_CONTAINER_MEMORY_LIMIT: 'null'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_TOLERATIONS: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_NODE_SELECTORS: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_ANNOTATIONS: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_POLICY: 'IfNotPresent'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_SIDECAR_CONTAINER_IMAGE_PULL_POLICY: 'IfNotPresent'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_SOCAT_IMAGE: 'alpine/socat:1.7.4.1-r1'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_BUSYBOX_IMAGE: 'busybox:1.28'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_CURL_IMAGE: 'curlimages/curl:7.77.0'
2022-05-26 16:08:31 INFO i.a.c.f.EnvVariableFeatureFlags(usesNewScheduler):15 - New Scheduler: true (post-migration)

server logs

2022-05-27 04:06:06 INFO i.a.s.ServerApp(start):135 - 
    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
--------------------------------------
 Now ready at http://localhost:8000/
--------------------------------------
Version: 0.39.1-alpha

2022-05-27 04:06:12 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.133.213 GET 200 /api/v1/health
2022-05-27 04:06:19 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-27 04:06:22 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/web_backend/connections/get - {"connectionId":"62682e6f-86b1-4d36-abc6-693493c11667"}
2022-05-27 04:06:22 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/source_definitions/get - {"sourceDefinitionId":"435bb9a5-7887-4809-aa58-28c27df0d7ad"}
2022-05-27 04:06:22 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/destination_definitions/get - {"destinationDefinitionId":"22f6c74f-5699-40ff-833c-4a879ea40133"}
2022-05-27 04:06:23 WARN o.h.v.i.p.j.JavaBeanExecutable(getParameters):216 - HV000254: Missing parameter metadata for JobConfigType(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
2022-05-27 04:06:24 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:06:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/destination_definitions/get - {"destinationDefinitionId":"22f6c74f-5699-40ff-833c-4a879ea40133"}
2022-05-27 04:06:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/source_definitions/get - {"sourceDefinitionId":"435bb9a5-7887-4809-aa58-28c27df0d7ad"}
2022-05-27 04:06:27 INFO i.a.c.f.EnvVariableFeatureFlags(usesNewScheduler):15 - New Scheduler: true (post-migration)
2022-05-27 04:06:27 INFO i.a.w.t.TemporalClient(startNewManualSync):277 - Manual sync request
2022-05-27 04:06:28 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:31 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:34 INFO i.a.w.t.ConnectionManagerUtils(signalWorkflowAndRepairIfNecessary):85 - Retrieved existing connection manager workflow for connection 62682e6f-86b1-4d36-abc6-693493c11667. Executing signal.
2022-05-27 04:06:35 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:38 INFO i.a.w.t.TemporalClient(startNewManualSync):307 - end of manual schedule
2022-05-27 04:06:38 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:41 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/connections/sync - {"connectionId":"62682e6f-86b1-4d36-abc6-693493c11667"}
2022-05-27 04:06:42 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:06:49 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:52 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:58 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:07:05 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:07:15 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:07:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health

2022-05-27 04:07:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 GET 200 /api/v1/health
2022-05-27 04:08:05 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:08:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 GET 200 /api/v1/health
2022-05-27 04:08:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:09:05 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 GET 200 /api/v1/health
2022-05-27 04:09:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:09:41 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:09:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:49 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:52 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:55 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:59 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:10:02 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:10:05 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 GET 200 /api/v1/health
2022-05-27 04:10:07 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}

worker logs
server-logs.txt (190.1 KB)

Hey, when you said it created folders in s3 can you also check if there are log files inside? Because in the scheduler logs all the I can see is there is no permission for s3

If this is not successful temporarily I would suggest to try with minio and default settings to see if that works

hi @harshith, do you mean this?

do need to adjust the config in minio/server/worker besides adding a bucket, key access/secret?

Hey can you try running with minio and default settings and see if that works? This way we can try to eliminate some things

I’ve tried the default settings but it still doesn’t work, does it need to be reinstalled from scratch so that everything is clean?

Yeah do you mind just trying with default settings and fresh?