Kubernetes - Bigquery sync failed

Asulaiman · May 25, 2022, 10:43am

Is this your first time deploying Airbyte?: / Yes
Deployment: Are you using Docker or Kubernetes deployment? Kubernetes
Airbyte Version: What version are you using now?
Source name/version: 0.39.1-alpha
Destination name/version: Big Query
Step: BigQuery sync failed
Description:

I’m new to airbyte and having trouble setting up bigquery. i’m using S3 to keep logs and already able to write to it , problem appears while doing bigquery sync.
please advise

Log4j2Appender says: Completing future exceptionally...
2022-05-25 09:34:38 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
Log4j2Appender says: Stopping cancellation check scheduling...
2022-05-25 09:34:38 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):236 - Stopping temporal heartbeating...
Log4j2Appender says: Stopping temporal heartbeating...
2022-05-25 09:34:38 WARN i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=9b3a92ff-05ab-3f9a-a118-fc45f5360cef, activityType=Replicate, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:234) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.replicate(ReplicationActivityImpl.java:111) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.8.1.jar:?]
    at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:448) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.8.1.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:138) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:154) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:229) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 14 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:154) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:229) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 14 more
Caused by: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:265) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 1 more
Caused by: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
    at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:177) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:260) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 1 more
Caused by: software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 16F24F5DEDF251C3, Extended Request ID: null)
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:135) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:161) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:84) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:62) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:62) ~[aws-core-2.16.84.jar:?]
    at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:4371) ~[s3-2.16.84.jar:?]
    at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:8123) ~[s3-2.16.84.jar:?]
    at io.airbyte.workers.storage.S3DocumentStoreClient.read(S3DocumentStoreClient.java:75) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocument(AsyncOrchestratorPodProcess.java:216) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.process.AsyncOrchestratorPodProcess.checkStatus(AsyncOrchestratorPodProcess.java:220) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocStoreStatus(AsyncOrchestratorPodProcess.java:229) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:137) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:260) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    ... 1 more
Log4j2Appender says: Activity failure. ActivityId=9b3a92ff-05ab-3f9a-a118-fc45f5360cef, activityType=Replicate, attempt=1
2022-05-25 09:34:38 ERROR i.t.i.s.WorkflowExecuteRunnable(logWorkflowExecutionException):125 - Workflow execution failure WorkflowId=sync_32, RunId=9963237c-9842-49ed-897d-719cfa04ecd5, WorkflowType=SyncWorkflow
io.temporal.failure.ActivityFailure: scheduledEventId=5, startedEventId=6, activityType='Replicate', activityId='9b3a92ff-05ab-3f9a-a118-fc45f5360cef', identity='1@airbyte-worker-788fcc8dd6-z556g', retryState=RETRY_STATE_NON_RETRYABLE_FAILURE
    at java.lang.Thread.getStackTrace(Thread.java:1610) ~[?:?]
    at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:48) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:77) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:70) ~[temporal-sdk-1.8.1.jar:?]
    at jdk.proxy2.$Proxy74.replicate(Unknown Source) ~[?:?]
    at io.airbyte.workers.temporal.sync.SyncWorkflowImpl.run(SyncWorkflowImpl.java:43) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:317) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:292) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.WorkflowExecuteRunnable.run(WorkflowExecuteRunnable.java:72) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:137) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:101) [temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:111) [temporal-sdk-1.8.1.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.lang.RuntimeException', nonRetryable=false
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:234) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.replicate(ReplicationActivityImpl.java:111) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method:0) ~[?:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.util.concurrent.ExecutionException', nonRetryable=false
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:154) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:229) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.replicate(ReplicationActivityImpl.java:111) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method:0) ~[?:?]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.lang.RuntimeException', nonRetryable=false
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:265) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='Running the launcher replication-orchestrator failed', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false
    at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:177) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:260) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:88) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.1-alpha.jar:?]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 16F24F5DEDF251C3, Extended Request ID: null)', type='software.amazon.awssdk.services.s3.model.NoSuchBucketException', nonRetryable=false
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.16.84.jar:?]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.16.84.jar:?]

harshith · May 25, 2022, 12:07pm

Hey I could see this error The specified bucket does not exist (Service: S3, Status Code: 404 does this make sense? As can you check if this is true and fix this?

Asulaiman · May 26, 2022, 11:47am

hi @harshith, airbyte write successfully in S3 bucket

Collecting content into /tmp/toBePublished9902462365415211188.tmp before uploading.
Publishing to S3 (bucket=airbyte-datatest-bucket; key=job-logging/workspace/37/0/logs.log/20220526123242_airbyte-worker-85dd69bff7-xrlpt_9c411a4e2f7b470eaa1b0591310e67a1):

is there something missing from the configuration?

Airbyte Version: 0.39.1-alpha
Source: MySQL (0.5.11)
Destination: BigQuery (1.1.6)

harshith · May 26, 2022, 5:53pm

Hey do you mind down-up of the whole instance and sharing the server/scheduler logs here?

Asulaiman · May 26, 2022, 10:34pm

@harshith this is the server and scheduler logs

scheduler logs

2022-05-26 16:07:23 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2022-05-26 16:07:23 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2022-05-26 16:07:23 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting...
2022-05-26 16:07:23 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed.
2022-05-26 16:07:27 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:29 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:31 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:33 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:35 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:37 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:39 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:41 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:43 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:45 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:47 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:49 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:51 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:53 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:55 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:57 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:07:59 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:01 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:03 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:05 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:07 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:09 INFO i.a.s.a.SchedulerApp(waitForServer):225 - Waiting for server to become available...
2022-05-26 16:08:16 INFO i.a.s.a.SchedulerApp(main):260 - Creating Job DB connection pool...
2022-05-26 16:08:16 WARN i.a.d.Database(createWithRetryTimeout):63 - Waiting for database to become available...
2022-05-26 16:08:16 INFO i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):26 - Testing if jobs database is ready...
2022-05-26 16:08:20 INFO i.a.d.Database(createWithRetry):52 - Database available!
2022-05-26 16:08:20 WARN i.a.d.Database(createWithRetryTimeout):63 - Waiting for database to become available...
2022-05-26 16:08:20 INFO i.a.d.i.c.ConfigsDatabaseInstance(lambda$static$3):28 - Testing if airbyte_configs has been created and seeded...
2022-05-26 16:08:20 INFO i.a.d.Database(createWithRetry):52 - Database available!
2022-05-26 16:08:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable MINIMUM_WORKSPACE_RETENTION_DAYS: '1'
2022-05-26 16:08:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable MAXIMUM_WORKSPACE_RETENTION_DAYS: '60'
2022-05-26 16:08:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable MAXIMUM_WORKSPACE_SIZE_MB: '5000'
2022-05-26 16:08:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-05-26 16:08:21 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '10'
2022-05-26 16:08:21 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable ACTIVITY_MAX_TIMEOUT_SECOND: '30'
2022-05-26 16:08:21 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable TEMPORAL_HISTORY_RETENTION_IN_DAYS: '30'
2022-05-26 16:08:21 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):174 - Waiting for temporal server...
2022-05-26 16:08:21 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):185 - Waiting for default namespace to be initialized in temporal...
2022-05-26 16:08:25 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-05-26 16:08:31 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):201 - Found temporal default namespace!
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable DD_AGENT_HOST: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable DD_DOGSTATSD_PORT: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable PUBLISH_METRICS: 'false'
2022-05-26 16:08:31 INFO i.a.s.a.SchedulerApp(main):296 - Launching scheduler...
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable SYNC_JOB_MAX_ATTEMPTS: '3'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_MAIN_CONTAINER_CPU_REQUEST: 'null'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_MAIN_CONTAINER_CPU_LIMIT: 'null'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_MAIN_CONTAINER_MEMORY_REQUEST: 'null'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_MAIN_CONTAINER_MEMORY_LIMIT: 'null'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_TOLERATIONS: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_NODE_SELECTORS: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_ANNOTATIONS: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET: ''
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_POLICY: 'IfNotPresent'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_SIDECAR_CONTAINER_IMAGE_PULL_POLICY: 'IfNotPresent'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_SOCAT_IMAGE: 'alpine/socat:1.7.4.1-r1'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_BUSYBOX_IMAGE: 'busybox:1.28'
2022-05-26 16:08:31 INFO i.a.c.EnvConfigs(getEnvOrDefault):893 - Using default value for environment variable JOB_KUBE_CURL_IMAGE: 'curlimages/curl:7.77.0'
2022-05-26 16:08:31 INFO i.a.c.f.EnvVariableFeatureFlags(usesNewScheduler):15 - New Scheduler: true (post-migration)

server logs

2022-05-27 04:06:06 INFO i.a.s.ServerApp(start):135 - 
    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
--------------------------------------
 Now ready at http://localhost:8000/
--------------------------------------
Version: 0.39.1-alpha

2022-05-27 04:06:12 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.133.213 GET 200 /api/v1/health
2022-05-27 04:06:19 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-27 04:06:22 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/web_backend/connections/get - {"connectionId":"62682e6f-86b1-4d36-abc6-693493c11667"}
2022-05-27 04:06:22 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/source_definitions/get - {"sourceDefinitionId":"435bb9a5-7887-4809-aa58-28c27df0d7ad"}
2022-05-27 04:06:22 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/destination_definitions/get - {"destinationDefinitionId":"22f6c74f-5699-40ff-833c-4a879ea40133"}
2022-05-27 04:06:23 WARN o.h.v.i.p.j.JavaBeanExecutable(getParameters):216 - HV000254: Missing parameter metadata for JobConfigType(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
2022-05-27 04:06:24 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:06:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/destination_definitions/get - {"destinationDefinitionId":"22f6c74f-5699-40ff-833c-4a879ea40133"}
2022-05-27 04:06:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/source_definitions/get - {"sourceDefinitionId":"435bb9a5-7887-4809-aa58-28c27df0d7ad"}
2022-05-27 04:06:27 INFO i.a.c.f.EnvVariableFeatureFlags(usesNewScheduler):15 - New Scheduler: true (post-migration)
2022-05-27 04:06:27 INFO i.a.w.t.TemporalClient(startNewManualSync):277 - Manual sync request
2022-05-27 04:06:28 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:31 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:34 INFO i.a.w.t.ConnectionManagerUtils(signalWorkflowAndRepairIfNecessary):85 - Retrieved existing connection manager workflow for connection 62682e6f-86b1-4d36-abc6-693493c11667. Executing signal.
2022-05-27 04:06:35 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:38 INFO i.a.w.t.TemporalClient(startNewManualSync):307 - end of manual schedule
2022-05-27 04:06:38 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:41 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/connections/sync - {"connectionId":"62682e6f-86b1-4d36-abc6-693493c11667"}
2022-05-27 04:06:42 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:06:49 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:52 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:06:58 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:07:05 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:07:15 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:07:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health

2022-05-27 04:07:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 GET 200 /api/v1/health
2022-05-27 04:08:05 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:08:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 GET 200 /api/v1/health
2022-05-27 04:08:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:09:05 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 GET 200 /api/v1/health
2022-05-27 04:09:25 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:09:41 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 GET 200 /api/v1/health
2022-05-27 04:09:45 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:49 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:52 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:55 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:09:59 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:10:02 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}
2022-05-27 04:10:05 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.132.91 GET 200 /api/v1/health
2022-05-27 04:10:07 INFO i.a.s.RequestLogger(filter):112 - REQ 172.31.131.128 POST 200 /api/v1/jobs/list - {"configId":"62682e6f-86b1-4d36-abc6-693493c11667","configTypes":["sync","reset_connection"]}

worker logs
server-logs.txt (190.1 KB)

harshith · May 27, 2022, 4:23am

Hey, when you said it created folders in s3 can you also check if there are log files inside? Because in the scheduler logs all the I can see is there is no permission for s3

If this is not successful temporarily I would suggest to try with minio and default settings to see if that works

Asulaiman · May 27, 2022, 6:57am

hi @harshith, do you mean this?

do need to adjust the config in minio/server/worker besides adding a bucket, key access/secret?

harshith · May 30, 2022, 6:11am

Hey can you try running with minio and default settings and see if that works? This way we can try to eliminate some things

Asulaiman · May 31, 2022, 7:29am

I’ve tried the default settings but it still doesn’t work, does it need to be reinstalled from scratch so that everything is clean?

harshith · June 1, 2022, 4:35am

Yeah do you mind just trying with default settings and fresh?

marcosmarxm · July 13, 2022, 12:00am

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.

Topic		Replies	Views
Failure Origin: replication, Message: Something went wrong during replication Connector Questions & Issues getting-started , data-loading	5	1563	March 3, 2023
Airbyte/destination-bigquery 400 Bad Request Invalid credential Connector Questions & Issues destination-bigquery , data-loading , connectors	3	394	May 13, 2022
Source Postgres: "Sync worker failed. Source cannot be stopped!" Connector Questions & Issues source-postgres , destination-bigquery , data-loading , connectors	2	692	July 19, 2022
Suddenly after two days some of the syncs are failing Connector Questions & Issues source-mysql , destination-bigquery	9	604	October 4, 2022
MySQL to BigQuery fails without a reason Connector Questions & Issues destination-bigquery , normalization , data-loading	4	324	August 9, 2022

Kubernetes - Bigquery sync failed

Related Topics