Summary
After upgrading Airbyte to version 0.50.43 and using Helm chart 0.50.21, errors occur during syncs related to a NullPointerException in the connection context, causing workflow tasks to fail.
Question
I upgraded Airbyte from 0.50.33 to 0.50.43 yesterday. I used Airbyte helm chart 0.50.21. when I start the syncs I get the following errors
2024-01-17 00:32:42 platform > kubectl cp /tmp/54967273-a382-44c7-b931-8d27d29228f8/FINISHED_UPLOADING airbyte-jobs/source-google-sheets-check-43910-3-qirba:/config/FINISHED_UPLOADING -c init
2024-01-17 00:32:42 platform > Waiting for kubectl cp to complete
2024-01-17 00:32:42 platform > kubectl cp complete, closing process
2024-01-17 00:32:42 platform > Waiting until pod is ready...
2024-01-17 00:32:55 platform > Reading pod IP...
2024-01-17 00:32:55 platform > Pod IP: 10.0.193.69
2024-01-17 00:32:55 platform > Using null stdin output stream...
2024-01-17 00:32:55 platform > Unable to gobble line(s) from input stream provided by generic: input stream is null.
2024-01-17 00:32:55 platform > Unexpected error while checking connection:
java.lang.NullPointerException: null
at java.base/java.io.Reader.<init>(Reader.java:168) ~[?:?]</init>
at java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:123) ~[?:?]</init>
at <http://io.airbyte.commons.io|io.airbyte.commons.io>.IOs.newBufferedReader(IOs.java:74) ~[io.airbyte-airbyte-commons-0.50.43.jar:?]
at io.airbyte.workers.WorkerUtils.getMessagesByType(WorkerUtils.java:204) ~[io.airbyte-airbyte-commons-worker-0.50.43.jar:?]
at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:81) ~[io.airbyte-airbyte-commons-worker-0.50.43.jar:?]
at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.43.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:214) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.43.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:199) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-01-17 00:32:55 platform >
2024-01-17 00:32:55 platform > ----- END CHECK -----
2024-01-17 00:32:55 platform >
Logs from worker
WorkflowId=sync_43905, RunId=9add5941-e64a-486d-9e4e-7731830d4a5f. If seen continuously the workflow might be stuck.
io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event 29 of type 'EVENT_TYPE_WORKFLOW_TASK_COMPLETED' during replay. {WorkflowTaskStartedEventId=45, CurrentStartedEventId=28}
at io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:373) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:297) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:260) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:249) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:231) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:165) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:133) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:98) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:413) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:320) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:261) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.RuntimeException: WorkflowTask: failure executing STARTED->WORKFLOW_TASK_COMPLETED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED, SCHEDULED->WORKFLOW_TASK_STARTED]
at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:163) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:103) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:84) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:419) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:295) ~[temporal-sdk-1.22.3.jar:?]
... 13 more
Caused by: java.lang.NullPointerException: Cannot invoke "io.airbyte.config.ConnectionContext.getOrganizationId()" because the return value of "io.airbyte.config.StandardSyncInput.getConnectionContext()" is null
at io.airbyte.workers.temporal.sync.SyncWorkflowImpl.generateNormalizationInput(SyncWorkflowImpl.java:199) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at io.airbyte.workers.temporal.sync.SyncWorkflowImpl.run(SyncWorkflowImpl.java:143) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at SyncWorkflowImplProxy.run$accessor$IMXFIkcG(Unknown Source) ~[?:?]
at SyncWorkflowImplProxy$auxiliary$mae4vIO5.call(Unknown Source) ~[?:?]
at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) ~[io.airbyte-airbyte-micronaut-temporal-0.50.43.jar:?]
at SyncWorkflowImplProxy.run(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
... 3 more
2024-01-17 01:38:08 [1;31mERROR[m i.t.i.w.PollerOptions$Builder(lambda$build$0):168 - uncaught exception
java.lang.RuntimeException: Failure processing workflow task. WorkflowId=sync_43905, RunId=9add5941-e64a-486d-9e4e-7731830d4a5f, Attempt=20
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:404) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:261) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:110) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event 29 of type 'EVENT_TYPE_WORKFLOW_TASK_COMPLETED' during replay. {WorkflowTaskStartedEventId=45, CurrentStartedEventId=28}
at io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:373) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:297) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:260) ~[temporal-sdk-1.22.3.jar:?]
Caused by: java.lang.RuntimeException: WorkflowTask: failure executing STARTED->WORKFLOW_TASK_COMPLETED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED, SCHEDULED->WORKFLOW_TASK_STARTED]
at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:163) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:103) ~[temporal-sdk-1.22.3.jar:?]
Caused by: java.lang.NullPointerException: Cannot invoke "io.airbyte.config.ConnectionContext.getOrganizationId()" because the return value of "io.airbyte.config.StandardSyncInput.getConnectionContext()" is null
at io.airbyte.workers.temporal.sync.SyncWorkflowImpl.generateNormalizationInput(SyncWorkflowImpl.java:199) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.