Summary
User encounters a NullPointerException when attempting to start syncs after upgrading Airbyte from version 0.50.33 to 0.50.43 using the helm chart 0.50.21. The error occurs during the connection check and indicates that the connection context is null.
Question
I upgraded Airbyte from 0.50.33 to 0.50.43 yesterday. I used Airbyte helm chart 0.50.21. when I start the syncs I get the following errors
2024-01-17 00:32:42 platform > kubectl cp /tmp/54967273-a382-44c7-b931-8d27d29228f8/FINISHED_UPLOADING airbyte-jobs/source-google-sheets-check-43910-3-qirba:/config/FINISHED_UPLOADING -c init
2024-01-17 00:32:42 platform > Waiting for kubectl cp to complete
2024-01-17 00:32:42 platform > kubectl cp complete, closing process
2024-01-17 00:32:42 platform > Waiting until pod is ready...
2024-01-17 00:32:55 platform > Reading pod IP...
2024-01-17 00:32:55 platform > Pod IP: 10.0.193.69
2024-01-17 00:32:55 platform > Using null stdin output stream...
2024-01-17 00:32:55 platform > Unable to gobble line(s) from input stream provided by generic: input stream is null.
2024-01-17 00:32:55 platform > Unexpected error while checking connection:
java.lang.NullPointerException: null
at java.base/java.io.Reader.<init>(Reader.java:168) ~[?:?]</init>
at java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:123) ~[?:?]</init>
at <http://io.airbyte.commons.io|io.airbyte.commons.io>.IOs.newBufferedReader(IOs.java:74) ~[io.airbyte-airbyte-commons-0.50.43.jar:?]
at io.airbyte.workers.WorkerUtils.getMessagesByType(WorkerUtils.java:204) ~[io.airbyte-airbyte-commons-worker-0.50.43.jar:?]
at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:81) ~[io.airbyte-airbyte-commons-worker-0.50.43.jar:?]
at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.43.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:214) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.43.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:199) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-01-17 00:32:55 platform >
2024-01-17 00:32:55 platform > ----- END CHECK -----
2024-01-17 00:32:55 platform >
Logs from worker
WorkflowId=sync_43905, RunId=9add5941-e64a-486d-9e4e-7731830d4a5f. If seen continuously the workflow might be stuck.
io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event 29 of type 'EVENT_TYPE_WORKFLOW_TASK_COMPLETED' during replay. {WorkflowTaskStartedEventId=45, CurrentStartedEventId=28}
at io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:373) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:297) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:260) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:249) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:231) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:165) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:133) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:98) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:413) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:320) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:261) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.RuntimeException: WorkflowTask: failure executing STARTED->WORKFLOW_TASK_COMPLETED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED, SCHEDULED->WORKFLOW_TASK_STARTED]
at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:163) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:103) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:84) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:419) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:295) ~[temporal-sdk-1.22.3.jar:?]
... 13 more
Caused by: java.lang.NullPointerException: Cannot invoke "io.airbyte.config.ConnectionContext.getOrganizationId()" because the return value of "io.airbyte.config.StandardSyncInput.getConnectionContext()" is null
at io.airbyte.workers.temporal.sync.SyncWorkflowImpl.generateNormalizationInput(SyncWorkflowImpl.java:199) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at io.airbyte.workers.temporal.sync.SyncWorkflowImpl.run(SyncWorkflowImpl.java:143) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
at SyncWorkflowImplProxy.run$accessor$IMXFIkcG(Unknown Source) ~[?:?]
at SyncWorkflowImplProxy$auxiliary$mae4vIO5.call(Unknown Source) ~[?:?]
at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) ~[io.airbyte-airbyte-micronaut-temporal-0.50.43.jar:?]
at SyncWorkflowImplProxy.run(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
... 3 more
2024-01-17 01:38:08 [1;31mERROR[m i.t.i.w.PollerOptions$Builder(lambda$build$0):168 - uncaught exception
java.lang.RuntimeException: Failure processing workflow task. WorkflowId=sync_43905, RunId=9add5941-e64a-486d-9e4e-7731830d4a5f, Attempt=20
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:404) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:261) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:110) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event 29 of type 'EVENT_TYPE_WORKFLOW_TASK_COMPLETED' during replay. {WorkflowTaskStartedEventId=45, CurrentStartedEventId=28}
at io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:373) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:297) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:260) ~[temporal-sdk-1.22.3.jar:?]
Caused by: java.lang.RuntimeException: WorkflowTask: failure executing STARTED->WORKFLOW_TASK_COMPLETED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED, SCHEDULED->WORKFLOW_TASK_STARTED]
at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:163) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:103) ~[temporal-sdk-1.22.3.jar:?]
Caused by: java.lang.NullPointerException: Cannot invoke "io.airbyte.config.ConnectionContext.getOrganizationId()" because the return value of "io.airbyte.config.StandardSyncInput.getConnectionContext()" is null
at io.airbyte.workers.temporal.sync.SyncWorkflowImpl.generateNormalizationInput(SyncWorkflowImpl.java:199) ~[io.airbyte-airbyte-workers-0.50.43.jar:?]
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.