Update to 0.40.4 - all syncs fail with Object not found. "io.airbyte.server.errors.IdNotFoundKnownException

Platform, Deploy & Infra Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu 22.04
  • Memory / Disk: 1TB
  • Deployment: Kubernetes
  • Airbyte Version: 0.40.4
  • Source name/version: Stripe/0.1.37
  • Destination name/version: BigQuery/1.1.16
  • Step:Sync

I was running airbyte now for several months without problems. After upgrading from 0.40.0 to 0.40.4, all my syncs fail with the following exception.
For deploying I use your kustomize “stable” overlay. No resources/overlays are adjusted.
Any hints, where I can start troubleshooting? Did I miss a migration between 0.40.0 and 0.40.4?

Exception

2022-09-05 07:45:00 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.4
2022-09-05 07:45:00 WARN i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=5aba9604-d72d-3dec-bb31-490392f7881c, activityType=RunWithJobOutput, attempt=3
io.airbyte.api.client.invoker.generated.ApiException: setWorkflowInAttempt call failed with: 404 - {"message":"Object not found. HTTP 404 Not Found","exceptionClassName":"io.airbyte.server.errors.IdNotFoundKnownException","exceptionStack":["io.airbyte.server.errors.IdNotFoundKnownException: Object not found. HTTP 404 Not Found","\tat io.airbyte.server.errors.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:22)","\tat io.airbyte.server.errors.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:14)","\tat org.glassfish.jersey.server.ServerRuntime$Responder.mapException(ServerRuntime.java:528)","\tat org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:403)","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:501)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)","\tat java.base/java.lang.Thread.run(Thread.java:1589)","Caused by: javax.ws.rs.NotFoundException: HTTP 404 Not Found","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)","\t... 39 more"],"rootCauseExceptionClassName":"java.lang.Class","rootCauseExceptionStack":["javax.ws.rs.NotFoundException: HTTP 404 Not Found","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:501)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)","\tat java.base/java.lang.Thread.run(Thread.java:1589)"]}
	at io.airbyte.api.client.generated.AttemptApi.getApiException(AttemptApi.java:68) ~[io.airbyte-airbyte-api-0.40.4.jar:?]
	at io.airbyte.api.client.generated.AttemptApi.setWorkflowInAttemptWithHttpInfo(AttemptApi.java:108) ~[io.airbyte-airbyte-api-0.40.4.jar:?]
	at io.airbyte.api.client.generated.AttemptApi.setWorkflowInAttempt(AttemptApi.java:86) ~[io.airbyte-airbyte-api-0.40.4.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.saveWorkflowIdForCancellation(TemporalAttemptExecution.java:147) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:114) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:79) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]
2

Ok, I migrated back to 0.40.1, then to 0.40.2, then to 0.40.3 and then again to 0.40.4.
This solved the issue.
Sorry for opening the issue prematurely.
But maybe it helps others who were similarly impatient and skipped some minor versions during upgrade :smile:

Knowing that the issue was purely my fault, but maybe it makes sense to add this note to the “Upgrading” docs? That one must not skip minor versions during upgrades?

3

No problem. We are here to help. Great that you found the solution.