Update to 0.40.4 - all syncs fail with Object not found. "io.airbyte.server.errors.IdNotFoundKnownException

andnig · September 5, 2022, 7:49am

Is this your first time deploying Airbyte?: No
OS Version / Instance: Ubuntu 22.04
Memory / Disk: 1TB
Deployment: Kubernetes
Airbyte Version: 0.40.4
Source name/version: Stripe/0.1.37
Destination name/version: BigQuery/1.1.16
Step:Sync

I was running airbyte now for several months without problems. After upgrading from 0.40.0 to 0.40.4, all my syncs fail with the following exception.
For deploying I use your kustomize “stable” overlay. No resources/overlays are adjusted.
Any hints, where I can start troubleshooting? Did I miss a migration between 0.40.0 and 0.40.4?

Exception

2022-09-05 07:45:00 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.4
2022-09-05 07:45:00 WARN i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=5aba9604-d72d-3dec-bb31-490392f7881c, activityType=RunWithJobOutput, attempt=3
io.airbyte.api.client.invoker.generated.ApiException: setWorkflowInAttempt call failed with: 404 - {"message":"Object not found. HTTP 404 Not Found","exceptionClassName":"io.airbyte.server.errors.IdNotFoundKnownException","exceptionStack":["io.airbyte.server.errors.IdNotFoundKnownException: Object not found. HTTP 404 Not Found","\tat io.airbyte.server.errors.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:22)","\tat io.airbyte.server.errors.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:14)","\tat org.glassfish.jersey.server.ServerRuntime$Responder.mapException(ServerRuntime.java:528)","\tat org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:403)","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:501)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)","\tat java.base/java.lang.Thread.run(Thread.java:1589)","Caused by: javax.ws.rs.NotFoundException: HTTP 404 Not Found","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)","\t... 39 more"],"rootCauseExceptionClassName":"java.lang.Class","rootCauseExceptionStack":["javax.ws.rs.NotFoundException: HTTP 404 Not Found","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:501)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)","\tat java.base/java.lang.Thread.run(Thread.java:1589)"]}
	at io.airbyte.api.client.generated.AttemptApi.getApiException(AttemptApi.java:68) ~[io.airbyte-airbyte-api-0.40.4.jar:?]
	at io.airbyte.api.client.generated.AttemptApi.setWorkflowInAttemptWithHttpInfo(AttemptApi.java:108) ~[io.airbyte-airbyte-api-0.40.4.jar:?]
	at io.airbyte.api.client.generated.AttemptApi.setWorkflowInAttempt(AttemptApi.java:86) ~[io.airbyte-airbyte-api-0.40.4.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.saveWorkflowIdForCancellation(TemporalAttemptExecution.java:147) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:114) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:79) ~[io.airbyte-airbyte-workers-0.40.4.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]

andnig · September 5, 2022, 8:18am

Ok, I migrated back to 0.40.1, then to 0.40.2, then to 0.40.3 and then again to 0.40.4.
This solved the issue.
Sorry for opening the issue prematurely.
But maybe it helps others who were similarly impatient and skipped some minor versions during upgrade

Knowing that the issue was purely my fault, but maybe it makes sense to add this note to the “Upgrading” docs? That one must not skip minor versions during upgrades?

harshith · September 6, 2022, 1:08pm

No problem. We are here to help. Great that you found the solution.

Topic		Replies	Views
Airbyte-Airflow Integration Issue: 'Object not found' Error When Updating Source API, Terraform and Other Topics airbyte , api , bug , airflow , 404-error	0	90	June 19, 2024
NoSuchBucket exception in S3DocumentStoreClient Platform, Deploy & Infra Issues kubernetes , deploy	7	1296	September 21, 2022
Kubernetes - Bigquery sync failed Connector Questions & Issues kubernetes	11	1347	July 14, 2022
Kubernetes Deploy: java.io.FileNotFoundException: /root/.kube/config (No such file or directory) Platform, Deploy & Infra Issues kubernetes , deploy	5	1036	July 14, 2022
Exception while emitting state count metrics for API records sync Platform Questions platform , question , 404-error , exception , state-count-metrics	4	129	May 16, 2024

Update to 0.40.4 - all syncs fail with Object not found. "io.airbyte.server.errors.IdNotFoundKnownException

Exception

Related topics