Summary
The user is facing issues with jobs not running in Airbyte hosted with Docker Compose on an EC2 instance. Error logs indicate service rate limit exceeded and task not found or already expired. The user is seeking help to understand the errors and recover from this state.
Question
Hello,
I am hosting Airbyte with docker compose in an EC2 instance and no jobs are running anymore. From the temporal container I see this logs:
“level”:“info”,“ts”:“2024-08-27T19:22:28.972Z”,“msg”:“history client encountered error”,“service”:“frontend”,“error”:“service rate limit exceeded”,“service-error-type”:“serviceerror.ResourceExhausted”,“logging-call-at”:“metric_client.go:90”}
And from the worker:
x2024-08-27 19:13:33 WARN i.t.i.w.WorkflowWorker$TaskHandlerImpl(logExceptionDuringResultReporting):416 - Failure while reporting workflow progress to the server. If seen continuously the workflow might be stuck. WorkflowId=connection_manager_6b4ace68-3ee4-4c0f-bc35-f2b8ff9e6d80, RunId=b49c49fd-0e6a-4413-9ff1-1214c9d52403, startedEventId=0
io.grpc.StatusRuntimeException: NOT_FOUND: query task not found, or already expired
Does anyone know what they mean and how to recover from this state? Or where to look for more information?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.