Airbyte Proxy - Nginx 504 Gateway Time-out EC2 Docker deployment

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: EC2 Amazon Linux t2.medium
  • Memory / Disk: 4 GB / 200 GB
  • Deployment: Docker
  • Airbyte Version: 0.40.19
  • Step: using Python
    conn = http.client.HTTPConnection("my_ip", 8000)
    
    
    payload = json.dumps({
  "connectionId": connectionId
})
    headers = {
  'Content-Type': 'application/json'
}
    conn.request("POST", "/api/v1/connections/sync", payload, headers)
    res = conn.getresponse()
  • Description: I recieve this error:
b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx/1.23.2</center>\r\n</body>\r\n</html>\r\n'

In logs:

2022-11-27 12:32:30 e[1;31mERRORe[m i.a.c.t.ConnectionManagerUtils(signalWorkflowAndRepairIfNecessary):116 - Failed to retrieve ConnectionManagerWorkflow for connection f687c96c-c78a-408d-93bf-1a4236768ad3. Repairing state by creating new workflow and starting with the signal.
io.airbyte.commons.temporal.exception.UnreachableWorkflowException: Failed to retrieve ConnectionManagerWorkflow for connection f687c96c-c78a-408d-93bf-1a4236768ad3 due to the following error:
	at io.airbyte.commons.temporal.ConnectionManagerUtils.getConnectionManagerWorkflow(ConnectionManagerUtils.java:192) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
	at io.airbyte.commons.temporal.ConnectionManagerUtils.signalWorkflowAndRepairIfNecessary(ConnectionManagerUtils.java:104) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
	at io.airbyte.commons.temporal.ConnectionManagerUtils.signalWorkflowAndRepairIfNecessary(ConnectionManagerUtils.java:67) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
	at io.airbyte.commons.temporal.TemporalClient.startNewManualSync(TemporalClient.java:210) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
	at io.airbyte.server.scheduler.TemporalEventRunner.startNewManualSync(TemporalEventRunner.java:27) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
	at io.airbyte.server.handlers.SchedulerHandler.submitManualSyncToWorker(SchedulerHandler.java:417) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
	at io.airbyte.server.handlers.SchedulerHandler.syncConnection(SchedulerHandler.java:347) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
	at io.airbyte.server.apis.ConnectionApiController.lambda$syncConnection$7(ConnectionApiController.java:77) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:17) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
	at io.airbyte.server.apis.ConnectionApiController.syncConnection(ConnectionApiController.java:77) ~[io.airbyte-airbyte-server-0.40.19.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.temporal.client.WorkflowQueryException: workflowId='connection_manager_f687c96c-c78a-408d-93bf-1a4236768ad3', runId='', workflowType='ConnectionManagerWorkflow'}
	at io.temporal.internal.sync.WorkflowStubImpl.throwAsWorkflowFailureExceptionForQuery(WorkflowStubImpl.java:371) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.query(WorkflowStubImpl.java:274) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.queryWorkflow(WorkflowInvocationHandler.java:309) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:272) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) ~[temporal-sdk-1.17.0.jar:?]
	at jdk.proxy2.$Proxy91.getState(Unknown Source) ~[?:?]
	at io.airbyte.commons.temporal.ConnectionManagerUtils.getConnectionManagerWorkflow(ConnectionManagerUtils.java:188) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
	... 54 more
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: java.lang.IllegalArgumentException: Unknown query type: getState, knownTypes=[]
	at io.temporal.internal.sync.QueryDispatcher.handleQuery(QueryDispatcher.java:80)
	at io.temporal.internal.sync.SyncWorkflowContext.handleQuery(SyncWorkflowContext.java:280)
	at io.temporal.internal.sync.WorkflowExecuteRunnable.handleQuery(WorkflowExecuteRunnable.java:111)
	at io.temporal.internal.sync.SyncWorkflow.query(SyncWorkflow.java:168)
	at io.temporal.internal.replay.ReplayWorkflowExecutor.query(ReplayWorkflowExecutor.java:138)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleDirectQueryWorkflowTask(ReplayWorkflowRunTaskHandler.java:180)
	at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:126)
	at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:97)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:336)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:246)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:188)
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1589)

	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.queryWorkflow(WorkflowServiceGrpc.java:4099) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.client.external.GenericWorkflowClientImpl.lambda$query$9(GenericWorkflowClientImpl.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.client.external.GenericWorkflowClientImpl.query(GenericWorkflowClientImpl.java:201) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.client.RootWorkflowClientInvoker.query(RootWorkflowClientInvoker.java:140) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.query(WorkflowStubImpl.java:270) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.queryWorkflow(WorkflowInvocationHandler.java:309) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:272) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) ~[temporal-sdk-1.17.0.jar:?]
	at jdk.proxy2.$Proxy91.getState(Unknown Source) ~[?:?]
	at io.airbyte.commons.temporal.ConnectionManagerUtils.getConnectionManagerWorkflow(ConnectionManagerUtils.java:188) ~[io.airbyte-airbyte-commons-temporal-0.40.19.jar:?]
	... 54 more

@natalyjazzviolin any updates on this? thanks

Hello rytis-molecule, it’s been a while without an update from us. Are you still having problems or did you find a solution?

Hello @marcosmarxm, no update from you so far… Thanks for catching up.

Did you try to edit this env variable

BASIC_AUTH_PROXY_TIMEOUT=600

https://github.com/airbytehq/airbyte/blob/5ceb14a91ef4c75b72cc46cf91267db6d54fdb68/.env#L39