Unable to access airbyte webapp after connections error due to rate limits

  • Is this your first time deploying Airbyte?: No
  • Memory / Disk: not sure
  • Deployment: AWS EKS with 2 m5.2xlarge
  • Airbyte Version: 0.39.1
  • Step: The issue is happening when we try to access the webapp via the browser or make an API call to it
  • Description: Our airbyte deployment was working perfectly at the start. After we had created a few connections, where the connections were hitting rate limits, it seems like it has completely frozen our airbyte EKS instance. Attached is an image of the schedule syncs failing due to rate limits.

Any ideas on how to restore our airbyte instance back to working order? Here’s a sample of our logs from airbyte-webapp where the connection is being refused.

Hi @averyyip, the failed jobs might have impacted the resources and/or scheduling capabilities of the cluster. Could you try deleting the failed jobs using kubectl delete?

If the web app is not more responsive after that, there might be more meaningful errors in the server logs.

And if you’re able, please share the output of kubectl get pods in airbyte’s namespace, that would give me a glimpse into the health of your Airbyte deployment.

Thanks for the help Nataly. Below is a screenshot of kubectl get pods

I’ve tried restarting the cluster (restarting airbyte pods) and deleting the failed jobs, but it is still nonresponsive.

When I make an api call to the server (e.g. /workspaces/list), it would occassionally succeed or return a 502 error

Thanks for going through those steps! While I look into this further, could you check if you’re getting a different error message from the server than from the webapp? I see in that 502 error it says to check the error log for details.

Hi Nataly. We ended up nuking our airbyte because we were in a rush to get some things working. I’ll follow up on this thread if I encounter this issue again.

Sorry to hear you had to nuke it, good luck with your setup and I’m always ready to help!