Unable to access airbyte webapp after connections error due to rate limits

Platform, Deploy & Infra Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • Memory / Disk: not sure
  • Deployment: AWS EKS with 2 m5.2xlarge
  • Airbyte Version: 0.39.1
  • Step: The issue is happening when we try to access the webapp via the browser or make an API call to it
  • Description: Our airbyte deployment was working perfectly at the start. After we had created a few connections, where the connections were hitting rate limits, it seems like it has completely frozen our airbyte EKS instance. Attached is an image of the schedule syncs failing due to rate limits.
    image
    image1302×498 57.7 KB

Any ideas on how to restore our airbyte instance back to working order? Here’s a sample of our logs from airbyte-webapp where the connection is being refused.

image
image3170×158 68.4 KB

2

Hi @averyyip, the failed jobs might have impacted the resources and/or scheduling capabilities of the cluster. Could you try deleting the failed jobs using kubectl delete?

If the web app is not more responsive after that, there might be more meaningful errors in the server logs.

And if you’re able, please share the output of kubectl get pods in airbyte’s namespace, that would give me a glimpse into the health of your Airbyte deployment.

3

Thanks for the help Nataly. Below is a screenshot of kubectl get pods

image
image1296×1266 149 KB

I’ve tried restarting the cluster (restarting airbyte pods) and deleting the failed jobs, but it is still nonresponsive.

When I make an api call to the server (e.g. /workspaces/list), it would occassionally succeed or return a 502 error

image
image1832×1106 186 KB

4

Thanks for going through those steps! While I look into this further, could you check if you’re getting a different error message from the server than from the webapp? I see in that 502 error it says to check the error log for details.

5

Hi Nataly. We ended up nuking our airbyte because we were in a rush to get some things working. I’ll follow up on this thread if I encounter this issue again.

6

Sorry to hear you had to nuke it, good luck with your setup and I’m always ready to help!

7

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.