Long loading time

Q&A
,
1

Hi Team,

We have a self-hosted Airbyte instance on AWS EC2 instance of t2.2xlarge with an initial 50 GB of storage.
Recently, the loading time of connections, source, and destinations page is so long around 5 minutes and sometimes the HTTP request gets 502 error. It seems the bottleneck is this request:

GET http://localhost:8000/api/v1/web_backend/connections/list

image
image3123×1118 195 KB

image
image1920×992 92.8 KB

Now I see even after several minutes I get 502 error from connections list request:

image
image2549×1193 212 KB

It seems it is happening in recent updates.

FYI, we have around 30 connections that sync every hour.

Do you have any suggestion downgrade that fix this issue?

2

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

3

Hello can you check how the docker stats and how much memory/cpu Airbyte containers are using?

4

image
image2894×1193 383 KB

image
image2698×896 276 KB

image
image2158×270 16.5 KB

5

Hello Arash, the team is investigating a possible issue with memory leaky in the logging system. Until a solution is merged the workaround is to restart airbyte-server and airbyte-worker containers. I created the issue to track the problem: https://github.com/airbytehq/airbyte/issues/16652

6

Hello Marcos,
I restarted these containers, but the awful slowness still exists. I cannot even switch between pages. Are you sure this is the workaround?

7

Yep, it should work. How is the frequency of your connections? 5min?

8

The frequence is 1 hour.

9

Sorry the delay here @arashlayeghi no updates besides the workaround suggested. Probably upgrading to latest version can help. Latest version of Airbyte tries to optimize queries executed to the database reducing the memory footprint.

10

Hello Arash Layeghi, it’s been a while without an update from us. Are you still having problems or did you find a solution?