Slow performance on Airbyte installed on EC2 instance using ABCTL

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Airbyte installed on EC2 instance using ABCTL is running slowly, seeking assistance to identify the cause.


Question

Hello! I have installed Airbyte using ABCTL on an EC2 instance with 32GB of RAM and 8 vCPUs, but it is running very slowly.
It’s a stock installation using ABCTL local install. Does anyone know what might be causing this?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["slow-performance", "airbyte", "ec2-instance", "abctl", "installation"]
2

Second thing,
abctl uses kind cluster (https://kind.sigs.k8s.io/) under the hood

Can you check with k9s how many resources are pods inside kind cluster consuming?

Install kubectl (https://kubernetes.io/docs/tasks/tools/#kubectl) and kubectx + kubens (https://github.com/ahmetb/kubectx) and k9s (https://k9scli.io/)

then execute commands:

KUBECONFIG=~/.airbyte/abctl/abctl.kubeconfig kubectl config view --flatten > ~/.kube/config
kubectx kind-airbyte-abctl
kubens airbyte-abctl
k9s```
with `k9s` you will see details about pods
3

Can you check with ctop https://github.com/bcicen/ctop how many resources is airbyte-abctl-control-plane Docker container consuming?

4

what connectors do you use for synchronizations?

5

What exactly is slow? Installation? Synchronizations?

6

The sycing. Lots of times happens this

2024-09-09 13:48:51 destination > INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-09-09 13:49:51 destination > INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: 10 MB (10.0 MB), %% used: 0.016276041676772164 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-09-09 13:49:51 destination > INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-09-09 13:50:51 destination > INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: 10 MB (10.0 MB), %% used: 0.016276041676772164 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-09-09 13:50:51 destination > INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-09-09 13:51:51 destination > INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: 10 MB (10.0 MB), %% used: 0.016276041676772164 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-09-09 13:51:51 destination > INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-09-09 13:52:51 destination > INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: 10 MB (10.0 MB), %% used: 0.016276041676772164 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-09-09 13:52:51 destination > INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-09-09 13:53:51 destination > INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: 10 MB (10.0 MB), %% used: 0.016276041676772164 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-09-09 13:53:51 destination > INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-09-09 13:54:51 destination > INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: 10 MB (10.0 MB), %% used: 0.016276041676772164 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-09-09 13:54:51 destination > INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-09-09 13:55:51 destination > INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: 10 MB (10.0 MB), %% used: 0.016276041676772164 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-09-09 13:55:51 destination > INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0```
7

And also it can sync 2 connections at the same time