Scaling concurrent syncs and memory allocation in abctl deployment

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is facing issues with scaling concurrent syncs in an abctl deployment, where the 5th sync gets stuck in pending until one of the 4 wraps up. Additionally, they want to allow for near full memory utilization in the cluster. They are seeking guidance on how to scale the concurrent syncs and adjust memory allocation.


Question

I migrated my docker-compose instance of airbyte in favor of a abctl deployment but I still have 2 questions that I cannot find in the docs or in this slack anywhere.

  1. My syncs are limited to 4 concurrent syncs and I want to scale. when I look into the logs of the 5th concurrent sync, the workload seems to be stuck in pending until one of the 4 wraps up. How do I scale this?
  2. ~Might be related to #1, but my cluster seems to only have half of the memory allocated, and I want near full utilization. I found this out by running docker exec -it airbyte-abctl-control-plane kubectl -n airbyte-abctl describe nodes and seeing the allocation limit is only at half of the total memory capacity. How do I allow for near full utilization?~ nvm on this, had a misunderstanding
    log where the 5th sync gets stuck:
2024-09-17 16:04:02 platform > Workload 843266b5-7f6c-487a-8c7f-32efd760017d_114289_0_sync is claimed
2024-09-17 16:06:02 platform > Workload 843266b5-7f6c-487a-8c7f-32efd760017d_114289_0_sync is claimed
2024-09-17 16:08:02 platform > Workload 843266b5-7f6c-487a-8c7f-32efd760017d_114289_0_sync is claimed```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1726591258099079) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["scaling", "concurrent-syncs", "memory-allocation", "abctl-deployment", "pending-workload", "full-memory-utilization"]
</sub>
2

not sure why this works, my guess is the default resource allocation was very high and airbyte refused to add more pods to not overallocate requested resources? might be talking out of my butt here

3

this is something i wish the documentation would’ve addressed for abctl deployment specifically.

4

you need to restart the server to make it applied to syncs

5

not sure why this works, my guess is the default resource allocation was very high and airbyte refused to add more pods to not overallocate requested resources? might be talking out of my butt here
You’re correct.

6

this is resolved. <Slack message> led me to try setting some resource limits in my values.yaml file:

  enabled: false

global:
  database:
    ...

  worker:
    ...

  jobs:
    resources:
      requests:
         memory: 1Gi
         cpu: 250m
      limit:
         memory: 4Gi
         cpu: 2```