Choosing compute power and memory for moving 300GB of data using Airbyte

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Determining the required compute power and memory for moving 300GB of data using Airbyte with 10 connectors working simultaneously


Question

Hi Team,

We are going to move close to 300 gb of data using airbyte.

For this we would like to buy a physical server.

What compute power and memory should we go for?

We will have less close to 10 connectors working at the same time.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["compute-power", "memory", "data-migration", "airbyte-platform", "connectors"]
2

Are you not using a cloud service provider?

3

Hi <@U064R6L3ZBJ>,

We have everything running on physical servers

4

we want this to run on physical servers too

5

I assume you’ve already done a cost benefit analysis on that, so won’t try to convince you on that one way or the other. But the size of the batches doesn’t really change for Airbyte so I feel like adding more memory wont really help a whole lot, CPU might not help a whole lot too – you would be limited by the batch size of the connector. Throwing more “power” at it doesn’t necessarily make it sync faster. I think you would be fine with 8 cores and 32 GB ram which is way more than we use for Airbyte in Kubernetes

6

See https://docs.airbyte.com/operator-guides/scaling-airbyte#what-to-scale

7

I’m not sure how you organize your stuff on-prem, but Airbyte can probably coexist on another machine you have if you are using a hypervisor, I don’t know that it makes sense to get a dedicated physical server for it. We run a lot of on-prem at my other company and we don’t dedicate a server for each app anymore

8

We are planning
Airbyte
Clickhouse
Superset

in one machine