The user is looking for ways to avoid AWS egress costs when hosting Airbyte in AWS with GCP BigQuery as the main data warehouse.
Hi, I host airbyte in AWS but main data warehouse is GCP bigquery. Do you have any idea how to avoid aws egress costs?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
So short of moving Airbyte to GCP
Thank you for your answer.
But all data sources are still in AWS. Do you think something like AWS PrivateLink could be a solution?
No, PrivateLink would only let you expose AWS services—connecting the two clouds would still result in egress costs. It’s possible using <https://cloud.google.com/network-connectivity/docs/interconnect/how-to/cci/aws/connectivity-overview|physical peering>, but I’m guessing the setup costs there would mean you’d be needing to do petabytes of volume to make it make sense.
How much data are you talking about per month? How much money is it costing you per month in egress fees? Until a certain scale, the time it takes to set up and maintain a solution at your billable hourly rate will likely exceed that. And at the point that you’re at that volume, you probably have access to a cloud specialist at Google who can help you architect a better solution.
Everything I can think of would both add complexity and probably higher costs than simple egress. So short of moving Airbyte to GCP I’m not sure there’s a solution that would be more affordable than just paying egress.