Optimizing pod placement in EKS for Airbyte syncs


How to ensure Airbyte sync pods run in the same Availability Zone in EKS to reduce EC2 Transfer costs without compromising high availability


Hi Airbyte crew! I am wondering how any of you may have solved this problem or have an idea on how to solve this problem. We have Airbyte deployed in EKS, running on a node group which is split across 3 Availability Zones. Because of this, sometimes pods for syncs are put onto different nodes in different AZs, causing increased costs in EC2 - Transfer. Is there a way to make it so that these pods all run in the same AZ/node so that there isn’t the increased cost without reducing the node group to one AZ and reducing how HA the service is? Thanks!

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["eks", "airbyte", "pod-placement", "availability-zone", "ec2-transfer-costs"]

If you want to have your jobs run in a specific AZ you can use the helm value at global.jobs.kube.nodeSelector

I am not 100% sure since I didn’t set up the EKS cluster at my company, but I think the nodes automatically get a topology label for AZ IDs which is <http://topology.kubernetes.io/zone|topology.kubernetes.io/zone>. For example, if I wanted the pods to be in us-west-2a I would do:

    <http://topology.kubernetes.io/zone|topology.kubernetes.io/zone>: "us-west-2a"```
So then it will only put it in the nodes with that label.