Optimizing pod placement in EKS for Airbyte syncs

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

How to ensure Airbyte sync pods run in the same Availability Zone in EKS to reduce EC2 Transfer costs without compromising high availability


Question

Hi Airbyte crew! I am wondering how any of you may have solved this problem or have an idea on how to solve this problem. We have Airbyte deployed in EKS, running on a node group which is split across 3 Availability Zones. Because of this, sometimes pods for syncs are put onto different nodes in different AZs, causing increased costs in EC2 - Transfer. Is there a way to make it so that these pods all run in the same AZ/node so that there isn’t the increased cost without reducing the node group to one AZ and reducing how HA the service is? Thanks!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["eks", "airbyte", "pod-placement", "availability-zone", "ec2-transfer-costs"]
2

If you want to have your jobs run in a specific AZ you can use the helm value at global.jobs.kube.nodeSelector
https://github.com/airbytehq/airbyte-platform/blob/main/charts/airbyte/values.yaml#L123-L126

I am not 100% sure since I didn’t set up the EKS cluster at my company, but I think the nodes automatically get a topology label for AZ IDs which is <http://topology.kubernetes.io/zone|topology.kubernetes.io/zone>. For example, if I wanted the pods to be in us-west-2a I would do:

    <http://topology.kubernetes.io/zone|topology.kubernetes.io/zone>: "us-west-2a"```
So then it will only put it in the nodes with that label.