Summary
The user wants to configure node selector in an EKS cluster based on the image tag of custom connector images for arm64 and x64 architectures. They are looking for a way to ensure that the aarch64 image connector runs on aarch64 machines and x64 image connector runs on x64 machines.
Question
Hi Folks, we would like to measure performance between arm64 vs x64 machines in eks. We have built our custom connector images for both x64 as well as arm64 arch and our eks cluster has node groups containing x64 & aarch64 machines. Is there anyway i can configure node selector based on image tag? So if aarch64 image connector is running, it should go to aarch64 otherwise x64 machine …
Should we do this via backend db change? like modifying cpu/resource limits for connections?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["eks-cluster", "node-selector", "custom-connector", "arm64", "x64", "image-tag"]
Based on the information provided in the extracts, it seems that you can assign node selectors to your pods in Kubernetes. This is demonstrated in the Github Issue (closed): Node selector not getting added to source-check as well as destination-check pods where a user shared a method of assigning node selectors to the check* pods using the airbyte helm values.yaml file. Here is the relevant code snippet: yaml global: ... jobs: ... kube: ... -- 1. set node annotations/selectors/tolerations here to set JOB_KUBE_* vars. nodeSelector: workload_type: some_value tolerations: - key: workload_type operator: Equal value: "some_value" effect: NoSchedule worker: ... extraEnv: -- 2. Reference JOB_KUBE_* vars in CHECK_JOB_KUBE_* vars - name: CHECK_JOB_KUBE_SELECTORS value: "$(JOB_KUBE_NODE_SELECTORS)" - name: CHECK_JOB_KUBE_TOLERATIONS value: "$(JOB_KUBE_TOLERATIONS)" You can modify this example to suit your needs. For instance, you can set the nodeSelector to target either x64 or aarch64 machines based on the image tag of your custom connector. As for modifying CPU/resource limits for connections, a user in the Forum Post: Ideas for sync jobs speed up suggested that if the bottleneck is resource-related, you can change the following environment variables to give more resources to the pod running sync: bash JOB_MAIN_CONTAINER_CPU_REQUEST= JOB_MAIN_CONTAINER_CPU_LIMIT= JOB_MAIN_CONTAINER_MEMORY_REQUEST= JOB_MAIN_CONTAINER_MEMORY_LIMIT= Please note that these are just examples and you should adjust them according to your specific requirements.