Based on the information provided in the extracts, it seems that you can assign node selectors to your pods in Kubernetes. This is demonstrated in the Github Issue (closed): Node selector not getting added to source-check as well as destination-check pods where a user shared a method of assigning node selectors to the check*
pods using the airbyte helm values.yaml
file. Here is the relevant code snippet: yaml global: ... jobs: ... kube: ... -- 1. set node annotations/selectors/tolerations here to set JOB_KUBE_* vars. nodeSelector: workload_type: some_value tolerations: - key: workload_type operator: Equal value: "some_value" effect: NoSchedule worker: ... extraEnv: -- 2. Reference JOB_KUBE_* vars in CHECK_JOB_KUBE_* vars - name: CHECK_JOB_KUBE_SELECTORS value: "$(JOB_KUBE_NODE_SELECTORS)" - name: CHECK_JOB_KUBE_TOLERATIONS value: "$(JOB_KUBE_TOLERATIONS)"
You can modify this example to suit your needs. For instance, you can set the nodeSelector
to target either x64
or aarch64
machines based on the image tag of your custom connector. As for modifying CPU/resource limits for connections, a user in the Forum Post: Ideas for sync jobs speed up suggested that if the bottleneck is resource-related, you can change the following environment variables to give more resources to the pod running sync: bash JOB_MAIN_CONTAINER_CPU_REQUEST= JOB_MAIN_CONTAINER_CPU_LIMIT= JOB_MAIN_CONTAINER_MEMORY_REQUEST= JOB_MAIN_CONTAINER_MEMORY_LIMIT=
Please note that these are just examples and you should adjust them according to your specific requirements.