Summary
In a Kubernetes environment, the user is running Airbyte with a custom rest source and Kafka destination connector, scheduled to run every two minutes. They are observing that a new pod is created for the source, destination, and cron job every time the job runs. The user is looking for a way to configure the connector to run the job with a single pod to reduce resource consumption.
Question
Hi,
We are running airbyte in Kubernetes environment. We are using a connector with custom rest source and kafka destination and scheduled a cron job to run every two minutes. I have observed that, whenever the cron job runs it’s creating a pod for source, destination and cronjob every time. Is there a way we can configure connector to run the job with single pod instead of creating the pod in every run to avoid resource consumption?
Please suggest.
Thanks.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.