Managing log rotation in Airbyte worker

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is facing issues with excessive log generation in the Airbyte worker container, leading to rapid disk filling. They have attempted to control log rotation using TEMPORAL_HISTORY_RETENTION_IN_DAYS environment variable but the problem persists.


Question

Hi guys, How can we manage log rotation in the Airbyte worker? Airbyte worker is generating an excessive amount of logs in the container, leading to rapid filling of the hard disk within a few days. How can we control log rotation to mitigate this issue? I tried to add TEMPORAL_HISTORY_RETENTION_IN_DAYS=1 , but still the logs are generating too much in the worker container.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["log-rotation", "airbyte-worker", "container", "disk-filling", "temporal-history-retention-in-days"]
2

If you’re hosting in the cloud, I’d highly recommend moving the logs out of system (using either S3 if you’re on AWS or GCS if you’re on GCP). With this setup, you can disable minio completely and remove one more moving part as well.