I have Airbyte deployed based on the Docker guidance. After a while the workspace started to use up too much space that was causing issues. In another other post I’ve come across the MINIMUM_WORKSPACE_RETENTION_DAYS, MAXIMUM_WORKSPACE_RETENTION_DAYS, and MAXIMUM_WORKSPACE_SIZE_MB env vars that seem to configure the retention and seems like the thing I’m looking for. It’s not clear, however how to get those configurations applied so any cleanup happens?
From looking around GitHub issues and so on, I’ve seen that these variables might be used by the airbyte-scheduler, but that’s not used in the default Docker Compose-based deployment. Just applying these values to the airbyte-server and airbyte-worker doesn’t seem to make any difference.
Have any guidance on this?
Is this your first time deploying Airbyte?: Yes
OS Version / Instance: Debian
Memory / Disk: you can use something like 4Gb / 8GB
Looks this feature was removed with the Scheduler. A ticket was created in Github https://github.com/airbytehq/airbyte/issues/15567 any updates I’ll return to you. Today as a workaround you need to clean the logs yourself
Thanks, that’s excellent info! Will keep an eye on that issue. In the meantime I’ve done exactly that (a systemd timer and a cleanup script, but it’s a bit too hacky).
Could we package it as a script to manually clean up as a workaround for now?
Hello updating the status of this issue.
The solution was merged but isn’t published to latest version, probably version: v0.40.5 will have those modifications.
