Unsure how to set workspace retention variables correctly in docker based deployment


I have Airbyte deployed based on the Docker guidance. After a while the workspace started to use up too much space that was causing issues. In another other post I’ve come across the MINIMUM_WORKSPACE_RETENTION_DAYS, MAXIMUM_WORKSPACE_RETENTION_DAYS, and MAXIMUM_WORKSPACE_SIZE_MB env vars that seem to configure the retention and seems like the thing I’m looking for. It’s not clear, however how to get those configurations applied so any cleanup happens?

From looking around GitHub issues and so on, I’ve seen that these variables might be used by the airbyte-scheduler, but that’s not used in the default Docker Compose-based deployment. Just applying these values to the airbyte-server and airbyte-worker doesn’t seem to make any difference.

Have any guidance on this?

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Debian
  • Memory / Disk: you can use something like 4Gb / 8GB
  • Deployment: Docker
  • Airbyte Version: 0.39.21-alpha

Those env variables aren’t available in latest .env file: https://github.com/airbytehq/airbyte/blob/master/.env
I think the responsibility now should be attributed to the server or worker but let me confirm it.

Thanks for looking into it! I’ve been checking the code as well a bit, but so far none the wiser, so any input is appreciated!

Looks this feature was removed with the Scheduler. A ticket was created in Github https://github.com/airbytehq/airbyte/issues/15567 any updates I’ll return to you. Today as a workaround you need to clean the logs yourself

Thanks, that’s excellent info! Will keep an eye on that issue. In the meantime I’ve done exactly that (a systemd timer and a cleanup script, but it’s a bit too hacky).

Could we package it as a script to manually clean up as a workaround for now?

Really appreciate this suggestion in the ticket!

Today there isn’t an easy script to run this Gergely. We need to wait a complete solution for now :frowning:

Hello updating the status of this issue.
The solution was merged but isn’t published to latest version, probably version: v0.40.5 will have those modifications.

You can check the discussion here: https://github.com/airbytehq/airbyte/pull/16247

Airbyte version 0.45.5 reimplemented the feature, you can change the default value changing the variable TEMPORAL_HISTORY_RETENTION_IN_DAYS

Thanks, will give that a try!

Just for reference, as I wasn’t sure reading your comment, I see that the current default value is 30 days, that’s good to know.

Did you update to latest version of Airbyte?

Please update to version 0.40.26 version see PR https://github.com/airbytehq/airbyte/pull/20317