S3 Storage Full Error and Cleanup Automation

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User encounters a storage full error on S3 due to reaching the minimum free drive threshold, causing connection crashes. They seek a solution for automated log cleanup.


Question

Hello everyone. I’m getting this error: java.lang.RuntimeException: Cannot publish to S3: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed. (Service: Amazon S3; Status Code: 507; Error Code: XMinioStorageFull; Request ID: 17CD86004FD00990; S3 Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8; Proxy: null)

I don’t have S3 logging configured, and I raised airbyte-minio-pv-claim-airbyte-minio-0 to 2GB. Today, when it reached 677MB, it crashed all my connections. If I clean manually inside /storage/airbyte-storage, it all goes back to work.

I’m running on EKS, and my Airbyte version is 0.59.1.

Is there any way to set up some kind of sweeper to clean the logs periodically?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['s3-storage', 'airbyte-minio', 'eks', 'log-cleanup', 'error-507']