Error with S3 storage backend and need for log cleaning sweeper setup

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is experiencing an error related to S3 storage backend reaching its minimum free drive threshold in Airbyte running on EKS. They are looking for a way to set up a sweeper to clean logs periodically.


Question

Hello everyone. I’m getting this error: java.lang.RuntimeException: Cannot publish to S3: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed. (Service: Amazon S3; Status Code: 507; Error Code: XMinioStorageFull; Request ID: 17CD86004FD00990; S3 Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8; Proxy: null)

I don’t have S3 logging configured, and I raised airbyte-minio-pv-claim-airbyte-minio-0 to 2GB. Today, when it reached 677MB, it crashed all my connections. If I clean manually inside /storage/airbyte-storage, it all goes back to work.

I’m running on EKS, and my Airbyte version is 0.59.1.

Is there any way to set up some kind of sweeper to clean the logs periodically?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["error", "s3-storage-backend", "eks", "airbyte-version-0.59.1", "log-cleaning-sweeper"]
2

Today, when it reached 677MB, it crashed all my connections.
I saw the same behaviour. PVC was nowhere near filling up

Is there any way to set up some kind of sweeper to clean the logs periodically?
I considered that though decided against it. Ended up writing the logs to GCS instead, been working fine ever since

3

If you do end up switching to AWS S3/GCP GCS, keep in mind you’ll lose the logs entries from your previous job runs. This was not a problem for us though

4

<@U05B9CPQSCV> Glad to hear that. I did a bit of digging on the channels, and it seems this issue is quite common.

I tried S3 logging following the documentation but had no luck.

Thanks for the feedback, though