S3 Minio Storage Full

  • Is this your first time deploying Airbyte?: No
  • Deployment: Kubernetes
  • Airbyte Version: * 0.39.32-alpha
  • Source name/version: Jira - 0.2.20
  • Destination name/version: Snowflake - 0.4.31
  • Step: During sync and resetting data
  • Description: On sync or resetting data, we are seeing errors within 20s saying
Last attempt:
NaN Bytes | no records | no records | 20s | Reset
Failure Origin: replication, Message: Something went wrong during replication
1:35PM 07/13
3 attempts
2022-07-13 17:36:52 - Additional Failure Information: message='io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.lang.RuntimeException', nonRetryable=false
Attempt 1
Attempt 2
Attempt 3
NaN Bytes | no records | no records | 20s
Failure Origin: replication, Message: Something went wrong during replication
/tmp/workspace/62/0/logs.log

Also no logs are available

Update found logs:

Cannot end publish with com.van.logging.aws.S3PublishHelper@6e9060a4 due t │
│ o error: Cannot end publishing: Cannot publish to S3: Storage backend has  │
│ reached its minimum free disk threshold. Please delete a few objects to pr │
│ oceed. (Service: Amazon S3; Status Code: 507; Error Code: XMinioStorageFul │
│ l; Request ID: 17016D70DC7D0A8C; S3 Extended Request ID: 4faaaaf4-1f8b-448 │
│ 6-867c-89a2f2720a10; Proxy: null)

How does one empty the Minio storage?

I would like to add that I ran this successfully yesterday with no changes to the configurations

Update: I sshd into the minio pods and deleted all logs under the storage path. That worked for me