Troubleshooting External MinIO Configuration with Airbyte Helm Chart

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is facing issues with directing all S3 requests to an external MinIO instance when configuring it with the latest Airbyte Helm chart. Logs are being stored correctly, but the state storage is not working as expected. The user is seeking guidance on additional configurations or specific settings in the Helm chart to fully support the external MinIO instance.


Question

Hi,

I’m currently trying to configure an external MinIO instance with the latest Airbyte Helm chart but am running into issues with directing all S3 requests to MinIO properly.

Here’s the key configuration I’m using in the Helm chart:

  value: "minio"
- name: global.storage.minio.endpoint
  value: "https://<masked-minio-endpoint>:443"
- name: global.storage.storageSecretName
  value: "airbyte-custom-secret"
- name: global.storage.minio.accessKeyIdSecretKey
  value: "MINIO_ACCESS_KEY"
- name: global.storage.minio.secretAccessKeySecretKey
  value: "MINIO_SECRET_KEY"
- name: global.storage.bucket.log
  value: "airbyte"
- name: global.storage.bucket.state
  value: "airbyte"
- name: global.storage.bucket.workloadOutput
  value: "airbyte"```
It seems that logs are being correctly stored in the external MinIO, but the state storage is not working as expected. Is there any additional configuration I should consider to fully support an external MinIO instance, or am I missing anything specific in the Helm chart?

I would greatly appreciate any guidance or suggestions to troubleshoot this issue.

Thanks in advance!

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1725634682543549) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["external-minio-configuration", "airbyte-helm-chart", "s3-requests", "state-storage", "troubleshooting"]
</sub>
2

an idea :bulb:
you can download charts https://github.com/airbytehq/helm-charts and extract version you have, then look for any minio-related stuff, or something related to “state”

3

That is what I’ve spent major part of the day doing without much luck sadly :disappointed_relieved: