- Is this your first time deploying Airbyte?: No
- OS Version / Instance: EKS cluster, kubernates Version 1.19
- Deployment: Kubernetes deployment
- Airbyte Version: 0.38.3-alpha
- Description: When installed in Helm chart for the first time, the following log is displayed and minio is fixed in the pending state.
The airbyte is executable, but the connector cannot be connected.
Internal Server Error: Unable to execute HTTP request: Connect to airbyte-minio:9000 [airbyte-minio/10.100.122.189] failed: Connection refused
Warning FailedScheduling 2m3s (x45 over 7h29m) default-scheduler error while running "VolumeBinding" prebind plugin for pod "airbyte-minio-7fcf9845b-d2z4n": Failed to bind volumes: timed out waiting for the condition
5m6s Warning FailedScheduling pod/airbyte-minio-7fcf9845b-d2z4n error while running "VolumeBinding" prebind plugin for pod "airbyte-minio-7fcf9845b-d2z4n": Failed to bind volumes: timed out waiting for the condition
4m9s Normal Provisioning persistentvolumeclaim/airbyte-minio External provisioner is provisioning volume for claim "airbyte/airbyte-minio"
2m49s Normal ExternalProvisioning persistentvolumeclaim/airbyte-minio waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
On the other hand, both pvc and pv were created and connected normally in postgres.
Hey can you share the complete scheduler and server logs here?
The server log is related to minio as follows.
Cannot start publish with com.van.logging.aws.S3PublishHelper@f460af5 due to error: Cannot start publishing: Unable to execute HTTP request: Connect to airbyte-minio:9000 [airbyte-minio/10.100.122.189] failed: Connection refused
Cannot end publish with com.van.logging.aws.S3PublishHelper@f460af5 due to error: Cannot end publishing: Cannot publish to S3: Unable to execute HTTP request: Connect to airbyte-minio:9000 [airbyte-minio/10.100.122.189] failed: Connection refused
I think it’s related to S3 connection, but I installed help with the default setting. Should I set the values related to s3 credential?
Hey airbyte-minio is one more container we launch with all our other services. Can you check if there is a container airbyte-minio?
Of course there is an airbyte-minio pod. However, the point is that this pod is in the pending state
Helm was installed as the default setting. Do I need to set a separate setting to run minio?
I don’t think you have to run any separate settings. Can you describe the pod and see why it’s in pending state?
Can you describe the pod and see why it’s in pending state?
Controlled By: ReplicaSet/airbyte-minio-7fcf9845b
Host Port: 0/TCP
Liveness: http-get http://:minio/minio/health/live delay=5s timeout=5s period=5s #success=1 #failure=5
Readiness: tcp-socket :minio delay=5s timeout=1s period=5s #success=1 #failure=5
MINIO_ACCESS_KEY: <set to the key 'access-key' in secret 'airbyte-minio'> Optional: false
MINIO_SECRET_KEY: <set to the key 'secret-key' in secret 'airbyte-minio'> Optional: false
/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from airbyte-minio-token-8tdgp (ro)
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
Type: Secret (a volume populated by a Secret)
QoS Class: BestEffort
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 7m32s (x173 over 29h) default-scheduler error while running "VolumeBinding" prebind plugin for pod "airbyte-minio-7fcf9845b-d2z4n": Failed to bind volumes: timed out waiting for the condition
I think it failed when mounting the volume.
Got it. If it is possible to create PV first and then deploy minio? Also can you deploy using kustomize yamls? As that is easy to debug
I think it’s not an airbyte problem, it’s a minio problem
The same phenomenon as this link is occurring.
Oh got it. Did it help with the resolution they suggested?
Hey, I solved this problem by proceeding this way.
There is one more question. Is there any way to load the log on s3 without going through minio?
values.yaml, even if the minio enabled item is designated as
false and s3 is changed to
true, minio pod still works.
On Kubernetes (Beta) | Airbyte Documentation you can refer to this. Those env variables (of minio) should be still present and should be empty
I already saw that document.
But that document is a description of kcustomize.
The explanation of helm is for reference to github readme.
values.yaml , even if the minio enabled item is designated as
false and s3 is changed to
true , minio pod still works.
And this is set up by referring to github readme.
Is it possible to share the values.yaml file?
Ok, I changed the value.yaml file as below.
## @section Logs parameters
## @param logs.accessKey.password Logs Access Key
## @param logs.accessKey.existingSecret
## @param logs.accessKey.existingSecretKey
password: accessKey of AWS credential
## @param logs.secretKey.password Logs Secret Key
## @param logs.secretKey.existingSecret
## @param logs.secretKey.existingSecretKey
password: secret key of AWS credential
## @param logs.minio.enabled Switch to enable or disable the Minio helm chart
## @param logs.externalMinio.enabled Switch to enable or disable an external Minio instance
## @param logs.externalMinio.host External Minio Host
## @param logs.externalMinio.port External Minio Port
## @param logs.s3.enabled Switch to enable or disable custom S3 Log location
## @param logs.s3.bucket Bucket name where logs should be stored
## @param logs.s3.bucketRegion Region of the bucket (must be empty if using minio)
bucketRegion: "bucket Region"
Right now you are saying logs are written to s3 and also minio pods are up?
Yes, exactly. Is this a normal case?
I don’t think this is normal. Can you check in Charts.yaml the condition is minio.enabled if so can you change it to logs.minio.enabled?
Minio does not appear when installing with ‘helm’ from this version(0.39.17).
I don’t know why yet.