Can't store logs to external Minio server

Platform, Deploy & Infra Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 128Gb / 1 Tb
  • Deployment: Kubernetes
  • Airbyte Version: 0.41.0 (helm: 0.44.5)
  • Source name/version: Snowflake/0.1.31
  • Destination name/version: Postgresql/0.3.26
  • Step: sync
  • Description: Can’t store logs into external Minio server.

Data Sync is okay, but there are exception on Worker logs and Minio server is not receiving any log. Any idea on this? Thanks in advance.

LOG

java.lang.RuntimeException: Cannot publish to S3: Unable to execute HTTP request: Unsupported or unrecognized SSL message
        at com.van.logging.aws.S3PublishHelper.publishFile(S3PublishHelper.java:127)
        at com.van.logging.AbstractFilePublishHelper.end(AbstractFilePublishHelper.java:61)
        at com.van.logging.BufferPublisher.endPublish(BufferPublisher.java:66)
        at com.van.logging.LoggingEventCache.publishEventsFromFile(LoggingEventCache.java:201)
        at com.van.logging.LoggingEventCache.lambda$publishCache$0(LoggingEventCache.java:243)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Unsupported or unrecognized SSL message
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
        at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
        at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6508)
        at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1856)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1816)
        at com.van.logging.aws.S3PublishHelper.publishFile(S3PublishHelper.java:124)
        ... 9 more
Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
        at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:450)
        at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:175)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
        at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
        at com.amazonaws.http.conn.$Proxy46.connect(Unknown Source)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
        ... 23 more
Cannot end publish with com.van.logging.aws.S3PublishHelper@4eb9cb33 due to error: Cannot end publishing: Cannot publish to S3: Unable to execute HTTP request: Unsupported or unrecognized SSL message

Helm yaml file Minio Section

    ## logs.storage.type Determines which log storage  will be utilized.  One of "MINIO", "S3" or "GCS"
    ##                   Used in conjunction with logs.minio.*, logs.s3.* or logs.gcs.*
    storage:
      type: "MINIO"

    ##  logs.minio.enabled Switch to enable or disable the Minio helm chart
    minio:
      enabled: false

    ##  logs.externalMinio.enabled Switch to enable or disable an external Minio instance
    ##  logs.externalMinio.host External Minio Host
    ##  logs.externalMinio.port External Minio Port
    ##  logs.externalMinio.endpoint Fully qualified hostname for s3-compatible storage
    externalMinio:
      enabled: true
      host: minio-svc.minio
      port: 9000

    ##  logs.s3.enabled Switch to enable or disable custom S3 Log location
    ##  logs.s3.bucket Bucket name where logs should be stored
    ##  logs.s3.bucketRegion Region of the bucket (must be empty if using minio)
    s3:
      enabled: false
      bucket: airbyte-dev-logs
      bucketRegion: ""
2

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

3

It was my fault, that I wrongfully put a abundant endpoint under log.externalMinio.

The setting I was wrong was like this.

log:
    externalMinio:
        enabled: true
        endpoint: minio-svc.minio:9000
        host: minio-svc.minio
        port: 9000

And this should be like following.

log:
    externalMinio:
        enabled: true
        host: minio-svc.minio
        port: 9000

Now, I know that you guys put http:// at helm chart, but my wrongful settings made url has no scheme, and the aws client made it https, then it couldn’t talk to my minio server which has no TLS support.

Thanks, though.