Error upgrading Airbyte version on GKE cluster

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Error message ‘Cannot end publish with com.van.logging.aws.S3PublishHelper’ after upgrading Airbyte to version 0.57.3 on GKE cluster. Issue occurs with one connection while others using the same destination work fine.


Question

Hello everybody,
I’m new to Airbyte and we’ve upgraded our Airbye to 0.57.3 from .0.48.0 Airbyte running on GKE cluster
after the upgrade we’re facing new issue.
ERROR c.v.l.BufferPublisher(endPublish):69 - Cannot end publish with com.van.logging.aws.S3PublishHelper@7747df10 due to error
This happens only with one connection there are other connections that are using same destination that are working just fine.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["error", "upgrade", "airbyte-platform", "GKE-cluster", "connection-issue"]
2

Several of the storage/logging configurations have changed between those versions. Here’s an example of what I’m using for GKE, which took some fiddling. These are overrides from values YAML for a Helm-based deployment (obviously they’d need to be combined with your other overrides):

  storage:
    type: GCS
    gcs:
      bucket: &storage-bucket <your-bucket-name>
      credentials: &log-creds-path /secrets/gcs-log-creds/gcp.json
      credentialsJson: &sa-creds-json <your-service-account-key-as-base64-encoded-string>
    bucket:
      activityPayload: *storage-bucket
      log: *storage-bucket
      state: *storage-bucket
      workloadOutput: *storage-bucket

  state:
    storage:
      type: GCS

  logs:
    minio:
      enabled: false
    storage:
      type: GCS
    gcs:
      bucket: *storage-bucket
      credentials: *log-creds-path
      credentialsJson: *sa-creds-json

worker:
  extraEnv:
    - name: STATE_STORAGE_GCS_BUCKET_NAME
      value: *storage-bucket
    - name: STATE_STORAGE_GCS_APPLICATION_CREDENTIALS
      value: *log-creds-path
    - name: CONTAINER_ORCHESTRATOR_SECRET_NAME
      value: airbyte-gcs-log-creds
    - name: CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH
      value: /secrets/gcs-log-creds

minio:
  enabled: false```
It's worth noting that some of these don't exactly match the docs, which are a mix of the old and new ways, but some of the charts seem to need this. YMMV