Unable to upgrade to v0.39.4-alpha or later via Helm chart

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Amazon Linux 2 (worker/job containers run on AWS Fargate node, the rest run on EKS managed EC2 nodes)
  • Memory / Disk: 2vCPU 8GB memory per node
  • Deployment: Kubernetes
  • Airbyte Version: v0.39.3-alpha
  • Description:

I’m currently running v0.39.3-alpha and I’m having trouble upgrading to any later version via the Helm chart (I think this PR deployed in v0.39.4-alpha might be the culprit).

When I run the Helm upgrade my Postgres container immediately shuts down and the bootloader container then fails due to a database connection timeout. I’m not sure if the replacement Postgres container is just taking too long to start or if there’s a deeper issue. Is there a way to raise the bootloaders DB connection timeout with a Helm deployment?

It appears that something is preventing the Postgres container from starting.

I’m seeing this error on the Postgres stateful set.

create Pod airbyte-postgresql-0 in StatefulSet airbyte-postgresql failed error: The POST operation against Pod could not be completed at this time, please try again.

I was able to resolve this by scaling the airbyte-postgresql StatefulSet down to zero pods before running the Helm upgrade.

Thank you @damiantw for sharing how you solved this.
Is it your first Airbyte upgrade on Kubernetes or doyou think that it’s specific to the changes made in v.0.39.4?