Summary
Inquiring about the Postgresql version requirement for migrating Airbyte OSS to Kubernetes and seeking suggestions/recommendations for the migration process.
Question
With Airbyte OSS, we currently are running it using docker on EC2 with an external Postgresql db v14.6, and are looking at migrating to Kubernetes. In the documentation there is a callout to use Postgresql v13; is that still a requirement? Anyone have any suggestions or recommendations to consider in this migration?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["airbyte-oss", "postgresql", "kubernetes", "migration", "postgresql-version"]
You can use any version of postgres 13 or later
Hah, when I asked about this recently people seemed to not really know why that requirement is listed or whether later versions are actually safe so didn’t want to go against the docs. Nice to hear someone else running a newer version.
<@U064R6L3ZBJ> If you get a chance, it might be worth putting a docs issue in on GitHub noting that everything actually does work in modern versions.
I get the sense that the docs lag behind but yeah I will do that
I feel like in open source projects like this, it’s common to find the docs are out of date
Yeah, I think it’s also a good reminder when documenting dependency requirements to document the REASON for them. If v13 is required because it added a functionality, it’s safe to assume later versions will also work. If it’s because later versions handle something incorrectly, then that should be noted as well along with links to issues or whatever else so people know when it’s been resolved and can update accordingly.
Once that knowledge is lost, it’s really hard to get back to, and then everyone is afraid to ignore the advice, even if it isn’t relevant anymore.
Have you migrated from docker to kubernetes? Or did you start with kubernetes initially?
We started with Kubernetes
We migrated from Docker to Kubernetes a couple years ago, and just did a move to a new K8s cluster from that one. We’re on GCP though, so we’re talking GCE and GKS.
Generally it’s pretty smooth to move between them, once you get past the complexities related to fine-tuning the Helm values and learning curve of the K8s stuff if you haven’t dealt with it before. You also have the advantage that Airbyte Cloud is deployed to AWS, so their documentation for AWS/EKS tends to be far better than their docs for GCP/GKE (which I’m going to start trying to improve through contributions).
Adding some thoughts from my experience. Postgres v13 was required (at least to new installation) because there was a Temporal dependency query it would only work in Postgres v13. I didn’t try to check if upgrade the database after having a running an instance cause some issues, I don’t think… because it won’t run the creation of Temporal database again.
I’ll give a try again in a fresh install and if it’s possible now to use more recent version I’ll update the docs.
<@U02TNBMDUMQ> as this can be something important to update in docs 
We launched airbyte initially on postgres 15 with no issue
then upgraded postgres to 16
I just succeed deploying. I can confirm Airbyte v0.63.1, Helm Chart v0.199.0 works well with a clean Postgres 16.3 
I put more details at https://github.com/airbytehq/airbyte/pull/39510#issuecomment-2177188829