Issue Connecting to Postgres Database on Kubernetes

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is experiencing connectivity issues with a Postgres database from an Airbyte OSS instance deployed on Kubernetes, despite successful local connections using psql. The user has verified that the Airbyte Network Services and pod IPs are correctly configured within the allowed IPv4 range.


Question

Hello :slightly_smiling_face:

I’m struggling to connect to a Postgres database using Airbyte OSS deployed on Kubernetes.

We have a GCP cluster with a set IPv4 range. The database is managed via AWS, and the same IP range was whitelisted.

Our Airbyte instance is running well, and the credentials for the Postgre db work (I can use psql from my computer to connect).

However, trying to connect with Airbyte outputs this error:

Our Airbyte Network Services seems OK to me, all the pods IP are in the range.

Has someone encountered this issue? How did you resolve it?

In advance, thank you :smile:


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['postgres-connection', 'airbyte-oss', 'kubernetes', 'gcp', 'aws', 'network-issues']
2

What kind of connectivity is between GCP cluster and Postgres database on AWS? Are they using public IP addresses? VPN or something else?

3

Hello <@U05JENRCF7C> :slightly_smiling_face:

There’s no peering or VPN. We have a public address, which is filtered on IP or IP range.

4

In cases like this I would start a pod in kubernetes cluster like this:
kubectl run -it postgres --image=postgres:latest --restart=Never --rm -- /bin/bash
and I would try to connect with psql to that database. Just to be sure that there is a connectivity between kubernetes cluster and PostgreSQL database

5

Hello <@U05JENRCF7C>, thank you for your advice :slightly_smiling_face:

6

I checked with our infra team, the issue is with the worker’s IP. Despite the screenshot, the worker’s IP are not in the range set in the cluster.

7

Do you know how to configure the workers so that their IP stays the same, and can be whitelisted?

8

You need to talk with your infra team, because these are networking topics. Maybe your company has some security or networking practices, and some solution are better than the others.

One approach is that outbound traffic from kubernetes is treated as the traffic from single public IP address.
https://cloud.google.com/nat/docs/overview
https://cloud.google.com/compute/docs/ip-addresses/configure-static-external-ip-address
I found something related to pods and persistent IP addresses
https://cloud.google.com/kubernetes-engine/docs/concepts/about-persistent-ip-addresses-for-gke-pods

You need to find solution that works for you. I’m not able to help your more on that, because I’m more into AWS than GCP, but high-level concepts are quite similar.

9

<@U05JENRCF7C> Thank you very much for the links! Will check that out :wink:

10

<@U05JENRCF7C> Here’s what I did so far:
• Reserved a public external IP on GCP
• Created a loadbalancer on Kubernetes
The lb looks like this:

kind: Service  
metadata:  
  name: airbyte-worker-service  
  namespace: airbyte  
spec:  
  type: LoadBalancer  
  loadBalancerIP: &lt;MY_IP_ADDRESS&gt;  
  selector:  
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: airbyte  
    airbyte: job-pod  
  ports:  
    - name: port-9001  
      protocol: TCP  
      port: 9001  
      targetPort: 9001  
...
    - name: port-9040  
      protocol: TCP  
      port: 9040  
      targetPort: 9040```
Seeing this paragraph of the documentation:

```TEMPORAL_WORKER_PORTS - Defines the local ports the Airbyte Worker pod uses to connect to the various Job pods. Port 9001 - 9040 are exposed by default in the Helm Chart.```
I listed the whole range port.

However, it's not working. If I create a pod with the same label as my loadbalancer and curl <https://ipinfo.io/ip>, I do not have the IP address I set in my loadbalancer.

I'm new to network topics, is there something I'm missing?