Setting Up CloudSQLProxy with Airbyte Deployment

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is facing issues with deploying Airbyte and connecting to an external DB on GCP, specifically about the first deployment using abctl while needing a dedicated CloudSQLProxy pod. The connection attempts to the database are failing, leading to timeout errors, and there is a concern about the dependency between the CloudSQLProxy pod and Airbyte’s deployment.


Question

hello :wave:
to properly use external DB on GCP, we need a dedicated pod for the CloudSQLProxy

how could we manage the “first” deployment of Airbyte using abctl ?
abctl local install --values=values.yaml --secret=secrets.yaml --chart-version=1.1.1
because we have the below error message:

ERROR   airbyte-bootloader: org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-1)
ERROR   airbyte-bootloader: Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30002ms (total=0, active=0, idle=0, waiting=0)
ERROR   airbyte-bootloader: Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
ERROR   airbyte-bootloader: Caused by: java.net.UnknownHostException: cloud-sql-proxy-service```

As the `CloudSQLProxy` pod needs to be deployed using the `airbyte` namespace and `kubeconfig` file, created by the `abctl local install`...
```kubectl apply -f cloud-sql-proxy.yaml -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig```
...but Airbyte deployments need to reach the external DB via the `CloudSQLProxy` pod which must exist :thinking_face:

so we face the chicken or the egg problem :face_with_head_bandage:

Thanks a lot ! :pray:

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1732884788599739) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['cloudsqlproxy', 'gcp', 'abctl', 'deployment', 'error', 'database-connection']
</sub>
2

Have you explored an idea mentioned in this thread? https://airbytehq.slack.com/archives/C021JANJ6TY/p1732037599060799?thread_ts=1732033531.453399&amp;cid=C021JANJ6TY

3

Another idea is that in values.yaml you set extraInitContainers for airbyte-bootloader
It could wait for CloudSQLProxy to be ready.
You start deployment with abctl and when kind cluster is created, you can deploy your CloudSQLProxy

4

thanks ! will look into it :slightly_smiling_face:

5

you can also combine both ideas in one helm chart + values.yaml — extraInitContainers waiting for CloudSQLProxy and deployment with your CloudSQLProxy

6

why not, but I prefer having a dedicated pod for the CloudSQLProxy , not a container on another pod

7

That’s not what I had in mind. Init container for bootloader should contain only script that checks if CloudSQLProxy is deployed and listens on port

8

that way looks perfect, <@U06CW2SDKCN> you’re our man :pray: :angel:

9

I run like this as well
but on previous chart installations the bootloader didnt like sidecar idea
so I have a pod that only runs for the bootloader
and the other pods connect via localhost/cloudsqlproxy sidecar

10

thanks, sidecar is GCP recommendation for Kubernetes:
• https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine
• https://cloud.google.com/sql/docs/postgres/connect-auth-proxy#overview
<@U06CW2SDKCN> FYI more inputs :fire:

11

you’re welcome :slightly_smiling_face:
just ask if you need any pointers
we also run with workload identity (so, no secrets/database credentials on the cluster)

12

wonderful, thank you :blush: