Controlling Google Secrets storage region in Airbyte

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

User is looking to control the region where Airbyte stores Google Secrets due to organizational constraints.


Question

Hi folks, I need to control the region where Airbyte stores Google Secrets because of organizational constraints. Has anyone been able to?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["control", "google-secrets", "storage-region", "organizational-constraints", "airbyte-platform"]
2

Are you talking about Kubernetes secrets, or are you using Google Secret Manager?

3

<@U035912NS77> secret manager!

4

And is this for Cloud or OSS?

5

currently inspecting this to find clues: https://github.com/airbytehq/airbyte/pull/6113/files

6

yeah, looks like it’s just set to use the automatic option (see: airbyte-config/persistence/src/main/java/io/airbyte/config/persistence/GoogleSecretsManager.java:120

You could modify this and pass the appropriate region list based on the Google SDK values.

But might be a good feature request to be able to set since data governance is a thing

7

I believe that auto setting does follow account level region restrictions, so another option might be locking that down at the GCP level and then making sure it applies here

8

well, this is sad :disappointed: where exactly is that mentioned in the docs?

9

Really odd that Google doesn’t make that obey account-level data residency settings by default

10

<@U035912NS77> thanks again for the help. I’m not used to Java so tweaking the platform code will quickly become a mess…

11

if you raise the issue in GitHub, someone else may be willing to implement it. I can’t imagine you’re the only one with data residency needs.

I suppose another option is to not use Secret Manager, since then the secrets location would be based on the regions of your GCE/GKE deployment and the database server region (which I would assume are already within your data residency requirements)

12

yeah, I’ll create an issue as soon as I find some time. I’ll probably move away from Google Secrets till then, which is far from ideal, but not the end of the world since I’m already on a secure internal network and have SSL turned on for the application database connections

13

yep. especially when everything is within a VPC I wouldn’t be too worried, but it is nice to be able to use Secret Manager

14

<@U066U84SC0P> Something I just noticed while look at another issue is that the YAML generated for AWS Secret Manager actually already has region-based configuration, so there’s already precedent for doing this with Google Secret Manager:
https://github.com/airbytehq/airbyte-platform/blob/d5ec39e2c836572c7ff84499a0a97c110bbc7d7c/charts/airbyte-api-server/templates/deployment.yaml#L127

15

oh, actually, in their docs it says that automatic only works “if resource creation in global is allowed.”

16

Here:
https://cloud.google.com/secret-manager/docs/choosing-replication#:~:text=For%20purposes%20of%20resource%20location%20organization%20policy%20evaluation%2C%20a%20secret%20with%20an%20automatic%20replication%20policy%20can%20only%20be%20created%20if%20resource%20creation%20in%20global%20is%20allowed.

so that just means you’ll need to modify the code to lock it down to a hard-coded list or environment variable with the options, and then maybe put a feature request or MR in for it along with the corresponding documentation

17

truly appreciate the attention <@U035912NS77>!