Airbyte requesting service account permissions at project level instead of bucket and dataset levels

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Airbyte is requesting service account permissions at the project level instead of the GCS bucket and BigQuery dataset levels, causing a security risk. Is there a solution to this problem?


Question

Hi,

We are facing an issue with Airbyte. We are using GCS as the source and BigQuery as the destination. However, Airbyte is requesting service account permissions at the project level, not at the GCS bucket and BigQuery dataset levels.

We have tried giving event admin access only to the buckets, but it still requires list access at the project level, which poses a security risk for us. Is there any solution to this problem?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte", "gcs-connector", "bigquery-connector", "service-account-permissions", "security-risk"]
2

<@U06TKQ571GU> The only permission that I’m aware of that needs to be granted at the project level is BigQuery Jobs User, since this permission cannot be granted at the dataset or table level (<BigQuery IAM roles and permissions  |  Google Cloud noted in the docs>). Basically, Jobs aren’t connected to datasets or tables, so they can’t be granted at that level. (Jobs also control where they query usage is billed to, so must be at project level)

In our case, we grant the Jobs.* permissions at the project level, but no other access . . . and then we grant Airbyte read/write access to the specific datasets.

However, as a best practice, I would also recommend creating Airbyte in its own project (e.g. orgname-airbyte). This way project-level permissions only apply to Airbyte’s usage, and it makes it easier to isolate billing and auditing data.

We do both of these in our security posture and it works quite well (although you could just also grant BigQuery Data Owner to the project at that point without really negatively affecting security posture). Then you just grant your other project’s user(s) or service account(s) read-only access to your Airbyte project’s data (either at project, dataset, or table level)—likely using the BigQuery Data Viewer pre-defined role. Those don’t need any Jobs roles, because they’ll run the jobs in their own project—the same query can reference BigQuery tables in multiple projects just fine as long as the permissions are set right.

In addition to the benefits to security posture, this also means a clean separation in billing reporting—in our case we know exactly our total costs or Airbyte for ETL (which we run in GKE on that same project), dbt for modeling (in its own project, dashboards for viz (which has reporting tables in its own project), etc. right in the billing viewer without having to query the log exports.