Source Bigquery - alternate project access

Connector Development
,
1

Hi community!
Please advise on the following:
We are using Bigquery as a source with service account json.
When selecting source tables we can see only datasets from project where service account is created.
Is it possible to select objects from other Bigquery Projects, to which service account was granted access?

3

Hey @LMRU-Ruslan-A,

Thanks for the question and welcome to the community.

What roles does the service account have?

4

Hi @sajarin !
In the “guest” project this service account has role “Bigquery Data Viewer”

5

Hey @LMRU-Ruslan-A,

Thanks for the response. Did you make sure to follow the instructions in our docs for setting up the permissions for the service account? https://docs.airbyte.com/integrations/sources/bigquery#:~:text=A%20Google%20Cloud%20Service%20Account%20with%20the%20"BigQuery%20User"%20and%20"BigQuery%20Data%20Editor"%20roles%20in%20your%20GCP%20project

I think you need to add more roles/permissions. Let me know if this works.

6

Hi @sajarin! Thank you for links provided, I read the documentation, and I am confused - if I use bigquery only as a source, why do I need to have data editor privileges?
The owners of the project which data we do use will not grant us such privileges.

7

Hey @LMRU-Ruslan-A,

Thanks for the reply. Could I ask what your config looks like? When you setup your connector, did you specify a default dataset id? Which project id did you specify for the connector? Does specifying another project id for a project that your service account has been granted access allow you to view the other datasets?

8

Hi @sajarin !

In interface I specify the projectId, the default Dataset ID and credentials JSON which I want to connect but when I press “Set up source” - the message says:
The connection tests failed. Could not connect with provided configuration. Error: Access Denied: Project YYY: User does not have bigquery.jobs.create permission in project YYY.

Though

  1. Actually service account is granted the requreied privilegy in the target dataset
  2. Why this permission is needed, if we set up source not target?
9

Hi @sajarin!
Are there any news about this issue?

10

Hello LMRU-Ruslan-A, it’s been a while without an update from us. Are you still having problems or did you find a solution?

11

Hi @marcosmarxm !
Yes, this issue is challenging and important for us.

12

Hey @LMRU-Ruslan-A,

This error is occurring because the service account being used to authenticate the connection to BigQuery does not have the necessary permissions to create jobs in the project. In order to create a job, the service account must have the bigquery.jobs.create permission in the project.

It’s worth noting that when you set up a source in Airbyte, the source is responsible for creating jobs in BigQuery to perform tasks such as reading data from the dataset or writing data to the dataset. This is why the service account needs the bigquery.jobs.create permission when setting up the source.

To fix this issue, you will need to ensure that the service account being used to authenticate the connection to BigQuery has the bigquery.jobs.create permission in the specified project. You can grant this permission to the service account using the Google Cloud Console or by using the gcloud command-line tool.

I hope this helps. Let me know if you have any additional questions or if you need further assistance.