I am trying to run a simple query on Google BigQuery via a python script, but am getting the below error that my service account is missing bigquery.jobs.create permission.
My service Account has the following roles applied:
Owner
BigQuery Admin
BigQuery Job User
I've also tried creating a custom role with bigquery.jobs.create and applying that to the service account, but still consistently get this error. What am I doing wrong?
from google.cloud import bigquery
from google.oauth2 import service_account
project_id = "my-test-project"
credentials = service_account.Credentials.from_service_account_file("credentials.json")
client = bigquery.Client(
credentials=credentials,
project=project_id
)
print(client.project) # returns "my-test-project"
query = client.query("select 1 as test;")
Access Denied: Project my-test-project: The user my-service-account #
my-test-project. iam.gserviceaccount.com does not have
bigquery.jobs.create permission in project my-test-project.
Authenticating the client using client = bigquery.Client.from_service_account_json("credentials.json") is the preferred method to avoid "Access Denied" errors. For one reason or another (I'm not sure why since bigquery does use oauth 2.0 access tokens to authorize requests), setting credentials through google.oauth2.service_account can lead to permission issues.
Related
I am trying to build an Airflow DAG (on Cloud Composer) that reads emails from Gmail, using the Google API Python client.
I would like to avoid the use of JSON files for Service Accounts, and therefore I am trying to take advantage of Workload Identity. Therefore, I performed the following steps:
Created a Service Account (my-service-account#my-project.iam.gserviceaccount.com) that will then be used to impersonate the Google mail my-email#my-domain.com
Granted Cloud Composer Service account the roles/iam.serviceAccountTokenCreator to the Google mail Service Account
Delegated domain-wide authority to the service account with the scopes 'https://www.googleapis.com/auth/gmail.readonly' such that the service account my-service-account#my-project.iam.gserviceaccount.com is authorized to access the emails of my-email#my-domain.com.
Now I'm trying to use the Google API Python client, in order to instantiate a Gmail service and use it to search the inbox of my-email#my-domain.com. Here's the code:
import google.auth
import google.auth.impersonated_credentials
SERVICE_ACCOUNT = 'my-service-account#my-project.iam.gserviceaccount.com'
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
credentials, project_id = google.auth.default()
logging.info(f'Obtained application default credentials for project {project_id}.')
impersonated_credentials = google.auth.impersonated_credentials.Credentials(
source_credentials=credentials,
target_principal=SERVICE_ACCOUNT,
target_scopes=SCOPES,
)
logging.info(f'Obtained impersonated credentials for {SERVICE_ACCOUNT}')
service = build(
serviceName='gmail',
version='v1',
credentials=impersonated_credentials,
cache_discovery=False,
)
So initially, the code infers the Application Default Credentials (Cloud Composer), and then impersonates Cloud composer to act like the my-service-account#my-project.iam.gserviceaccount.com Service Account). Finally, it uses the returned credentials to build the gmail service.
When attempting to run a query:
results = service.users().messages().list(userId='me', q='from: someEmail#outlook.com').execute()
I get the following error:
[2022-11-14, 18:23:47 UTC] {standard_task_runner.py:93} ERROR - Failed to execute job 604219 for task test (<HttpError 400 when requesting https://gmail.googleapis.com/gmail/v1/users/me/messages?q=from%3A+someEmail%40outlook.com&alt=json returned "Precondition check failed.". Details: "Precondition check failed.">; 30352)
Any clue what I might be missing here? I've found a few similar questions but apparently they all use Service Account JSON files, which is clearly not the case here.
I am trying to migrate my simple python script that accesses an O365 email account using basic auth to modern auth:
Here's the current setup.
import exchangelib as exch
credentials = exch.Credentials('my_username', 'my_password')
configInfo = exch.Configuration(server='smtp.office365.com', credentials=credentials)
tz = exch.EWSTimeZone.localzone()
account = exch.Account(primary_smtp_address='my_email_address', config=configInfo, autodiscover=False, access_type=exch.DELEGATE)
It works fine. From here I can access items such as account.inbox to iterate emails, etc.
Moving to Azure Identity, my devops team assigned me the following:
a registered app in the Azure Portal
an app ID
an object ID
a tenant ID
a secret value ID
a secret key ID
an https://ps.outlook.com/ URL
I've run this...
pip install azure-identity
And now I can run this...
from azure.identity import DefaultAzureCredential
# Now what? A service client perhaps?
I'm at a loss as to what comes next. My goal is to authenticate using the IDs above, then create an account object as before, and continue processing. Can anyone help? Thank you.
I am trying to use BigQuery inside python to query a table that is generated via a sheet:
from google.cloud import bigquery
# Prepare connexion and query
bigquery_client = bigquery.Client(project="my_project")
query = """
select * from `table-from-sheets`
"""
df = bigquery_client.query(query).to_dataframe()
I can usually do queries to BigQuery tables, but now I am getting the following error:
Forbidden: 403 Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.
What do I need to do to access drive from python?
Is there another way around?
You are missing the scopes for the credentials. I'm pasting the code snippet from the official documentation.
In addition, do not forget to give at least VIEWER access to the Service Account in the Google sheet.
from google.cloud import bigquery
import google.auth
# Create credentials with Drive & BigQuery API scopes.
# Both APIs must be enabled for your project before running this code.
credentials, project = google.auth.default(
scopes=[
"https://www.googleapis.com/auth/drive",
"https://www.googleapis.com/auth/bigquery",
]
)
# Construct a BigQuery client object.
client = bigquery.Client(credentials=credentials, project=project)
O want to write a python script that upload data from a file to a bigquery table.
here is the code:
from google.cloud import bigquery
client = bigquery.Client(project=project_id, location='US').from_service_account_json('my-key.json')
dataset_ref = client.dataset(dataset_id)
table_ref = dataset_ref.table(table_name)
client.load_table_from_file(filename, table_ref)
I am using a gcp vm created in the same project as where my bigquery table is. I am also using a service account that has Bigquery admin role.
I have an error that says that the user doesn't have the bigquery.jobs.create permission.
I don't know if it is a useful information but i am able to read my table.
I don't know what to do.
Thanks for your help.
(There are a lot of similar threads here but unfortunately I couldn't find the answer to my error anywhere here or on Goolge)
I'm trying to query a federated table in BigQuery which is pointing to a spreadsheet in Drive.
I've run the following command to create default application credentials for gcloud:
$ gcloud auth application-default login
But this doesn't include Drive into the scope so I'm getting the following error message (which makes sense): Forbidden: 403 Access Denied: BigQuery BigQuery: No OAuth token with Google Drive scope was found.
Then I've tried to auth with explicit Drive scope:
$ gcloud auth application-default login --scopes=https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/bigquery
After that I'm getting the following error when I try to use bigquery python api:
"Forbidden: 403 Access Denied: BigQuery BigQuery: Access Not Configured. Drive API has not been used in project 764086051850 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/drive.googleapis.com/overview?project=764086051850 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry."
The project number above does not exist in our organisation and the provided link leads to a page which says:
The API "drive.googleapis.com" doesn't exist or you don't have permission to access it
Drive API is definitely enabled for the default project, so the error message doesn't make much sense. I can also query the table from the terminal using bq query_string command.
I'm currently out of ideas on how to debug this further, anyone suggestions?
Configuration:
Google Cloud SDK 187.0.0
Python 2.7
google-cloud 0.27.0
google-cloud-bigquery 0.29.0
There might be issues when using the default credentials. However, you can use a service account, save the credentials in a JSON file and add the necessary scopes. I did a quick test and this code worked for me:
from google.cloud import bigquery
from google.oauth2.service_account import Credentials
scopes = (
'https://www.googleapis.com/auth/bigquery',
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/drive'
)
credentials = Credentials.from_service_account_file('/path/to/credentials.json')
credentials = credentials.with_scopes(scopes)
client = bigquery.Client(credentials=credentials)
query = "SELECT * FROM dataset.federated_table LIMIT 5"
query_job = client.query(query)
rows = query_job.result()
for row in rows: print(row)
If you get a 404 not found error is because you need to share the spreadsheet with the service account (view permission)