Develop Reference

GMail Python API returns: 400 Precondition check failed when running on Cloud Composer with Workload Identity

I am trying to build an Airflow DAG (on Cloud Composer) that reads emails from Gmail, using the Google API Python client.
I would like to avoid the use of JSON files for Service Accounts, and therefore I am trying to take advantage of Workload Identity. Therefore, I performed the following steps:
Created a Service Account (my-service-account#my-project.iam.gserviceaccount.com) that will then be used to impersonate the Google mail my-email#my-domain.com
Granted Cloud Composer Service account the roles/iam.serviceAccountTokenCreator to the Google mail Service Account
Delegated domain-wide authority to the service account with the scopes 'https://www.googleapis.com/auth/gmail.readonly' such that the service account my-service-account#my-project.iam.gserviceaccount.com is authorized to access the emails of my-email#my-domain.com.
Now I'm trying to use the Google API Python client, in order to instantiate a Gmail service and use it to search the inbox of my-email#my-domain.com. Here's the code:
import google.auth
import google.auth.impersonated_credentials
SERVICE_ACCOUNT = 'my-service-account#my-project.iam.gserviceaccount.com'
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
credentials, project_id = google.auth.default()
logging.info(f'Obtained application default credentials for project {project_id}.')
impersonated_credentials = google.auth.impersonated_credentials.Credentials(
source_credentials=credentials,
target_principal=SERVICE_ACCOUNT,
target_scopes=SCOPES,
)
logging.info(f'Obtained impersonated credentials for {SERVICE_ACCOUNT}')
service = build(
serviceName='gmail',
version='v1',
credentials=impersonated_credentials,
cache_discovery=False,
)
So initially, the code infers the Application Default Credentials (Cloud Composer), and then impersonates Cloud composer to act like the my-service-account#my-project.iam.gserviceaccount.com Service Account). Finally, it uses the returned credentials to build the gmail service.
When attempting to run a query:
results = service.users().messages().list(userId='me', q='from: someEmail#outlook.com').execute()
I get the following error:
[2022-11-14, 18:23:47 UTC] {standard_task_runner.py:93} ERROR - Failed to execute job 604219 for task test (<HttpError 400 when requesting https://gmail.googleapis.com/gmail/v1/users/me/messages?q=from%3A+someEmail%40outlook.com&alt=json returned "Precondition check failed.". Details: "Precondition check failed.">; 30352)
Any clue what I might be missing here? I've found a few similar questions but apparently they all use Service Account JSON files, which is clearly not the case here.

Retrive endpoint url of deployed app from google cloud run with Python

I want to send requests to a deployed app on a cloud run with python, but inside the test file, I don't want to hardcode the endpoint; how can I get the URL of the deployed app with python script inside the test file so that I can send requests to that URL?

You can use gcloud to fetch the url of the service like this
gcloud run services describe SERVICE_NAME
--format="value(status.url)"

In a pure Python way, you can use Google's API Client Library for Run.
To my knowledge, there isn't a Cloud Client Library
The method is namespaces.services.get and it is documented by APIs Explorer namespaces.services.get.
One important fact with Cloud Run is that the API endpoint differs by Cloud Run region.
See service endpoint. You will need to override the client configuration (using ClientOptions) with the correct (region-specific) api_endpoint.
The following is from-memory! I've not run this code but it should be (nearly) correct:
import google.auth
import os
from googleapiclient import discovery
from google.api_core.client_options import ClientOptions
creds, project = google.auth.default()
REGION = os.getenv("REGION")
SERVICE = os.getenv("SERVICE")
# Must override the default run.googleapis.com endpoint
# with region-specific endpoint
api_endpoint = "https://{region}-run.googleapis.com".format(
region=REGION
)
options = ClientOptions(
api_endpoint=api_endpoint
)
service = discovery.build("run", "v1",
client_options=options,
credentials=creds
)
name = "namespaces/{namespace}/services/{service}".format(
namespace=project,
service=SERVICE
)
rqst = service.namespaces().services().get(name=name)
resp = rqst.execute()
The resp will be Service and you can grab its ServiceStatus url.

Querying federated tables (Google Drive) via BigQuery python api

(There are a lot of similar threads here but unfortunately I couldn't find the answer to my error anywhere here or on Goolge)
I'm trying to query a federated table in BigQuery which is pointing to a spreadsheet in Drive.
I've run the following command to create default application credentials for gcloud:
$ gcloud auth application-default login
But this doesn't include Drive into the scope so I'm getting the following error message (which makes sense): Forbidden: 403 Access Denied: BigQuery BigQuery: No OAuth token with Google Drive scope was found.
Then I've tried to auth with explicit Drive scope:
$ gcloud auth application-default login --scopes=https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/bigquery
After that I'm getting the following error when I try to use bigquery python api:
"Forbidden: 403 Access Denied: BigQuery BigQuery: Access Not Configured. Drive API has not been used in project 764086051850 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/drive.googleapis.com/overview?project=764086051850 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry."
The project number above does not exist in our organisation and the provided link leads to a page which says:
The API "drive.googleapis.com" doesn't exist or you don't have permission to access it
Drive API is definitely enabled for the default project, so the error message doesn't make much sense. I can also query the table from the terminal using bq query_string command.
I'm currently out of ideas on how to debug this further, anyone suggestions?
Configuration:
Google Cloud SDK 187.0.0
Python 2.7
google-cloud 0.27.0
google-cloud-bigquery 0.29.0

There might be issues when using the default credentials. However, you can use a service account, save the credentials in a JSON file and add the necessary scopes. I did a quick test and this code worked for me:
from google.cloud import bigquery
from google.oauth2.service_account import Credentials
scopes = (
'https://www.googleapis.com/auth/bigquery',
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/drive'
)
credentials = Credentials.from_service_account_file('/path/to/credentials.json')
credentials = credentials.with_scopes(scopes)
client = bigquery.Client(credentials=credentials)
query = "SELECT * FROM dataset.federated_table LIMIT 5"
query_job = client.query(query)
rows = query_job.result()
for row in rows: print(row)
If you get a 404 not found error is because you need to share the spreadsheet with the service account (view permission)

Google App Engine - Issue authenticating deployed version of app

I built a simple python application to be run on the Google App Engine. Code:
import webapp2
from oauth2client.contrib.appengine import AppAssertionCredentials
from apiclient.discovery import build
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
self.response.write('BigQuery App')
credentials = AppAssertionCredentials(
'https://www.googleapis.com/auth/sqlservice.admin')
service = discovery.build('bigquery', 'v2', credentials=credentials)
projectId = '<Project-ID>'
query_request_body = {
"query": "SELECT a from Data.test LIMIT 10"
}
request = service.jobs().query(projectId=projectId, body=query_request_body)
response = request.execute()
self.response.write(response)
app = webapp2.WSGIApplication([
('/', MainPage),
], debug=True)
I am able to deploy this code locally (http://localhost:8080) and everything works correctly, however I get the following error 500 Server Error when I try to deploy it to GAE using:
appcfg.py -A <Project-Id> -V v1 update .
This is the error I get from the Error Report Console:
error: An error occured while connecting to the server: DNS lookup failed for URL:http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/https://www.googleapis.com/auth/sqlservice.admin/?recursive=True
I believe it is an auth issue and to make sure my service account was authorized I went through the gcloud authentification for service accounts and I also set the set environment variables from the SDK.
I have been trying to get around this for a while, any pointers are very appreciated. Thank you.
Also, I have been using Service Account Auth by following these docs: https://developers.google.com/identity/protocols/OAuth2ServiceAccount where it says that I shouldn't be able to run AppAsseritionCredenitals locally, which adds to my confusion because I actually can with no errors.
EDIT:
After reuploading and reauthorizing my service account I was able to connect to the server. However, the authorization error continues with this:
HttpError: <HttpError 403 when requesting https://www.googleapis.com/bigquery/v2/projects/sqlserver-1384/queries?alt=json returned "Insufficient Permission">

To fix the "error while connecting to the server", follow the instructions listed in this answer: https://stackoverflow.com/questions/31651973/default-credentials-in-google-app-engine-invalid-credentials-error#=
and then re-upload the app
Then, to fix the HttpError 403 when requesting ... returned "Insufficient Permission", you have to change the scope you were requesting. In my case I was requesting:
credentials = AppAssertionCredentials(
'https://www.googleapis.com/auth/sqlservice.admin')
however, the correct scope for Google BigQuery is: https://www.googleapis.com/auth/bigquery. Which looks like this:
credentials = AppAssertionCredentials(
'https://www.googleapis.com/auth/bigquery')
If you are using a different API, use whichever scope is outlined in the documentations.

Working with Google Drive API on the google cloud

I have a google app engine site, and what I want to do, is get access to the files on my drive and publish them. Note that, my account owns both the drive and the app engine page.
I have tried looking at the google drive api, and the problem is that I don't know where to start with the following boilerplate code located in their documentation.
If you take a look at this function:
def get_credentials(authorization_code, state):
"""Retrieve credentials using the provided authorization code.
This function exchanges the authorization code for an access token and queries
the UserInfo API to retrieve the user's e-mail address.
If a refresh token has been retrieved along with an access token, it is stored
in the application database using the user's e-mail address as key.
If no refresh token has been retrieved, the function checks in the application
database for one and returns it if found or raises a NoRefreshTokenException
with the authorization URL to redirect the user to.
Args:
authorization_code: Authorization code to use to retrieve an access token.
state: State to set to the authorization URL in case of error.
Returns:
oauth2client.client.OAuth2Credentials instance containing an access and
refresh token.
Raises:
CodeExchangeError: Could not exchange the authorization code.
NoRefreshTokenException: No refresh token could be retrieved from the
available sources.
"""
email_address = ''
try:
credentials = exchange_code(authorization_code)
user_info = get_user_info(credentials)
email_address = user_info.get('email')
user_id = user_info.get('id')
if credentials.refresh_token is not None:
store_credentials(user_id, credentials)
return credentials
else:
credentials = get_stored_credentials(user_id)
if credentials and credentials.refresh_token is not None:
return credentials
except CodeExchangeException, error:
logging.error('An error occurred during code exchange.')
# Drive apps should try to retrieve the user and credentials for the current
# session.
# If none is available, redirect the user to the authorization URL.
error.authorization_url = get_authorization_url(email_address, state)
raise error
except NoUserIdException:
logging.error('No user ID could be retrieved.')
# No refresh token has been retrieved.
authorization_url = get_authorization_url(email_address, state)
raise NoRefreshTokenException(authorization_url)
This is a part of the boilerplate code. However, where am I supposed to get authorisation_code from?

I recently had to implement something similar, and it is quite tricky to find the relevant pieces of documentation.
This is what worked for me.
One-time setup to enable Google Drive for your Google App Engine project
Go to the Google APIs Console and select your App Engine project. If you don't see your App Engine project listed, you need to enable the cloud integration in the App Engine admin tool first (Administration > Application Settings > Cloud Integration > Create project)
In Google APIs Console, now go to Services and look for the "Drive API" in that long list. Turn it on.
Go to the API Access section on Google APIs Console, and find back the "Simple API Access" API Key. (see screenshot below)
Getting and installing the Python Drive API Client
Download the Python Drive API Client: https://developers.google.com/api-client-library/python/start/installation#appengine
Documentation on this Python API: https://google-api-client-libraries.appspot.com/documentation/drive/v2/python/latest/
Using the Python Drive API Client
To create the Drive service object, I use this:
import httplib2
def createDriveService():
"""Builds and returns a Drive service object authorized with the
application's service account.
Returns:
Drive service object.
"""
from oauth2client.appengine import AppAssertionCredentials
from apiclient.discovery import build
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/drive')
http = httplib2.Http()
http = credentials.authorize(http)
return build('drive', 'v2', http=http, developerKey=API_KEY)
You can then use this service object to execute Google Drive API calls, for example, to create a folder:
service = createDriveService()
res = {'title': foldername,
'mimeType': "application/vnd.google-apps.folder"}
service.files().insert(body=res).execute()
Caveats
I was not able to get the Drive API to work in unittesting, nor on the dev_appserver. I always get an error that my credentials are not valid. However, it works fine on the real app engine server.