Develop Reference

GMail Python API returns: 400 Precondition check failed when running on Cloud Composer with Workload Identity

I am trying to build an Airflow DAG (on Cloud Composer) that reads emails from Gmail, using the Google API Python client.
I would like to avoid the use of JSON files for Service Accounts, and therefore I am trying to take advantage of Workload Identity. Therefore, I performed the following steps:
Created a Service Account (my-service-account#my-project.iam.gserviceaccount.com) that will then be used to impersonate the Google mail my-email#my-domain.com
Granted Cloud Composer Service account the roles/iam.serviceAccountTokenCreator to the Google mail Service Account
Delegated domain-wide authority to the service account with the scopes 'https://www.googleapis.com/auth/gmail.readonly' such that the service account my-service-account#my-project.iam.gserviceaccount.com is authorized to access the emails of my-email#my-domain.com.
Now I'm trying to use the Google API Python client, in order to instantiate a Gmail service and use it to search the inbox of my-email#my-domain.com. Here's the code:
import google.auth
import google.auth.impersonated_credentials
SERVICE_ACCOUNT = 'my-service-account#my-project.iam.gserviceaccount.com'
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
credentials, project_id = google.auth.default()
logging.info(f'Obtained application default credentials for project {project_id}.')
impersonated_credentials = google.auth.impersonated_credentials.Credentials(
source_credentials=credentials,
target_principal=SERVICE_ACCOUNT,
target_scopes=SCOPES,
)
logging.info(f'Obtained impersonated credentials for {SERVICE_ACCOUNT}')
service = build(
serviceName='gmail',
version='v1',
credentials=impersonated_credentials,
cache_discovery=False,
)
So initially, the code infers the Application Default Credentials (Cloud Composer), and then impersonates Cloud composer to act like the my-service-account#my-project.iam.gserviceaccount.com Service Account). Finally, it uses the returned credentials to build the gmail service.
When attempting to run a query:
results = service.users().messages().list(userId='me', q='from: someEmail#outlook.com').execute()
I get the following error:
[2022-11-14, 18:23:47 UTC] {standard_task_runner.py:93} ERROR - Failed to execute job 604219 for task test (<HttpError 400 when requesting https://gmail.googleapis.com/gmail/v1/users/me/messages?q=from%3A+someEmail%40outlook.com&alt=json returned "Precondition check failed.". Details: "Precondition check failed.">; 30352)
Any clue what I might be missing here? I've found a few similar questions but apparently they all use Service Account JSON files, which is clearly not the case here.

Firestore SDK hangs in production

I'm using the Firebase Admin Python SDK to read/write data to Firestore. I've created a service account with the necessary permissions and saved the credentials .json file in the source code (I know this isn't the most secure, but I want to get the thing running before fixing security issues). When testing the integration locally, it works flawlessly. But after deploying to GCP, where our service is hosted, calls to Firestore don't work properly and retry for a while before throwing 503 Deadline Exceeded errors. However, SSHing into a GKE pod and calling the SDK manually works without issues. It's just when the SDK is used in code flow that causes problems.
Our service runs in Google Kubernetes Engine in one project (call it Project A), but the Firestore database is in another project (call it project B). The service account that I'm trying to use is owned by Project B, so it should still be able to access the database even when it is being initialized from inside Project A.
Here's how I'm initiating the SDK:
from firebase_admin import get_app
from firebase_admin import initialize_app
from firebase_admin.credentials import Certificate
from firebase_admin.firestore import client
from google.api_core.exceptions import AlreadyExists
credentials = Certificate("/path/to/credentials.json")
try:
app = initialize_app(credential=credentials, name="app_name")
except ValueError:
app = get_app(name="app_name")
client = client(app=app)
Another wrinkle is that another part of our code is able to successfully use the same service account to produce Firebase Access Tokens. The successful code is:
import firebase_admin
from firebase_admin import auth as firebase_admin_auth
if "app_name" in firebase_admin._apps:
# Already initialized
app = firebase_admin.get_app(name="app_name")
else:
# Initialize
credentials = firebase_admin.credentials.Certificate("/path/to/credentials.json")
app = firebase_admin.initialize_app(credential=credentials, name="app_name")
firebase_token = firebase_admin_auth.create_custom_token(
uid="id-of-user",
developer_claims={"admin": is_admin, "site_slugs": read_write_site_slugs},
app=app,
)
Any help appreciated.

Turns out that the problem here was a conflict between gunicorn's gevents and the SDK's use of gRCP. Something related to websockets. I found the solution here. I added the following code to our Django app's settings:
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

how do I get the project of a service account?

I'm using the python google.cloud api
For example using the metrics module
from google.cloud import monitoring
client = monitoring.Client()
client.query(my/gcp/metric, minutes=10)
For my GOOGLE_APPLICATION_CREDENTIALS im using a service account that has specific access to a gcp project.
Does google.cloud have any modules that can let me derive the project from the service account (like get what project the service account is in)?
This would be convenient because each service account only has access to a single project, so I could set my service account and be able to reference that project in code.

Not sure if this will work, you may need to tweak it:
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = discovery.build('yourservicename', credentials=credentials)
request = service.projects().list()[0]

Google Cloud Identity and Access Management (IAM) API has ‘serviceAccounts.get’ method and which shows the projects associated with a service account as shown here. You need to have proper permissions on the projects for the API to work.

The method google.auth.default return a tuple (project_id, credentials) if that information is available on the environment.
Also, the client object knows to which project it is linked from (either client.project or client.project_id, I'm not sure which one for the Monitoring API).
If you set the service account manually with the GOOGLE_APPLICATION_CREDENTIALS env var, you can open the file and load its json. One of the parameters in a service account key file is the project id.

Cloud Storage: how to setup service account credentials for python boto library

I'm following this tutorial to upload a file to a bucket I've manually created:
https://cloud.google.com/storage/docs/xml-api/gspythonlibrary
I seem to have trouble with setting up the credentials both as service account or user account. I want to use this in a web server so ideally it should be setup with service account.
I created an API using API Manager in Console and downloaded the JSON. Meanwhile my gcloud auth is setup with my OAUTH login. I did try gsutil config -e and got error:
CommandException: OAuth2 is the preferred authentication mechanism with the Cloud SDK. Run "gcloud auth login" to configure authentication, unless you want to authenticate with an HMAC access key and secret, in which case run "gsutil config -a".
I also tried to authenticate the service account using:
gcloud auth activate-service-account --key-file <json file>
but still no luck with enabling access with python boto. I also copied the ID and Key from ~/.config/gcloud/ to ~/.boto but that didn't work either. I'm not sure how am I supposed to setup the authentication for the python server to access cloud storage. I'm not using App Engine but Cloud Compute to setup the webserver.
Here's my source code:
import boto
import gcs_oauth2_boto_plugin
import os
import shutil
import StringIO
import tempfile
import time
CLIENT_ID = 'my client id from ~/.config/gcloud/credentials'
CLIENT_SECRET = 'my client secret from ~/.config/gcloud/credentials'
gcs_oauth2_boto_plugin.SetFallbackClientIdAndSecret(CLIENT_ID, CLIENT_SECRET)
uri = boto.storage_uri('', 'gs')
project_id = 'my-test-project-id'
header_values = {"x-test-project-id": project_id}
# If the default project is defined, call get_all_buckets() without arguments.
for bucket in uri.get_all_buckets(headers=header_values):
print bucket.name
Most recent error:
Traceback (most recent call last):
File "upload/uploader.py", line 14, in <module>
for bucket in uri.get_all_buckets(headers=header_values):
File "/Users/ankitjain/dev/metax/venv/lib/python2.7/site-packages/boto/storage_uri.py", line 574, in get_all_buckets
return conn.get_all_buckets(headers)
File "/Users/ankitjain/dev/metax/venv/lib/python2.7/site-packages/boto/s3/connection.py", line 444, in get_all_buckets
response.status, response.reason, body)
boto.exception.GSResponseError: GSResponseError: 403 Forbidden
<?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidSecurity</Code><Message>The provided security credentials are not valid.</Message><Details>Incorrect Authorization header</Details></Error>

Okay after some more experiments I gave up on using GCE library for uploading data to Cloud Storage. I actually found using AWS boto to upload to cloud storage works much better. All I had to do was specify Google's host in s3 library:
conn = boto.connect_s3(app.config['S3_KEY'], app.config['S3_SECRET'], "c.storage.googleapis.com")
bucket = conn.get_bucket(app.config['S3_BUCKET'], validate=False)
I used the HMAC credentials generated the way described in the Google Docs. Reference:
https://cloud.google.com/storage/docs/migrating

Working with Google Drive API on the google cloud

I have a google app engine site, and what I want to do, is get access to the files on my drive and publish them. Note that, my account owns both the drive and the app engine page.
I have tried looking at the google drive api, and the problem is that I don't know where to start with the following boilerplate code located in their documentation.
If you take a look at this function:
def get_credentials(authorization_code, state):
"""Retrieve credentials using the provided authorization code.
This function exchanges the authorization code for an access token and queries
the UserInfo API to retrieve the user's e-mail address.
If a refresh token has been retrieved along with an access token, it is stored
in the application database using the user's e-mail address as key.
If no refresh token has been retrieved, the function checks in the application
database for one and returns it if found or raises a NoRefreshTokenException
with the authorization URL to redirect the user to.
Args:
authorization_code: Authorization code to use to retrieve an access token.
state: State to set to the authorization URL in case of error.
Returns:
oauth2client.client.OAuth2Credentials instance containing an access and
refresh token.
Raises:
CodeExchangeError: Could not exchange the authorization code.
NoRefreshTokenException: No refresh token could be retrieved from the
available sources.
"""
email_address = ''
try:
credentials = exchange_code(authorization_code)
user_info = get_user_info(credentials)
email_address = user_info.get('email')
user_id = user_info.get('id')
if credentials.refresh_token is not None:
store_credentials(user_id, credentials)
return credentials
else:
credentials = get_stored_credentials(user_id)
if credentials and credentials.refresh_token is not None:
return credentials
except CodeExchangeException, error:
logging.error('An error occurred during code exchange.')
# Drive apps should try to retrieve the user and credentials for the current
# session.
# If none is available, redirect the user to the authorization URL.
error.authorization_url = get_authorization_url(email_address, state)
raise error
except NoUserIdException:
logging.error('No user ID could be retrieved.')
# No refresh token has been retrieved.
authorization_url = get_authorization_url(email_address, state)
raise NoRefreshTokenException(authorization_url)
This is a part of the boilerplate code. However, where am I supposed to get authorisation_code from?

I recently had to implement something similar, and it is quite tricky to find the relevant pieces of documentation.
This is what worked for me.
One-time setup to enable Google Drive for your Google App Engine project
Go to the Google APIs Console and select your App Engine project. If you don't see your App Engine project listed, you need to enable the cloud integration in the App Engine admin tool first (Administration > Application Settings > Cloud Integration > Create project)
In Google APIs Console, now go to Services and look for the "Drive API" in that long list. Turn it on.
Go to the API Access section on Google APIs Console, and find back the "Simple API Access" API Key. (see screenshot below)
Getting and installing the Python Drive API Client
Download the Python Drive API Client: https://developers.google.com/api-client-library/python/start/installation#appengine
Documentation on this Python API: https://google-api-client-libraries.appspot.com/documentation/drive/v2/python/latest/
Using the Python Drive API Client
To create the Drive service object, I use this:
import httplib2
def createDriveService():
"""Builds and returns a Drive service object authorized with the
application's service account.
Returns:
Drive service object.
"""
from oauth2client.appengine import AppAssertionCredentials
from apiclient.discovery import build
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/drive')
http = httplib2.Http()
http = credentials.authorize(http)
return build('drive', 'v2', http=http, developerKey=API_KEY)
You can then use this service object to execute Google Drive API calls, for example, to create a folder:
service = createDriveService()
res = {'title': foldername,
'mimeType': "application/vnd.google-apps.folder"}
service.files().insert(body=res).execute()
Caveats
I was not able to get the Drive API to work in unittesting, nor on the dev_appserver. I always get an error that my credentials are not valid. However, it works fine on the real app engine server.