Develop Reference

Why do I get `secretmanager.versions.access` denied in GCP?

I am trying to access a secret stored in secrets manager.
I created a service account with owner role. I created a key from it. I run:
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = './keyfile.json'
from google.cloud import secretmanager
secret_client = secretmanager.SecretManagerServiceClient()
secret_name = f'projects/{project_id}/secrets/{secret_id}/versions/{version_id}'
response = secret_client.access_secret_version(request={"name": secret_name})
but I get:
google.api_core.exceptions.PermissionDenied: 403 Permission 'secretmanager.versions.access'
denied for resource 'projects/myprojnumber/secrets/mysecret/versions/1' (or it may not exist).
I checked the secret_name was the same as the secret's value in secret manager.
I have tried adding Secret Manager Secret Accessor and Secret Manager Viewer roles.
Edit: running this from cloud shell.

I think the issue is that the code is taking the Default Credentials of the Cloud Shell instead of using your SA key.
You can specify the credentials when creating the client
from google.cloud import secretmanager
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file("./keyfile.json")
secret_client = secretmanager.SecretManagerServiceClient(credentials=credentials)
secret_name = f'projects/{project_id}/secrets/{secret_id}/versions/{version_id}'
response = secret_client.access_secret_version(request={"name": secret_name})
Another option using some of the methods found in the library docs:
from google.cloud import secretmanager
secret_client = secretmanager.SecretManagerServiceClient.from_service_account_file("./keyfile.json")
secret_name = f'projects/{project_id}/secrets/{secret_id}/versions/{version_id}'
response = secret_client.access_secret_version(request={"name": secret_name})
Just as an advice, being newbie does not mean you cannot Google a little more to search for something like how to use a SA as credential for the client of the library you're using.
For example you could easily find this doc which shows a sample.
Anyway, good luck with GCP!

GDrive export using Service Account creds fails with 404

I have a script to export text from a GDrive file using an OAuth client, which works perfectly well -
import googleapiclient.discovery as google
from apiclient.http import MediaIoBaseDownload
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
import datetime, io, os, pickle
Scopes=" ".join(['https://www.googleapis.com/auth/drive.file',
'https://www.googleapis.com/auth/drive.metadata',
'https://www.googleapis.com/auth/drive.readonly'])
TokenFile="token.pickle"
def init_creds(clientfile,
scopes,
tokenfile=TokenFile):
token=None
if os.path.exists(tokenfile):
with open(tokenfile, 'rb') as f:
token=pickle.load(f)
if (not token or
not token.valid or
token.expiry < datetime.datetime.utcnow()):
if (token and
token.expired and
token.refresh_token):
token.refresh(Request())
else:
flow=InstalledAppFlow.from_client_secrets_file(clientfile, scopes)
token=flow.run_local_server(port=0)
with open(tokenfile, 'wb') as f:
pickle.dump(token, f)
return token
def export_text(id,
clientfile,
scopes=Scopes):
creds=init_creds(clientfile=clientfile,
scopes=scopes)
service=google.build('drive', 'v3', credentials=creds)
request=service.files().export_media(fileId=id,
mimeType='text/plain')
buf=io.BytesIO()
downloader, done = MediaIoBaseDownload(buf, request), False
while done is False:
status, done = downloader.next_chunk()
destfilename="tmp/%s.txt" % id
return buf.getvalue().decode("utf-8")
if __name__=='__main__':
print (export_text(id="#{redacted}"
clientfile="/path/to/oath/client.json"))
But it's a pain to have to go through the OAuth flow every time, and since it's only me using the script I want to simplify things and use a Service Account instead, following on from this post -
Google Drive API Python Service Account Example
My new Service Account script, doing exactly the same thing, is as follows -
import googleapiclient.discovery as google
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.http import MediaIoBaseDownload
import io
Scopes=" ".join(['https://www.googleapis.com/auth/drive.file',
'https://www.googleapis.com/auth/drive.metadata',
'https://www.googleapis.com/auth/drive.readonly'])
def export_text(id,
clientfile,
scopes=Scopes):
creds=ServiceAccountCredentials.from_json_keyfile_name(clientfile,
scopes)
service=google.build('drive', 'v3', credentials=creds)
request=service.files().export_media(fileId=id,
mimeType='text/plain')
buf=io.BytesIO()
downloader, done = MediaIoBaseDownload(buf, request), False
while done is False:
status, done = downloader.next_chunk()
destfilename="tmp/%s.txt" % id
return buf.getvalue().decode("utf-8")
if __name__=='__main__':
print (export_text(id="#{redacted}",
clientfile="path/to/service/account.json"))
but when I run it for the same id, I get the following -
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://www.googleapis.com/drive/v3/files/#{redacted}/export?mimeType=text%2Fplain&alt=media returned "File not found: #{redacted}.">
It feels like the Service Account script is passing the authentication step (ie Service Account creds are okay) but then failing when trying to fetch the file - weird as I can fetch it fine using the OAuth version :/
Any thoughts on what might be causing this 404 error in the Service Account version, given the OAuth client version clearly works for the same id?

Answer:
You need to share your file with the service account.
More Information:
As you would with any file, you need to give a user explicit permissions to be able to see it. As a service account is a separate entitiy to you, this goes for them as well.
Using the file sharing settings (you can just do this in the Drive UI by right-clicking the file and hitting Share), give the email address of the service account the correct permission (read/write). The email address of the service account is in the form:
service-account-name#project-id.iam.gserviceaccount.com

Before making your call do a File.list to see which files the service account has access to. Doing a file.get on a file that the service account doesn't have access to will result in a file not found error. Remember that the service account is not you, it has its own google drive account. Any files you want to access need to be uploaded to its account or shared with the service account.
If the file.list fails then it would suggest to me that there is something wrong with the authorization and you should ensure that the service account has access to client file maybe its that file it cant find.
Granting service account acccess
Create a directory on your personal google drive account. Take the service account email address, it can be found in the key file you downloaded it has a # in it. Then share that directory on your drive account with the service account like you would share with any other user.
Adding files to that directory may or may not give the service account access to them automatically permissions is a pain you may need to also share the file with the service account.
Remember to have the service account grant your personal account permissions to the file when it uploads it its going to be the owner.

gsuite service account for directory api returns http 400 errors: Bad request/Invalid input

I have started to develop some apis to create users in my G suite directory.
I followed the service account tutorials along with the Directory tutorials for python. The code I have is very simple just to test out how it will work.
from google.oauth2 import service_account
from googleapiclient.discovery import build
SCOPES = ['https://www.googleapis.com/auth/admin.directory.user']
SERVICE_ACCOUNT_FILE = 'file'
creds = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
service = build('admin', 'directory_v1', credentials=creds)
results = service.users().list(customer='i am not sure what customer is', maxResults=10, orderBy='email').execute()
#this line produces the error.
#Vscode also states the service has no member users. But I did install all #the libraries
users = results.get('users', [])
print(users)
The documentation to me is unclear about most things. When I run this I get
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=students&maxResults=10&orderBy=email&alt=json returned "Bad Request">
When I change customer from my_customer to something else I get Invalid Input.
Any suggestions to what may cause this error and preferably how to work with this api via a service account?
Now I did enable the directory api and create the service account and downloaded the service account file as well. Am I missing a step?
I would also prefer if someone has a better documentation that I was unable to find.

Finally I resolved this issue by setting another parameter "subject" when calling "service_account.Credentials.from_service_account_file".

Ensure that the service account has domain-wide-delegation enabled and has the proper scopes.

How to get a GCP Bearer token programmatically with python

gcloud auth print-access-token gives me a Bearer token that I can use later on; however, this is a shell command. How would I obtain one programmatically via the Google Cloud Python API?
I see a prior example using oauth2client, but oauth2client is now deprecated. How would I do this with google.auth and oauthlib?

While the above answer is quite informative, it misses one important point - credentials object obtained from google.auth.default() or compute_engine.Credentials() will not have token in it. So back to the original question of what is the programmatic alternative to gcloud auth print-access-token, my answer would be:
import google.auth
import google.auth.transport.requests
creds, project = google.auth.default()
# creds.valid is False, and creds.token is None
# Need to refresh credentials to populate those
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
# Now you can use creds.token
I'm using the official google-auth package and default credentials, which will get you going both in local dev and on remote GCE/GKE app.
Too bad this is not properly documented and I had to read google-auth code to figure our how to obtain the token.

The answer depends on your environment and how you want to create / obtain credentials.
What are Google Cloud Credentials?
Google Cloud credentials are an OAuth 2.0 token. This token has at a minimum an Access Token and optionally a Refresh Token, Client ID Token, and supporting parameters such as expiration, Service Account Email or Client Email, etc.
The important item in Google Cloud APIs is the Access Token. This token is what authorizes access to the cloud. This token can be used in programs such as curl, software such as python, etc and does not require an SDK. The Access Token is used in the HTTP Authorization header.
What is an Access Token?
An access token is an opaque value generated by Google that is derived from a Signed JWT, more correctly called JWS. A JWT consists of a header and claims (the payload) Json structures. These two Json structures are signed with the Service Account's Private Key. These values are base64 encoded and concatenated to create the Access Key.
The format of an Access Token is: base64(header) + '.' + base64(payload) + '.' + base64(signature).
Here is an example JWT:
Header:
{
"alg": "RS256",
"typ": "JWT",
"kid": "42ba1e234ac91ffca687a5b5b3d0ca2d7ce0fc0a"
}
Payload:
{
"iss": "myservice#myproject.iam.gserviceaccount.com",
"iat": 1493833746,
"aud": "myservice.appspot.com",
"exp": 1493837346,
"sub": "myservice#myproject.iam.gserviceaccount.com"
}
Using an Access Token:
Example that will start a VM instance. Replace PROJECT_ID, ZONE and INSTANCE_NAME. This example is for Windows.
curl -v -X GET -H "Authorization: Bearer <access_token_here>" ^
https://www.googleapis.com/compute/v1/projects/%PROJECT_ID%/zones/%ZONE%/instances/%INSTANCE_NAME%/start
Compute Engine Service Account:
Dustin's answer is correct for this case, but I will include for completeness with some additional information.
These credentials are automatically created for you by GCP and are obtained from the VM Instance metadata. Permissions are controlled by Cloud API access scopes in the Google Console.
However, these credentials have some limitations. To modify the credentials you must stop the VM Instance first. Additionally, not all permissions (roles) are supported.
from google.auth import compute_engine
cred = compute_engine.Credentials()
Service Account Credentials:
Until you understand all of the types of credentials and their use cases, these are the credentials that you will use for everything except for gcloud and gsutil. Understanding these credentials will make working with Google Cloud much simpler when writing programs. Obtaining credentials from a Google Service Account Json file is easy. The only item to make note of is that credentials expire (typically 60 minutes) and either need to be refreshed or recreated.
gcloud auth print-access-token is NOT recommended. Service Account Credentials are the recommended method by Google.
These credentials are created by the Console, gcloud or via programs / APIs. Permissions are assigned to the creditials by IAM and function inside Compute Engine, App Engine, Firestore, Kubernetes, etc. as well as other environments outside of Google Cloud. These credentials are downloaded from Google Cloud and stored in a Json file. Notice the scopes parameter. This defines permissions that are granted to the resulting credentials object.
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = 'service-account-credentials.json'
from google.oauth2 import service_account
cred = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
Google OAuth 2.0 Credentials:
These credentials are derived from a full OAuth 2.0 flow. These credentials are generated when your browser is launched to access Google Accounts for authorizing access. This process is much more complicated and requires a fair amount of code to implement and requires a built-in web server for the callback for authorization.
This method provides additional features such as being able to run everything in a browser, example you can create a Cloud Storage File Browser, but be careful that you understand the security implications. This method is the technique used to support Google Sign-In, etc. I like to use this method to authenticate users before allowing posting on websites, etc. The possibilities are endless with correctly authorized OAuth 2.0 identities and scopes.
Example code using google_auth_oauthlib:
from google_auth_oauthlib.flow import InstalledAppFlow
flow = InstalledAppFlow.from_client_secrets_file(
'client_secrets.json',
scopes=scope)
cred = flow.run_local_server(
host='localhost',
port=8088,
authorization_prompt_message='Please visit this URL: {url}',
success_message='The auth flow is complete; you may close this window.',
open_browser=True)
Example code using the requests_oauthlib library:
from requests_oauthlib import OAuth2Session
gcp = OAuth2Session(
app.config['gcp_client_id'],
scope=scope,
redirect_uri=redirect_uri)
# print('Requesting authorization url:', authorization_base_url)
authorization_url, state = gcp.authorization_url(
authorization_base_url,
access_type="offline",
prompt="consent",
include_granted_scopes='true')
session['oauth_state'] = state
return redirect(authorization_url)
# Next section of code after the browser approves the request
token = gcp.fetch_token(
token_url,
client_secret=app.config['gcp_client_secret'],
authorization_response=request.url)

In some cases, it's not possible to set environment variables on the server or container while needing a Bearer access token to call Google cloud APIs. I present the following to solve such problem:
# pip3 install google-auth
# pip3 install requests
import google.auth
import google.auth.transport.requests
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file('/home/user/secrets/hil-test.json', scopes=['https://www.googleapis.com/auth/cloud-platform'])
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
credentials.token
The last line would print the access token for calling Google cloud APIs. Replace ya29<REDACTED> in the following curl command with the printed token from python as a test:
curl https://example.googleapis.com/v1alpha1/projects/PROJECT_ID/locations -H "Authorization: Bearer ya29<REDACTED>"
It may not make sense to execute python to get the token then curl in BASH to call an API. The purpose is to demonstrate getting the token to call Google cloud Alpha API which may not have any Python client library but REST API. Developers can then use Python requests HTTP library to call the APIs.

import google.auth
import google.auth.transport.requests
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
#getting request object
auth_req = google.auth.transport.requests.Request()
print(credentials.valid) # prints False
credentials.refresh(auth_req) #refresh token
#cehck for valid credentials
print(credentials.valid) # prints True
print(credentials.token) # prints token

This may not be the recommended way but for Rest API in my application this was an easy way to get the token.
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
token = cmdline("gcloud auth application-default print-access-token")
print("Token:"+token)

I found myself here when looking for a way to use the python SDK without creating a service account. I wanted a way to locally develop a script that would run in the cloud. I was able to achieve this by using an artifact of the gcloud command:
export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/legacy_credentials/<me>/adc.json

Merging suggestions from this post and the google cloud documentation, I wrote an auxiliary function that returns a token. It generates a token if possible, and if not takes it from the environment, then checks that it's valid.
import google
import os
import requests
GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS"
GCS_OAUTH_TOKEN = "GCS_OAUTH_TOKEN"
SCOPE = "https://www.googleapis.com/auth/cloud-platform"
URL = "https://www.googleapis.com/oauth2/v1/tokeninfo"
PAYLOAD = "access_token={}"
HEADERS = {"content-type": "application/x-www-form-urlencoded"}
OK = "OK"
def get_gcs_token():
"""
Returns gcs access token.
Ideally, this function generates a new token, requries that GOOGLE_APPLICATION_CREDENTIALS be set in the environment
(os.environ).
Alternatively, environment variable GCS_OAUTH_TOKEN could be set if a token already exists
"""
if GOOGLE_APPLICATION_CREDENTIALS in os.environ:
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=[SCOPE])
# getting request object
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req) # refresh token
token = credentials.token
elif GCS_OAUTH_TOKEN in os.environ:
token = os.environ[GCS_OAUTH_TOKEN]
else:
raise ValueError(
f"""Could not generate gcs token because {GOOGLE_APPLICATION_CREDENTIALS} is not set in the environment.
Alternatively, environment variable {GCS_OAUTH_TOKEN} could be set if a token already exists, but it was not"""
)
r = requests.post(URL, data=PAYLOAD.format(token), headers=HEADERS)
if not r.reason == OK:
raise ValueError(
f"Could not verify token {token}\n\nResponse from server:\n{r.text}"
)
if not r.json()["expires_in"] > 0:
raise ValueError(f"token {token} expired")
return token

Official documentation code example
I followed this official documentation for Cloud Functions, which works for any GCP API:
auth_req = google.auth.transport.requests.Request()
id_token = google.oauth2.id_token.fetch_id_token(
auth_req,
# This is an OAuth authorisation scope that you must pass
# depending on the API.
# You can see an example of the need for this scope here: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert#authorization-scopes
"https://www.googleapis.com/auth/bigquery"
)
Now, you can use id_token in the Authorisation header:
headers = {'Authorization': f'Bearer {id_token}'}

how do I get the project of a service account?

I'm using the python google.cloud api
For example using the metrics module
from google.cloud import monitoring
client = monitoring.Client()
client.query(my/gcp/metric, minutes=10)
For my GOOGLE_APPLICATION_CREDENTIALS im using a service account that has specific access to a gcp project.
Does google.cloud have any modules that can let me derive the project from the service account (like get what project the service account is in)?
This would be convenient because each service account only has access to a single project, so I could set my service account and be able to reference that project in code.

Not sure if this will work, you may need to tweak it:
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = discovery.build('yourservicename', credentials=credentials)
request = service.projects().list()[0]

Google Cloud Identity and Access Management (IAM) API has ‘serviceAccounts.get’ method and which shows the projects associated with a service account as shown here. You need to have proper permissions on the projects for the API to work.

The method google.auth.default return a tuple (project_id, credentials) if that information is available on the environment.
Also, the client object knows to which project it is linked from (either client.project or client.project_id, I'm not sure which one for the Monitoring API).
If you set the service account manually with the GOOGLE_APPLICATION_CREDENTIALS env var, you can open the file and load its json. One of the parameters in a service account key file is the project id.