Develop Reference

How to create Google Cloud storage access token programmatically python

I need to have a public URL for a file that I am creating inside a google function.
I want therefore to create an access token :
I am able to upload the file from a python google function with the function blob.upload_from_string(blob_text), but I do not know how I can create a public url (or create an access token) for it.
Could you help me with it ?
EDITING WITH THE ANSWER (almost copy paste from Marc Anthony B answer )
blob = bucket.blob(storage_path)
token = uuid4()
metadata = {"firebaseStorageDownloadTokens": token}
blob.metadata = metadata
download_url = 'https://firebasestorage.googleapis.com/v0/b/{}/o/{}?alt=media&token={}' \
.format(bucket.name, storage_path.replace("/", "%2F"), token)
with open(video_file_path, 'rb') as f:
blob.upload_from_file(f)

Firebase Storage for Python still doesn't have its own SDK but you can use firebase-admin instead. Firebase Admin SDKs depend on the Google Cloud Storage client libraries to provide Cloud Storage access. The bucket references returned by the Admin SDK are objects defined in these libraries.
When uploading an object to Firebase Storage, you must incorporate a custom access token. You may use UUID4 for this case. See code below:
import firebase_admin
from firebase_admin import credentials
from firebase_admin import storage
from uuid import uuid4
projectId = '<PROJECT-ID>'
storageBucket = '<BUCKET-NAME>'
cred = credentials.ApplicationDefault()
firebase_admin.initialize_app(cred, {
'projectId': projectId,
'storageBucket': storageBucket
})
bucket = storage.bucket()
# E.g: "upload/file.txt"
bucket_path = "<BUCKET-PATH>"
blob = bucket.blob(bucket_path)
# Create a token from UUID.
# Technically, you can use any string to your token.
# You can assign whatever you want.
token = uuid4()
metadata = {"firebaseStorageDownloadTokens": token}
# Assign the token as metadata
blob.metadata = metadata
blob.upload_from_filename(filename="<FILEPATH>")
# Make the file public (OPTIONAL). To be used for Cloud Storage URL.
blob.make_public()
# Fetches a public URL from GCS.
gcs_storageURL = blob.public_url
# Generates a URL with Access Token from Firebase.
firebase_storageURL = 'https://firebasestorage.googleapis.com/v0/b/{}/o/{}?alt=media&token={}'.format(storageBucket, bucket_path, token)
print({
"gcs_storageURL": gcs_storageURL,
"firebase_storageURL": firebase_storageURL
})
As you can see from the code above, I've mentioned GCS and Firebase URLs. If you want a public URL from GCS then you should make the object public by using the make_public() method. If you want to use the access token generated, then just concatenate the default Firebase URL with the variables required.
If the objects are already in the Firebase Storage and already have access tokens incorporated on it, then you can get it by getting the objects metadata. See code below:
# E.g: "upload/file.txt"
bucket_path = "<BUCKET-PATH>"
blob = bucket.get_blob(bucket_path)
# Fetches object metadata
metadata = blob.metadata
# Firebase Access Token
token = metadata['firebaseStorageDownloadTokens']
firebase_storageURL = 'https://firebasestorage.googleapis.com/v0/b/{}/o/{}?alt=media&token={}'.format(storageBucket, bucket_path, token)
print(firebase_storageURL)
For more information, you may check out this documentation:
Google Cloud Storage Library for Python
Introduction to the Admin Cloud Storage API

Consuming API Workspace for Developers

I'm trying to consume Google's Workspace APIs, but I'm having extreme difficulty with the documentation to create my first code, following the first steps I did the following
I created a project within Google Cloud
I enabled the Admin SDK API
I created a service account
I created a key in Json format
in the Workspace dashboard under delegation across domain I added the unique id and the following scope
[
'https://www.googleapis.com/auth/apps.order',
'https://www.googleapis.com/auth/siteverification',
'https://www.googleapis.com/auth/directory.readonly',
'https://www.googleapis.com/auth/admin.directory.user',
'https://www.googleapis.com/auth/admin.reports.usage.readonly',
'https://www.googleapis.com/auth/admin.reports.audit.readonly',
'https://www.googleapis.com/auth/gmail.send'
]
I would like to use the document from the link https://developers.google.com/admin-sdk/reports/reference/rest to consult the activities of a specific user but I can't find an example code to consume this API using these credentials in Python , I'm new in this area and would like some help.
Generate a token and when I tried to use an api it didn't work and it was unauthorized, below is the code I used
import requests
url = "https://admin.googleapis.com/admin/reports/v1/activity/users/usuario#exemplo.com/applications/calendar"
payload = ""
headers = {"Authorization": "Bearer xptoz_exemple_test=PHQbcdddx3xxxxxxxxxxxxddddddddd"}
response = requests.request("GET", url, data=payload, headers=headers)
print(response.text)

You are getting unauthorized because the service account doesn't have permission to do what ever it is you are trying to do. To get permission you need to be using the proper scope, and the service account needs to not only have permission to use this scope but it must have delegated to a user on the domain that has access to the data.
First you need to be sure that your authorization code is delegating to a user on your domain.
#!/usr/bin/python
from oauth2client.service_account import ServiceAccountCredentials
from httplib2 import Http
from apiclient.discovery import build
scopes = [ 'https://www.googleapis.com/auth/admin.reports.usage.readonly' ]
credentials = ServiceAccountCredentials.from_json_keyfile_name('/path/to/my/key.json', scopes)
delegated_credentials = credentials.create_delegated('me#mydomain.com')
http_auth = credentials.authorize(Http())
service = build('admin', 'directory_v1', credentials=creds)
Then you should consider having a look at Python quickstart the auth in this is set up to use an installed app. However the rest of the code should show you how to use the Client library rather then sending all the requests manually like you are now.

Azure rest api with Python

I am new to Azure. I need to set up an automation script to list all Ips in azure using Azure Rest APi in Python. However i couldnt been able to get any result from the API url I am passing to the request.
I am following this https://learn.microsoft.com/en-us/rest/api/virtualnetwork/public-ip-addresses/list-all#code-try-0
In code block, I have tried using requests
import requests
apijson = requests.get("https://management.azure.com/subscriptions/9540b342-9f94-4dd9-9eca-0698dda0107c/providers/Microsoft.Network/publicIPAddresses?api-version=2021-05-01"
I know I need to setup Oauth/authentication but I don't know what I need to declare or pass any token from my script

I know I need to setup Oauth/authentication but I don't know what I need to declare or pass any token from my script
You can declare/pass token, for example:
import sys
import requests
import json
import time
test_api_url = "Add URL which you want to test"
#function to obtain a new OAuth 2.0 token from the authentication server
def get_new_token():
auth_server_url = "https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token'"
client_id = 'XXXX-XXXX-XXXX'
client_secret = 'XXXX'
token_req_payload = {'grant_type': 'client_credentials'}
token_response = requests.post(auth_server_url,
data=token_req_payload, verify=False, allow_redirects=False,
auth=(client_id, client_secret))
if token_response.status_code !=200:
print("Failed to obtain token from the OAuth 2.0 server", file=sys.stderr)
sys.exit(1)
print("Successfuly obtained a new token")
tokens = json.loads(token_response.text)
return tokens['access_token']
token = get_new_token()
while True:
api_call_headers = {'Authorization': 'Bearer ' + token}
api_call_response = requests.get(test_api_url, headers=api_call_headers,
verify+False)
if api_call_response.status_code == 401:
token = get_new_token()
else:
print(api_call_response.text)
time.sleep(30)
You can refer to How to authenticate and authorize Python applications on Azure, Azure AD Authentication Python Web API and Protect an API by using OAuth 2.0 with Azure Active Directory and API Management

How to get a GCP Bearer token programmatically with python

gcloud auth print-access-token gives me a Bearer token that I can use later on; however, this is a shell command. How would I obtain one programmatically via the Google Cloud Python API?
I see a prior example using oauth2client, but oauth2client is now deprecated. How would I do this with google.auth and oauthlib?

While the above answer is quite informative, it misses one important point - credentials object obtained from google.auth.default() or compute_engine.Credentials() will not have token in it. So back to the original question of what is the programmatic alternative to gcloud auth print-access-token, my answer would be:
import google.auth
import google.auth.transport.requests
creds, project = google.auth.default()
# creds.valid is False, and creds.token is None
# Need to refresh credentials to populate those
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
# Now you can use creds.token
I'm using the official google-auth package and default credentials, which will get you going both in local dev and on remote GCE/GKE app.
Too bad this is not properly documented and I had to read google-auth code to figure our how to obtain the token.

The answer depends on your environment and how you want to create / obtain credentials.
What are Google Cloud Credentials?
Google Cloud credentials are an OAuth 2.0 token. This token has at a minimum an Access Token and optionally a Refresh Token, Client ID Token, and supporting parameters such as expiration, Service Account Email or Client Email, etc.
The important item in Google Cloud APIs is the Access Token. This token is what authorizes access to the cloud. This token can be used in programs such as curl, software such as python, etc and does not require an SDK. The Access Token is used in the HTTP Authorization header.
What is an Access Token?
An access token is an opaque value generated by Google that is derived from a Signed JWT, more correctly called JWS. A JWT consists of a header and claims (the payload) Json structures. These two Json structures are signed with the Service Account's Private Key. These values are base64 encoded and concatenated to create the Access Key.
The format of an Access Token is: base64(header) + '.' + base64(payload) + '.' + base64(signature).
Here is an example JWT:
Header:
{
"alg": "RS256",
"typ": "JWT",
"kid": "42ba1e234ac91ffca687a5b5b3d0ca2d7ce0fc0a"
}
Payload:
{
"iss": "myservice#myproject.iam.gserviceaccount.com",
"iat": 1493833746,
"aud": "myservice.appspot.com",
"exp": 1493837346,
"sub": "myservice#myproject.iam.gserviceaccount.com"
}
Using an Access Token:
Example that will start a VM instance. Replace PROJECT_ID, ZONE and INSTANCE_NAME. This example is for Windows.
curl -v -X GET -H "Authorization: Bearer <access_token_here>" ^
https://www.googleapis.com/compute/v1/projects/%PROJECT_ID%/zones/%ZONE%/instances/%INSTANCE_NAME%/start
Compute Engine Service Account:
Dustin's answer is correct for this case, but I will include for completeness with some additional information.
These credentials are automatically created for you by GCP and are obtained from the VM Instance metadata. Permissions are controlled by Cloud API access scopes in the Google Console.
However, these credentials have some limitations. To modify the credentials you must stop the VM Instance first. Additionally, not all permissions (roles) are supported.
from google.auth import compute_engine
cred = compute_engine.Credentials()
Service Account Credentials:
Until you understand all of the types of credentials and their use cases, these are the credentials that you will use for everything except for gcloud and gsutil. Understanding these credentials will make working with Google Cloud much simpler when writing programs. Obtaining credentials from a Google Service Account Json file is easy. The only item to make note of is that credentials expire (typically 60 minutes) and either need to be refreshed or recreated.
gcloud auth print-access-token is NOT recommended. Service Account Credentials are the recommended method by Google.
These credentials are created by the Console, gcloud or via programs / APIs. Permissions are assigned to the creditials by IAM and function inside Compute Engine, App Engine, Firestore, Kubernetes, etc. as well as other environments outside of Google Cloud. These credentials are downloaded from Google Cloud and stored in a Json file. Notice the scopes parameter. This defines permissions that are granted to the resulting credentials object.
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = 'service-account-credentials.json'
from google.oauth2 import service_account
cred = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
Google OAuth 2.0 Credentials:
These credentials are derived from a full OAuth 2.0 flow. These credentials are generated when your browser is launched to access Google Accounts for authorizing access. This process is much more complicated and requires a fair amount of code to implement and requires a built-in web server for the callback for authorization.
This method provides additional features such as being able to run everything in a browser, example you can create a Cloud Storage File Browser, but be careful that you understand the security implications. This method is the technique used to support Google Sign-In, etc. I like to use this method to authenticate users before allowing posting on websites, etc. The possibilities are endless with correctly authorized OAuth 2.0 identities and scopes.
Example code using google_auth_oauthlib:
from google_auth_oauthlib.flow import InstalledAppFlow
flow = InstalledAppFlow.from_client_secrets_file(
'client_secrets.json',
scopes=scope)
cred = flow.run_local_server(
host='localhost',
port=8088,
authorization_prompt_message='Please visit this URL: {url}',
success_message='The auth flow is complete; you may close this window.',
open_browser=True)
Example code using the requests_oauthlib library:
from requests_oauthlib import OAuth2Session
gcp = OAuth2Session(
app.config['gcp_client_id'],
scope=scope,
redirect_uri=redirect_uri)
# print('Requesting authorization url:', authorization_base_url)
authorization_url, state = gcp.authorization_url(
authorization_base_url,
access_type="offline",
prompt="consent",
include_granted_scopes='true')
session['oauth_state'] = state
return redirect(authorization_url)
# Next section of code after the browser approves the request
token = gcp.fetch_token(
token_url,
client_secret=app.config['gcp_client_secret'],
authorization_response=request.url)

In some cases, it's not possible to set environment variables on the server or container while needing a Bearer access token to call Google cloud APIs. I present the following to solve such problem:
# pip3 install google-auth
# pip3 install requests
import google.auth
import google.auth.transport.requests
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file('/home/user/secrets/hil-test.json', scopes=['https://www.googleapis.com/auth/cloud-platform'])
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
credentials.token
The last line would print the access token for calling Google cloud APIs. Replace ya29<REDACTED> in the following curl command with the printed token from python as a test:
curl https://example.googleapis.com/v1alpha1/projects/PROJECT_ID/locations -H "Authorization: Bearer ya29<REDACTED>"
It may not make sense to execute python to get the token then curl in BASH to call an API. The purpose is to demonstrate getting the token to call Google cloud Alpha API which may not have any Python client library but REST API. Developers can then use Python requests HTTP library to call the APIs.

import google.auth
import google.auth.transport.requests
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
#getting request object
auth_req = google.auth.transport.requests.Request()
print(credentials.valid) # prints False
credentials.refresh(auth_req) #refresh token
#cehck for valid credentials
print(credentials.valid) # prints True
print(credentials.token) # prints token

This may not be the recommended way but for Rest API in my application this was an easy way to get the token.
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
token = cmdline("gcloud auth application-default print-access-token")
print("Token:"+token)

I found myself here when looking for a way to use the python SDK without creating a service account. I wanted a way to locally develop a script that would run in the cloud. I was able to achieve this by using an artifact of the gcloud command:
export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/legacy_credentials/<me>/adc.json

Merging suggestions from this post and the google cloud documentation, I wrote an auxiliary function that returns a token. It generates a token if possible, and if not takes it from the environment, then checks that it's valid.
import google
import os
import requests
GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS"
GCS_OAUTH_TOKEN = "GCS_OAUTH_TOKEN"
SCOPE = "https://www.googleapis.com/auth/cloud-platform"
URL = "https://www.googleapis.com/oauth2/v1/tokeninfo"
PAYLOAD = "access_token={}"
HEADERS = {"content-type": "application/x-www-form-urlencoded"}
OK = "OK"
def get_gcs_token():
"""
Returns gcs access token.
Ideally, this function generates a new token, requries that GOOGLE_APPLICATION_CREDENTIALS be set in the environment
(os.environ).
Alternatively, environment variable GCS_OAUTH_TOKEN could be set if a token already exists
"""
if GOOGLE_APPLICATION_CREDENTIALS in os.environ:
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=[SCOPE])
# getting request object
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req) # refresh token
token = credentials.token
elif GCS_OAUTH_TOKEN in os.environ:
token = os.environ[GCS_OAUTH_TOKEN]
else:
raise ValueError(
f"""Could not generate gcs token because {GOOGLE_APPLICATION_CREDENTIALS} is not set in the environment.
Alternatively, environment variable {GCS_OAUTH_TOKEN} could be set if a token already exists, but it was not"""
)
r = requests.post(URL, data=PAYLOAD.format(token), headers=HEADERS)
if not r.reason == OK:
raise ValueError(
f"Could not verify token {token}\n\nResponse from server:\n{r.text}"
)
if not r.json()["expires_in"] > 0:
raise ValueError(f"token {token} expired")
return token

Official documentation code example
I followed this official documentation for Cloud Functions, which works for any GCP API:
auth_req = google.auth.transport.requests.Request()
id_token = google.oauth2.id_token.fetch_id_token(
auth_req,
# This is an OAuth authorisation scope that you must pass
# depending on the API.
# You can see an example of the need for this scope here: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert#authorization-scopes
"https://www.googleapis.com/auth/bigquery"
)
Now, you can use id_token in the Authorisation header:
headers = {'Authorization': f'Bearer {id_token}'}

Interacting with Azure Key Vault using python w/ rest api

I am very interested in using the new service recently released for secret management within Azure. I have found a few example guides walking through how to interact with key vault via powershell cmdlets and c#, however haven't found much at all in regards to getting started with using the rest API.
The thing I am particularly confused with is the handling of oauth2 w/ active directory. I have written a oauth2 application listener, built a web application with an AD instance and can now generate a "access_token". It is very unclear to me how to proceed beyond this though, as I seem to consistently receive a 401 HTTP resp code whenever attempting to use my access_token to perform a key vault API call.
Any guides / tips on using azure key vault with python would be greatly appreciated!

Here are some steps you'll need to do before the following code will work... Hopefully I remembered everything!
You'll need to have an application in AD with at least get access
note: you need this to get the CLIENT_ID and CLIENT_SECRET anyway
then run:
azure keyvault set-policy --vault-name 'VAULTNAME' --spn CLIENT_ID --perms-to-secrets '["get"]'
You'll also need the id's for your secrets, which you can get with the Azure CLI using:
azure keyvault secret show [vault] [secret]
or
azure keyvault secret show -h # if this is unclear
Copy the key (last argument in the URL)
Then the following code will allow you to query the key vault using oauth2:
import json
import requests
AUTHORITY_HOST = "login.windows.net"
TENANT_ID = < your tenant id >
CLIENT_ID = < your client id >
CLIENT_SECRET = < your client secret >
VAULT = 'MyVault'
data = { "grant_type" : "client_credentials",
"client_id" : CLIENT_ID,
"client_secret" : CLIENT_SECRET,
"resource" : "https://vault.azure.net"
}
secrets = [( "i_like_pie", "8a7680a2cf5e4d539494aa0ce265297" )]
headers = { "Content-Type" : "application/x-www-form-urlencoded" }
r = requests.post("https://login.windows.net/{}/oauth2/token".format(TENANT_ID), data=data, headers=headers)
access_token = r.json()['access_token']
for secret, secret_id in secrets.iteritems():
headers = {"Authorization":"Bearer {}".format(access_token) }
r = requests.get('https://{}.vault.azure.net/secrets/{}/{}?api-version=2015-06-01'.format(VAULT, secret, secret_id), headers=headers)
print('##### {} #####'.format(secret))
print(r.json())
print('')

Here are a couple of things that you can check:
When you make the request for the Bearer token, make sure that you include the "resource" header, and that it is set to "https://vault.azure.net". If you don't, you'll get a token, but you won't be able to access any vault data with it.
When you make a call to the vault.azure.net URL, make sure you include the correct "api-version" It can be found in the API documentation. The current value is "2015-02-01-preview".
Of course, check that the Key Vault Access Policy is set correctly for the vault you are trying to access.

For working with Key Vault's REST API, there's reference documentation and service documentation that should help.
Using Key Vault with Python is now more easily done with the Azure SDK. There are three Python packages for working with existing vault data, and one for creating/managing vaults:
azure-keyvault-certificates (Migration guide)
azure-keyvault-keys (Migration guide)
azure-keyvault-secrets (Migration guide)
azure-mgmt-keyvault
azure-identity is also the package that should be used with these for authentication.
With the SDK, using an access token to work with an existing vault from an authorized application is as easy as creating a credential and client:
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
credential = DefaultAzureCredential()
client = SecretClient("https://{vault-name}.vault.azure.net", credential)
secret = client.get_secret("secret-name")
(I work on the Azure SDK in Python)

When Key Vault returns a 401 response, it includes a www-authenticate header containing authority and resource. You must use both to get a valid bearer token. Then you can redo your request with that token, and if you use the same token on subsequent requests against the same vault, it shouldn't return a 401 until the token expires.
You can know the authority and resource in advance, but it's generally more robust to prepare your code to always handle the 401, specially if you use multiple vaults.
Be sure to only trust on a www-authenticate header of a valid SSL connection, otherwise you might be a victim of spoofing!

I have written a simple python wrapper for the REST APIs for Azure Key Vault.
You can check out out here
AzureKeyVaultPythonSDK
Crust of the logic is here
class AzureKeyVaultManager(object):
section_name="KeyVaultSection"
# Constructor
def __init__(self, fileName="private.properties"):
prop_file=os.path.dirname(os.path.realpath(sys.argv[0])) + "/" + fileName
config = ConfigParser.RawConfigParser()
config.read(prop_file)
self.client_id=config.get(self.section_name,'client.id')
self.client_secret=config.get(self.section_name,'client.secret')
self.tenant_id=config.get(self.section_name,'tenant.id')
self.resource=config.get(self.section_name,'resource')
self.key_vault=config.get(self.section_name,'key.vault')
# Authenticate
def initialize(self):
if self.client_id and self.client_secret and self.tenant_id and self.resource and self.key_vault:
print "Got all the properties from file "
token_url="https://login.windows.net/{0}/oauth2/token".format(self.tenant_id)
payload = {'client_id':self.client_id, 'client_secret':self.client_secret, 'resource':self.resource, 'grant_type':'client_credentials'}
response=requests.post(token_url, data=payload).json()
self.access_token=response['access_token']
else:
raise ValueError("Couldn't get the key vault properties from properties file")
# Get secret from a specific keyvault
def getSecretFromKeyVault(self, secretName, keyVault=None):
if keyVault is None:
keyVault=self.key_vault
endpoint = 'https://{0}.vault.azure.net/secrets/{1}?api-version=2015-06-01'.format(keyVault, secretName)
headers = {"Authorization": 'Bearer ' + self.access_token}
response = requests.get(endpoint,headers=headers).json()
if 'value' in response:
return response['value']
else:
raise ValueError("Value not found in response")