Develop Reference

Google API refresh token is None when requesting tokens

I have followed the guide below to obtain a Google Ads API refresh token for my application.
https://github.com/googleads/googleads-python-lib/wiki/API-access-on-behalf-of-your-clients-(web-flow)
Using the script below, everything worked, but the response only had an access token, while the refresh token was None.
from googleads import oauth2
import google.oauth2.credentials
import google_auth_oauthlib.flow
# Initialize the flow using the client ID and secret downloaded earlier.
# Note: You can use the GetAPIScope helper function to retrieve the
# appropriate scope for AdWords or Ad Manager.
flow = google_auth_oauthlib.flow.Flow.from_client_secrets_file(
'client_secret.json',
[oauth2.GetAPIScope('adwords')])
# Indicate where the API server will redirect the user after the user completes
# the authorization flow. The redirect URI is required.
flow.redirect_uri = 'https://www.example.com'
# Generate URL for request to Google's OAuth 2.0 server.
# Use kwargs to set optional request parameters.
authorization_url, state = flow.authorization_url(
# Enable offline access so that you can refresh an access token without
# re-prompting the user for permission. Recommended for web server apps.
access_type='offline',
# Enable incremental authorization. Recommended as a best practice.
include_granted_scopes='true',
# approval_prompt='force'
)
print("\n" + authorization_url)
print("\nVisit the above URL and grant access. You will be redirected. Get the 'code' from the query params of the redirect URL.")
auth_code = input('\nCode: ').strip()
flow.fetch_token(code=auth_code)
credentials = flow.credentials
print(credentials.__dict__)

The problem seemed to be that I have already completed these steps before.
The solution was to include approval_prompt='force' in flow.authorization_url(). After generating the authorization_url this way, the response included a refresh token as well.

Different scopes than I requested with requests_oauthlib in Microsoft Graph

I am creating a Python script to use Microsoft Graph API services using the requests_oauthlib library. I am able to create successfully an OAuth2.0 session, get an authorization URL to open in an internet browser window to authenticate, and then I am redirected to the redirect URL that I previously indicated when I registered my app in the Azure portal (https://portal.azure.com). Then I copy the full redirect URL to paste into my application. At that point, my app reads the URL that I pasted, exchanges the authentication code that is embedded in the URL for an OAuth authentication token that is perfectly valid. To make sure, I check it in https://jwt.ms, and it is perfect except for the scopes granted. These scopes do not match the scopes that I requested in my OAuth session.
SCRIPT CODE
# details from the library can be found at https://pypi.org/project/requests-oauthlib/
from requests_oauthlib import OAuth2Session
client_id = <the client id from the Azure Portal when I registered my app>
client_secret = <the client secret I got from the Azure Portal>
redirect_uri = <the redirect_uri that I specified in the Azure Portal>
authorization_base_url = 'https://login.microsoftonline.com/<my tenant code>/oauth2/v2.0/authorize'
token_url = 'https://login.microsoftonline.com/<my tenant code>/oauth2/v2.0/token'
scopes = ["https://graph.microsoft.com/User.Read", "https://graph.microsoft.com/offline_access"]
# OAuth2.0 Authentication
msgraph = OAuth2Session(client_id, scope = scopes, redirect_uri=redirect_uri) # creates a OAuth 2.0 session object
# Redirect user to microsoft for authorization
# offline for refresh token
# force to always make user click authorize
authorization_url, state = msgraph.authorization_url(authorization_base_url, access_type="offline", prompt="select_account")
print('Please go here and authorize,', authorization_url) # user needs to click on this URL, authenticate and copy the URL that will be given
# Get the authorization verifier code from the callback url
redirect_response = input('Paste the full redirect URL here: ') # the user has to paste the url with the authorizaton code provided after authenticating
print('redirect_response: ', redirect_response)
# Fetches the access token AFTER the authentication code was given in the previous step
token = msgraph.fetch_token(token_url, client_secret=client_secret, authorization_response=redirect_response) # gets the access token
print('token: ', token)
but I get the following warning message:
Warning: Scope has changed from "https://graph.microsoft.com/User.Read https://graph.microsoft.com/offline_access" to "profile https://graph.microsoft.com/User.Read openid email".
API PERMISSIONS IN AZURE PORTAL
Microsoft Graph (2)
Files.ReadWrite.All
offline_access
As you can see in the Azure permissions above, the privileges (scopes) in the Azure portal are exactly the same scopes that I requested, so my question is where did these 'openid' and 'email' scopes come from? I have been able to overcome the warning message, but I can't request the privileges that I need. I even created a brand new application in the Azure portal, but I have the same problem. Is there something wrong with the requests_oauthlib library or I'm doing something wrong?
Thank you

When requesting scopes, you don't need a fully qualified domain name (FQDN) for Graph scopes (they're the default) and you shouldn't use them for non-Graph scopes (openid, profile, email, and offline_access are OpenID/AAD scopes, not Graph).
scopes = ["User.Read", "offline_access"]

How to get a GCP Bearer token programmatically with python

gcloud auth print-access-token gives me a Bearer token that I can use later on; however, this is a shell command. How would I obtain one programmatically via the Google Cloud Python API?
I see a prior example using oauth2client, but oauth2client is now deprecated. How would I do this with google.auth and oauthlib?

While the above answer is quite informative, it misses one important point - credentials object obtained from google.auth.default() or compute_engine.Credentials() will not have token in it. So back to the original question of what is the programmatic alternative to gcloud auth print-access-token, my answer would be:
import google.auth
import google.auth.transport.requests
creds, project = google.auth.default()
# creds.valid is False, and creds.token is None
# Need to refresh credentials to populate those
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
# Now you can use creds.token
I'm using the official google-auth package and default credentials, which will get you going both in local dev and on remote GCE/GKE app.
Too bad this is not properly documented and I had to read google-auth code to figure our how to obtain the token.

The answer depends on your environment and how you want to create / obtain credentials.
What are Google Cloud Credentials?
Google Cloud credentials are an OAuth 2.0 token. This token has at a minimum an Access Token and optionally a Refresh Token, Client ID Token, and supporting parameters such as expiration, Service Account Email or Client Email, etc.
The important item in Google Cloud APIs is the Access Token. This token is what authorizes access to the cloud. This token can be used in programs such as curl, software such as python, etc and does not require an SDK. The Access Token is used in the HTTP Authorization header.
What is an Access Token?
An access token is an opaque value generated by Google that is derived from a Signed JWT, more correctly called JWS. A JWT consists of a header and claims (the payload) Json structures. These two Json structures are signed with the Service Account's Private Key. These values are base64 encoded and concatenated to create the Access Key.
The format of an Access Token is: base64(header) + '.' + base64(payload) + '.' + base64(signature).
Here is an example JWT:
Header:
{
"alg": "RS256",
"typ": "JWT",
"kid": "42ba1e234ac91ffca687a5b5b3d0ca2d7ce0fc0a"
}
Payload:
{
"iss": "myservice#myproject.iam.gserviceaccount.com",
"iat": 1493833746,
"aud": "myservice.appspot.com",
"exp": 1493837346,
"sub": "myservice#myproject.iam.gserviceaccount.com"
}
Using an Access Token:
Example that will start a VM instance. Replace PROJECT_ID, ZONE and INSTANCE_NAME. This example is for Windows.
curl -v -X GET -H "Authorization: Bearer <access_token_here>" ^
https://www.googleapis.com/compute/v1/projects/%PROJECT_ID%/zones/%ZONE%/instances/%INSTANCE_NAME%/start
Compute Engine Service Account:
Dustin's answer is correct for this case, but I will include for completeness with some additional information.
These credentials are automatically created for you by GCP and are obtained from the VM Instance metadata. Permissions are controlled by Cloud API access scopes in the Google Console.
However, these credentials have some limitations. To modify the credentials you must stop the VM Instance first. Additionally, not all permissions (roles) are supported.
from google.auth import compute_engine
cred = compute_engine.Credentials()
Service Account Credentials:
Until you understand all of the types of credentials and their use cases, these are the credentials that you will use for everything except for gcloud and gsutil. Understanding these credentials will make working with Google Cloud much simpler when writing programs. Obtaining credentials from a Google Service Account Json file is easy. The only item to make note of is that credentials expire (typically 60 minutes) and either need to be refreshed or recreated.
gcloud auth print-access-token is NOT recommended. Service Account Credentials are the recommended method by Google.
These credentials are created by the Console, gcloud or via programs / APIs. Permissions are assigned to the creditials by IAM and function inside Compute Engine, App Engine, Firestore, Kubernetes, etc. as well as other environments outside of Google Cloud. These credentials are downloaded from Google Cloud and stored in a Json file. Notice the scopes parameter. This defines permissions that are granted to the resulting credentials object.
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = 'service-account-credentials.json'
from google.oauth2 import service_account
cred = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
Google OAuth 2.0 Credentials:
These credentials are derived from a full OAuth 2.0 flow. These credentials are generated when your browser is launched to access Google Accounts for authorizing access. This process is much more complicated and requires a fair amount of code to implement and requires a built-in web server for the callback for authorization.
This method provides additional features such as being able to run everything in a browser, example you can create a Cloud Storage File Browser, but be careful that you understand the security implications. This method is the technique used to support Google Sign-In, etc. I like to use this method to authenticate users before allowing posting on websites, etc. The possibilities are endless with correctly authorized OAuth 2.0 identities and scopes.
Example code using google_auth_oauthlib:
from google_auth_oauthlib.flow import InstalledAppFlow
flow = InstalledAppFlow.from_client_secrets_file(
'client_secrets.json',
scopes=scope)
cred = flow.run_local_server(
host='localhost',
port=8088,
authorization_prompt_message='Please visit this URL: {url}',
success_message='The auth flow is complete; you may close this window.',
open_browser=True)
Example code using the requests_oauthlib library:
from requests_oauthlib import OAuth2Session
gcp = OAuth2Session(
app.config['gcp_client_id'],
scope=scope,
redirect_uri=redirect_uri)
# print('Requesting authorization url:', authorization_base_url)
authorization_url, state = gcp.authorization_url(
authorization_base_url,
access_type="offline",
prompt="consent",
include_granted_scopes='true')
session['oauth_state'] = state
return redirect(authorization_url)
# Next section of code after the browser approves the request
token = gcp.fetch_token(
token_url,
client_secret=app.config['gcp_client_secret'],
authorization_response=request.url)

In some cases, it's not possible to set environment variables on the server or container while needing a Bearer access token to call Google cloud APIs. I present the following to solve such problem:
# pip3 install google-auth
# pip3 install requests
import google.auth
import google.auth.transport.requests
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file('/home/user/secrets/hil-test.json', scopes=['https://www.googleapis.com/auth/cloud-platform'])
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
credentials.token
The last line would print the access token for calling Google cloud APIs. Replace ya29<REDACTED> in the following curl command with the printed token from python as a test:
curl https://example.googleapis.com/v1alpha1/projects/PROJECT_ID/locations -H "Authorization: Bearer ya29<REDACTED>"
It may not make sense to execute python to get the token then curl in BASH to call an API. The purpose is to demonstrate getting the token to call Google cloud Alpha API which may not have any Python client library but REST API. Developers can then use Python requests HTTP library to call the APIs.

import google.auth
import google.auth.transport.requests
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
#getting request object
auth_req = google.auth.transport.requests.Request()
print(credentials.valid) # prints False
credentials.refresh(auth_req) #refresh token
#cehck for valid credentials
print(credentials.valid) # prints True
print(credentials.token) # prints token

This may not be the recommended way but for Rest API in my application this was an easy way to get the token.
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
token = cmdline("gcloud auth application-default print-access-token")
print("Token:"+token)

I found myself here when looking for a way to use the python SDK without creating a service account. I wanted a way to locally develop a script that would run in the cloud. I was able to achieve this by using an artifact of the gcloud command:
export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/legacy_credentials/<me>/adc.json

Merging suggestions from this post and the google cloud documentation, I wrote an auxiliary function that returns a token. It generates a token if possible, and if not takes it from the environment, then checks that it's valid.
import google
import os
import requests
GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS"
GCS_OAUTH_TOKEN = "GCS_OAUTH_TOKEN"
SCOPE = "https://www.googleapis.com/auth/cloud-platform"
URL = "https://www.googleapis.com/oauth2/v1/tokeninfo"
PAYLOAD = "access_token={}"
HEADERS = {"content-type": "application/x-www-form-urlencoded"}
OK = "OK"
def get_gcs_token():
"""
Returns gcs access token.
Ideally, this function generates a new token, requries that GOOGLE_APPLICATION_CREDENTIALS be set in the environment
(os.environ).
Alternatively, environment variable GCS_OAUTH_TOKEN could be set if a token already exists
"""
if GOOGLE_APPLICATION_CREDENTIALS in os.environ:
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=[SCOPE])
# getting request object
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req) # refresh token
token = credentials.token
elif GCS_OAUTH_TOKEN in os.environ:
token = os.environ[GCS_OAUTH_TOKEN]
else:
raise ValueError(
f"""Could not generate gcs token because {GOOGLE_APPLICATION_CREDENTIALS} is not set in the environment.
Alternatively, environment variable {GCS_OAUTH_TOKEN} could be set if a token already exists, but it was not"""
)
r = requests.post(URL, data=PAYLOAD.format(token), headers=HEADERS)
if not r.reason == OK:
raise ValueError(
f"Could not verify token {token}\n\nResponse from server:\n{r.text}"
)
if not r.json()["expires_in"] > 0:
raise ValueError(f"token {token} expired")
return token

Official documentation code example
I followed this official documentation for Cloud Functions, which works for any GCP API:
auth_req = google.auth.transport.requests.Request()
id_token = google.oauth2.id_token.fetch_id_token(
auth_req,
# This is an OAuth authorisation scope that you must pass
# depending on the API.
# You can see an example of the need for this scope here: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert#authorization-scopes
"https://www.googleapis.com/auth/bigquery"
)
Now, you can use id_token in the Authorisation header:
headers = {'Authorization': f'Bearer {id_token}'}

Azure Active Directory get token request http error

I'm using Active Directory Authentication library for python following the documentation. Earlier on I managed to get the access_token through the Acquire Token with Client Credentials sample:
import adal
RESOURCE_URI = 'https://<mydomain>.crm.dynamics.com'
AUTHORITY_URL = "https://login.microsoftonline.com/<tenant_id>"
CLIENT_ID = 'xxxx' #application_id
CLIENT_SECRET = 'xxxx'
context = adal.AuthenticationContext(AUTHORITY_URL)
token = context.acquire_token_with_client_credentials(
RESOURCE_URI,
CLIENT_ID,
CLIENT_SECRET)
print token
But I get an error message when I tried the Acquire token and Refresh token sample
context = adal.AuthenticationContext(AUTHORITY_URL)
token = context.acquire_token_with_username_password(
RESOURCE_URI,
USERNAME,
PASSWORD,
CLIENT_ID)
print token
>>> adal.adal_error.AdalError: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS70002: The request body must contain the following parameter: 'client_secret or client_assertion'.........."correlation_id"......}

adal.adal_error.AdalError: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS70002: The request body must contain the following parameter: 'client_secret or client_assertion'.........."correlation_id"......}
There are two kinds of app we can register on Azure, native or web app. Based on the error message, it seems that you have register a confident app which requires provide its client secret to acquire the access token.
For this issue please register a native app instead of web app. Also the resource owner password credentials flow should be consider used carefully since this may leak the credentials. Refer the flows from link below:
The OAuth 2.0 Authorization Framework - Authorization Grant

I suffered from the same error.
In app registration section in azure active directory I registered the app as web host/api.
When I changed it to native app everything started to work fine.

Working with the Box.com SDK for Python

I am trying to get started with the Box.com SDK and I have a few questions.
from boxsdk import OAuth2
oauth = OAuth2(
client_id='YOUR_CLIENT_ID',
client_secret='YOUR_CLIENT_SECRET',
store_tokens=your_store_tokens_callback_method,
)
auth_url, csrf_token = oauth.get_authorization_url('http://YOUR_REDIRECT_URL')
def store_tokens(access_token, refresh_token):
# store the tokens at secure storage (e.g. Keychain)
1)What is the redirect URL and how do I use it? Do I need to have a server running to use this?
2)What sort of code to I need in the store_tokens method?

The redirect URL is only required if you're runng a Web application that needs to respond to user's requests to authenticate. If you're programtically authenticating, you can simply set this as http://localhost. In a scenario where you require the user to manually authenticate, the redirect URL should invoke some function in your web app to store and process the authentication code returned. Do you need a server running? Well, if you want to do something with the authentication code returned, the URL you specify should be under your control and invoke code to do something useful.
Here's an example of what the store_tokens function should look like. It should accept two parameters, access_token and refresh_token. In the example below, the function will commit these to a local store for use when the API needs to re-authenticate:
From here:
"""An example of Box authentication with external store"""
import keyring
from boxsdk import OAuth2
from boxsdk import Client
CLIENT_ID = 'specify your Box client_id here'
CLIENT_SECRET = 'specify your Box client_secret here'
def read_tokens():
"""Reads authorisation tokens from keyring"""
# Use keyring to read the tokens
auth_token = keyring.get_password('Box_Auth', 'mybox#box.com')
refresh_token = keyring.get_password('Box_Refresh', 'mybox#box.com')
return auth_token, refresh_token
def store_tokens(access_token, refresh_token):
"""Callback function when Box SDK refreshes tokens"""
# Use keyring to store the tokens
keyring.set_password('Box_Auth', 'mybox#box.com', access_token)
keyring.set_password('Box_Refresh', 'mybox#box.com', refresh_token)
def main():
"""Authentication against Box Example"""
# Retrieve tokens from secure store
access_token, refresh_token = read_tokens()
# Set up authorisation using the tokens we've retrieved
oauth = OAuth2(
client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
access_token=access_token,
refresh_token=refresh_token,
store_tokens=store_tokens,
)
# Create the SDK client
client = Client(oauth)
# Get current user details and display
current_user = client.user(user_id='me').get()
print('Box User:', current_user.name)
if __name__ == '__main__':
main()

I suggest taking a look at the OAuth 2 tutorial. It will help give a better understanding of how OAuth works and what the various parameters are used for.
The redirect URL is set in your Box application's settings:
This is the URL where Box will send an auth code that can be used to obtain an access token. For example, if your redirect URL is set to https://myhost.com, then your server will receive a request with a URL that looks something like https://myhost.com?code=123456abcdef.
Note that your redirect URI doesn't need to be a real server. For example, apps that use a WebView will sometimes enter a fake redirect URL and then extract the auth code directly from the URL in the WebView.
The store_tokens callback is optional, but it can be used to save the access and refresh tokens in case your application needs to shutdown. It will be invoked every time the access token and refresh token changes, giving you an opportunity to save them somewhere (to disk, a DB, etc.).
You can then pass in these tokens to your OAuth2 constructor at a later time so that your users don't need to login again.

If you're just testing, you can also pass in a developer token. This tutorial explains how.
This is the most basic example that worked for me:
from boxsdk import Client, OAuth2
CLIENT_ID = ''
CLIENT_SECRET = ''
ACCESS_TOKEN = '' # this is the developer token
oauth2 = OAuth2(CLIENT_ID, CLIENT_SECRET, access_token=ACCESS_TOKEN)
client = Client(oauth2)
my = client.user(user_id='me').get()
print(my.name)
print(my.login)
print(my.avatar_url)