I have followed the guide below to obtain a Google Ads API refresh token for my application.
https://github.com/googleads/googleads-python-lib/wiki/API-access-on-behalf-of-your-clients-(web-flow)
Using the script below, everything worked, but the response only had an access token, while the refresh token was None.
from googleads import oauth2
import google.oauth2.credentials
import google_auth_oauthlib.flow
# Initialize the flow using the client ID and secret downloaded earlier.
# Note: You can use the GetAPIScope helper function to retrieve the
# appropriate scope for AdWords or Ad Manager.
flow = google_auth_oauthlib.flow.Flow.from_client_secrets_file(
'client_secret.json',
[oauth2.GetAPIScope('adwords')])
# Indicate where the API server will redirect the user after the user completes
# the authorization flow. The redirect URI is required.
flow.redirect_uri = 'https://www.example.com'
# Generate URL for request to Google's OAuth 2.0 server.
# Use kwargs to set optional request parameters.
authorization_url, state = flow.authorization_url(
# Enable offline access so that you can refresh an access token without
# re-prompting the user for permission. Recommended for web server apps.
access_type='offline',
# Enable incremental authorization. Recommended as a best practice.
include_granted_scopes='true',
# approval_prompt='force'
)
print("\n" + authorization_url)
print("\nVisit the above URL and grant access. You will be redirected. Get the 'code' from the query params of the redirect URL.")
auth_code = input('\nCode: ').strip()
flow.fetch_token(code=auth_code)
credentials = flow.credentials
print(credentials.__dict__)
The problem seemed to be that I have already completed these steps before.
The solution was to include approval_prompt='force' in flow.authorization_url(). After generating the authorization_url this way, the response included a refresh token as well.
I am trying to convert the following command from CLI (that works) to python but I am having some problems.
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" SERVICE_URL
The problem is that I cannot request a valid Bearer with application default local credentials token to make authorized request to Google Cloud Run. If I generate Bearer token from CLI gcloud auth print-identity-token and use it in a python request all works fine
request_url = 'https://<my endpoint>'
identity_token = '<>' # response of gcloud auth print-identity-token)
header= {'Authorization': f'Bearer {identity_token}'}
requests.get(url=request_url, headers=receiving_service_headers)
From google auth documentation I understood that Cloud Run communicationis based on Identity Tokens that support Impersonated authentication but I cannot generate valid credential.
from google.auth import impersonated_credentials, default
from google.auth.transport.requests import AuthorizedSession
request_url = 'https://<my endpoint>'
source_credentials, project = default()
creds = impersonated_credentials.IDTokenCredentials(
source_credentials,
target_audience=request_url)
authed_session = AuthorizedSession(creds)
resp = authed_session.get(request_url)
print(resp)
bui I get following error
google.auth.exceptions.GoogleAuthError: Provided Credential must be impersonated_credentials
Thanks
Today it's impossible. I talked about it in an article. You need a service account to generate a valid JWT token with the Google Auth library. It's the same problem with all the library and all the languages.
Last week, I pushed a merge request in the Java auth library to solve this. I don't know why Google don't implement it by itself.
On your local environment, if you want to use the same code locally and in the cloud, you have to generate a service account key file and use it with ADC. And it's sadly a security issue...
Please follow the enter link if you want to convert the curl command to python request. Here is converted code:
import requests
headers = {
'Authorization': 'Bearer $(gcloud auth print-identity-token)',
}
response = requests.get('http://SERVICE_URL', headers=headers)
I have a cloud function with trigger type set to HTTP and also have a service account which is having permissions to Invoke the cloud function. I want to invoke the cloud function from a python script. I am using the following script to invoke the function:
from google.oauth2 import service_account
from google.auth.transport.urllib3 import AuthorizedHttp
credentials = service_account.Credentials.from_service_account_file('/path/to/service-account-credentials.json')
scoped_credentials = credentials.with_scopes(['https://www.googleapis.com/auth/cloud-platform'])
authed_http = AuthorizedHttp(scoped_credentials)
response = authed_http.request('GET', 'https://test-123456.cloudfunctions.net/my-cloud-function')
print(response.status)
I am getting Unauthorized ( 401 ) error in response. Is this the correct way of invocation?
To be able to call your cloud function you need an ID Token against a Cloud Functions end point
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
url = 'https://test-123456.cloudfunctions.net/my-cloud-function'
creds = service_account.IDTokenCredentials.from_service_account_file(
'/path/to/service-account-credentials.json', target_audience=url)
authed_session = AuthorizedSession(creds)
# make authenticated request and print the response, status_code
resp = authed_session.get(url)
print(resp.status_code)
print(resp.text)
gcloud auth print-access-token gives me a Bearer token that I can use later on; however, this is a shell command. How would I obtain one programmatically via the Google Cloud Python API?
I see a prior example using oauth2client, but oauth2client is now deprecated. How would I do this with google.auth and oauthlib?
While the above answer is quite informative, it misses one important point - credentials object obtained from google.auth.default() or compute_engine.Credentials() will not have token in it. So back to the original question of what is the programmatic alternative to gcloud auth print-access-token, my answer would be:
import google.auth
import google.auth.transport.requests
creds, project = google.auth.default()
# creds.valid is False, and creds.token is None
# Need to refresh credentials to populate those
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
# Now you can use creds.token
I'm using the official google-auth package and default credentials, which will get you going both in local dev and on remote GCE/GKE app.
Too bad this is not properly documented and I had to read google-auth code to figure our how to obtain the token.
The answer depends on your environment and how you want to create / obtain credentials.
What are Google Cloud Credentials?
Google Cloud credentials are an OAuth 2.0 token. This token has at a minimum an Access Token and optionally a Refresh Token, Client ID Token, and supporting parameters such as expiration, Service Account Email or Client Email, etc.
The important item in Google Cloud APIs is the Access Token. This token is what authorizes access to the cloud. This token can be used in programs such as curl, software such as python, etc and does not require an SDK. The Access Token is used in the HTTP Authorization header.
What is an Access Token?
An access token is an opaque value generated by Google that is derived from a Signed JWT, more correctly called JWS. A JWT consists of a header and claims (the payload) Json structures. These two Json structures are signed with the Service Account's Private Key. These values are base64 encoded and concatenated to create the Access Key.
The format of an Access Token is: base64(header) + '.' + base64(payload) + '.' + base64(signature).
Here is an example JWT:
Header:
{
"alg": "RS256",
"typ": "JWT",
"kid": "42ba1e234ac91ffca687a5b5b3d0ca2d7ce0fc0a"
}
Payload:
{
"iss": "myservice#myproject.iam.gserviceaccount.com",
"iat": 1493833746,
"aud": "myservice.appspot.com",
"exp": 1493837346,
"sub": "myservice#myproject.iam.gserviceaccount.com"
}
Using an Access Token:
Example that will start a VM instance. Replace PROJECT_ID, ZONE and INSTANCE_NAME. This example is for Windows.
curl -v -X GET -H "Authorization: Bearer <access_token_here>" ^
https://www.googleapis.com/compute/v1/projects/%PROJECT_ID%/zones/%ZONE%/instances/%INSTANCE_NAME%/start
Compute Engine Service Account:
Dustin's answer is correct for this case, but I will include for completeness with some additional information.
These credentials are automatically created for you by GCP and are obtained from the VM Instance metadata. Permissions are controlled by Cloud API access scopes in the Google Console.
However, these credentials have some limitations. To modify the credentials you must stop the VM Instance first. Additionally, not all permissions (roles) are supported.
from google.auth import compute_engine
cred = compute_engine.Credentials()
Service Account Credentials:
Until you understand all of the types of credentials and their use cases, these are the credentials that you will use for everything except for gcloud and gsutil. Understanding these credentials will make working with Google Cloud much simpler when writing programs. Obtaining credentials from a Google Service Account Json file is easy. The only item to make note of is that credentials expire (typically 60 minutes) and either need to be refreshed or recreated.
gcloud auth print-access-token is NOT recommended. Service Account Credentials are the recommended method by Google.
These credentials are created by the Console, gcloud or via programs / APIs. Permissions are assigned to the creditials by IAM and function inside Compute Engine, App Engine, Firestore, Kubernetes, etc. as well as other environments outside of Google Cloud. These credentials are downloaded from Google Cloud and stored in a Json file. Notice the scopes parameter. This defines permissions that are granted to the resulting credentials object.
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = 'service-account-credentials.json'
from google.oauth2 import service_account
cred = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
Google OAuth 2.0 Credentials:
These credentials are derived from a full OAuth 2.0 flow. These credentials are generated when your browser is launched to access Google Accounts for authorizing access. This process is much more complicated and requires a fair amount of code to implement and requires a built-in web server for the callback for authorization.
This method provides additional features such as being able to run everything in a browser, example you can create a Cloud Storage File Browser, but be careful that you understand the security implications. This method is the technique used to support Google Sign-In, etc. I like to use this method to authenticate users before allowing posting on websites, etc. The possibilities are endless with correctly authorized OAuth 2.0 identities and scopes.
Example code using google_auth_oauthlib:
from google_auth_oauthlib.flow import InstalledAppFlow
flow = InstalledAppFlow.from_client_secrets_file(
'client_secrets.json',
scopes=scope)
cred = flow.run_local_server(
host='localhost',
port=8088,
authorization_prompt_message='Please visit this URL: {url}',
success_message='The auth flow is complete; you may close this window.',
open_browser=True)
Example code using the requests_oauthlib library:
from requests_oauthlib import OAuth2Session
gcp = OAuth2Session(
app.config['gcp_client_id'],
scope=scope,
redirect_uri=redirect_uri)
# print('Requesting authorization url:', authorization_base_url)
authorization_url, state = gcp.authorization_url(
authorization_base_url,
access_type="offline",
prompt="consent",
include_granted_scopes='true')
session['oauth_state'] = state
return redirect(authorization_url)
# Next section of code after the browser approves the request
token = gcp.fetch_token(
token_url,
client_secret=app.config['gcp_client_secret'],
authorization_response=request.url)
In some cases, it's not possible to set environment variables on the server or container while needing a Bearer access token to call Google cloud APIs. I present the following to solve such problem:
# pip3 install google-auth
# pip3 install requests
import google.auth
import google.auth.transport.requests
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file('/home/user/secrets/hil-test.json', scopes=['https://www.googleapis.com/auth/cloud-platform'])
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
credentials.token
The last line would print the access token for calling Google cloud APIs. Replace ya29<REDACTED> in the following curl command with the printed token from python as a test:
curl https://example.googleapis.com/v1alpha1/projects/PROJECT_ID/locations -H "Authorization: Bearer ya29<REDACTED>"
It may not make sense to execute python to get the token then curl in BASH to call an API. The purpose is to demonstrate getting the token to call Google cloud Alpha API which may not have any Python client library but REST API. Developers can then use Python requests HTTP library to call the APIs.
import google.auth
import google.auth.transport.requests
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
#getting request object
auth_req = google.auth.transport.requests.Request()
print(credentials.valid) # prints False
credentials.refresh(auth_req) #refresh token
#cehck for valid credentials
print(credentials.valid) # prints True
print(credentials.token) # prints token
This may not be the recommended way but for Rest API in my application this was an easy way to get the token.
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
token = cmdline("gcloud auth application-default print-access-token")
print("Token:"+token)
I found myself here when looking for a way to use the python SDK without creating a service account. I wanted a way to locally develop a script that would run in the cloud. I was able to achieve this by using an artifact of the gcloud command:
export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/legacy_credentials/<me>/adc.json
Merging suggestions from this post and the google cloud documentation, I wrote an auxiliary function that returns a token. It generates a token if possible, and if not takes it from the environment, then checks that it's valid.
import google
import os
import requests
GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS"
GCS_OAUTH_TOKEN = "GCS_OAUTH_TOKEN"
SCOPE = "https://www.googleapis.com/auth/cloud-platform"
URL = "https://www.googleapis.com/oauth2/v1/tokeninfo"
PAYLOAD = "access_token={}"
HEADERS = {"content-type": "application/x-www-form-urlencoded"}
OK = "OK"
def get_gcs_token():
"""
Returns gcs access token.
Ideally, this function generates a new token, requries that GOOGLE_APPLICATION_CREDENTIALS be set in the environment
(os.environ).
Alternatively, environment variable GCS_OAUTH_TOKEN could be set if a token already exists
"""
if GOOGLE_APPLICATION_CREDENTIALS in os.environ:
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=[SCOPE])
# getting request object
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req) # refresh token
token = credentials.token
elif GCS_OAUTH_TOKEN in os.environ:
token = os.environ[GCS_OAUTH_TOKEN]
else:
raise ValueError(
f"""Could not generate gcs token because {GOOGLE_APPLICATION_CREDENTIALS} is not set in the environment.
Alternatively, environment variable {GCS_OAUTH_TOKEN} could be set if a token already exists, but it was not"""
)
r = requests.post(URL, data=PAYLOAD.format(token), headers=HEADERS)
if not r.reason == OK:
raise ValueError(
f"Could not verify token {token}\n\nResponse from server:\n{r.text}"
)
if not r.json()["expires_in"] > 0:
raise ValueError(f"token {token} expired")
return token
Official documentation code example
I followed this official documentation for Cloud Functions, which works for any GCP API:
auth_req = google.auth.transport.requests.Request()
id_token = google.oauth2.id_token.fetch_id_token(
auth_req,
# This is an OAuth authorisation scope that you must pass
# depending on the API.
# You can see an example of the need for this scope here: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert#authorization-scopes
"https://www.googleapis.com/auth/bigquery"
)
Now, you can use id_token in the Authorisation header:
headers = {'Authorization': f'Bearer {id_token}'}
So i got a refresh token in this way and can I keep it?
And if so, how do I use it next time, so that there is no need for me to open browser?
Right now I'm thinking about creating OAuth2Credentials object directly, is this the right way?
from urllib.parse import urlparse, parse_qs
from oauth2client.client import flow_from_clientsecrets, OAuth2Credentials
from oauth2client.file import Storage
from oauth2client.tools import argparser, run_flow
from apiclient.discovery import build
from apiclient.errors import HttpError
from oauth2client.contrib import gce
import httplib2
import webbrowser
CLIENT_SECRETS_FILE = "bot_credentials.json"
flow = client.flow_from_clientsecrets(
CLIENT_SECRETS_FILE,
scope=scope,
redirect_uri='http://127.0.0.1:65010')
flow.params['include_granted_scopes'] = 'true'
flow.params['access_type'] = 'offline'
auth_uri = flow.step1_get_authorize_url()
webbrowser.open(auth_uri)
url = input('Please enter the redirected url with code')
code = get_url_param(url, 'code')
if code is None:
print('there is an error in your redirect link with code parameter, check if it exists')
exit()
print(code)
credentials = flow.step2_exchange(code[0])
print(credentials.to_json())#refresh_token here!!!
If the user consents to authorize your application to access those resources, Google will return a token to your application. Depending on your application's type, it will either validate the token or exchange it for a different type of token. Check this documentation.
For example, a server-side web application would exchange the returned token for an access token and a refresh token. The access token would let the application authorize requests on the user's behalf, and the refresh token would let the application retrieve a new access token when the original access token expires.
Basically, if your application obtains a refresh token during the authorization process, then you will need to periodically use that token to obtain a new, valid access token. Server-side web applications, installed applications, and devices all obtain refresh tokens.
It is stated here that at any time, your application can send a POST request to Google's authorization server that specifies your client ID, your client secret, and the refresh token for the user. The request should also set the grant_type parameter value to refresh_token.
The following example demonstrates this request:
POST /o/oauth2/token HTTP/1.1
Host: accounts.google.com
Content-Type: application/x-www-form-urlencoded
client_id=21302922996.apps.googleusercontent.com&
client_secret=XTHhXh1SlUNgvyWGwDk1EjXB&
refresh_token=1/6BMfW9j53gdGImsixUH6kU5RsR4zwI9lUVX-tqf8JXQ&
grant_type=refresh_token
The authorization server will return a JSON object that contains a new access token:
{
"access_token":"1/fFAGRNJru1FTz70BzhT3Zg",
"expires_in":3920,
"token_type":"Bearer"
}
You can check this sample on GitHub to generate a refresh token for the YouTube API. Note that this will will also create a file called generate_token.py-oauth that contains this information.