Develop Reference

google.auth.exceptions.RefresError: 'No access token in response.'

Environment details
OS type and version:
Python version: 3.9.0
pip version: 22.0.4
google-api-python-client version: 2.48.0
Description
Hi, I'm running into an error when trying to fetch the Google Play Console reports of our mobile apps (such as installations, errors etc.). I first tried with this manual but it seems to be outdated and didn't work. So after some research I changed it similar to this one, that it fits to the current google api (see code snippet below).
Steps I have done:
Created a project on "console.cloud.google.com"
Created the service account
Created the json key file
Invited the service account on play.google.com/console and gave him full admin rights (normally "see app information and download bulk reports" should be enough)
Added the role "Storage Object Viewer" to the Service account in https://console.cloud.google.com/iam-admin/iam?authuser=1&project=myproject
waited for 24h to make sure there are no errors because of syncs or so.
(I anonymized some of the values below).
Code example
from googleapiclient.discovery import build
from google.oauth2 import service_account
scopes = ['https://www.googleapis.com/auth/devstorage.read_only','https://www.googleapis.com/auth/cloud-platform.read_only']
key_file_location = 'files/access_token/mykeyfile.json'
cloud_storage_bucket = r'pubsite_prod_rev_00123456789'
report_to_download = 'installs/installs_com.my.app_202201_country.csv'
creds = service_account.Credentials.from_service_account_file(key_file_location,scopes=scopes)
service = build('storage','v1', credentials=creds)
print(service.objects().get(bucket = cloud_storage_bucket, object= report_to_download).execute())
Stack trace
Traceback (most recent call last):
File "C:\Users\myuser\project\z_10_ext_google_play_store.py", line 46, in <module>
print(service.objects().get(bucket = cloud_storage_bucket, object= report_to_download).execute())
File "D:\Programs\Python\lib\site-packages\googleapiclient\_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "D:\Programs\Python\lib\site-packages\googleapiclient\http.py", line 923, in execute
resp, content = _retry_request(
File "D:\Programs\Python\lib\site-packages\googleapiclient\http.py", line 191, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "D:\Programs\Python\lib\site-packages\google_auth_httplib2.py", line 209, in request
self.credentials.before_request(self._request, method, uri, request_headers)
File "D:\Programs\Python\lib\site-packages\google\auth\credentials.py", line 133, in before_request
self.refresh(request)
File "D:\Programs\Python\lib\site-packages\google\oauth2\service_account.py", line 410, in refresh
access_token, expiry, _ = _client.jwt_grant(
File "D:\Programs\Python\lib\site-packages\google\oauth2\_client.py", line 199, in jwt_grant
six.raise_from(new_exc, caught_exc)
File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: ('No access token in response.', {'id_token': 'eyJ...'})
I hope that I provided enough information and I'm sorry in advance if I made a stupid mistake.

How to read data from Google Sheets in Python

I have a Google Sheet https://docs.google.com/spreadsheets/d/1Ycg7zTxds9DZnDvTrFcyNNKuTUxg6Yy6WF0a8Wc02WQ/edit#gid=0 I can read only. I need to read it in Python. I uploaded a video of what I am doing https://youtu.be/4cK0g9Bm_7w
import gspread
from oauth2client.service_account import ServiceAccountCredentials
scope = ["https://docs.google.com/spreadsheets/d/1Ycg7zTxds9DZnDvTrFcyNNKuTUxg6Yy6WF0a8Wc02WQ/edit#gid=0",
"https://docs.google.com/spreadsheets/d/1Ycg7zTxds9DZnDvTrFcyNNKuTUxg6Yy6WF0a8Wc02WQ/edit?usp=sharing"]
creds = ServiceAccountCredentials.from_json_keyfile_name("creds.json", scope)
client = gspread.authorize(creds)
sheet = client.open("[XO] developer test data").sheet1
I have an error:
Traceback (most recent call last):
File "C:\Python379\lib\site-packages\google\oauth2\_client.py", line 196, in jwt_grant
access_token = response_data["access_token"]
KeyError: 'access_token'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "E:/Прогрммирование/Последние алгоритмы/Git/GoogleSheets/sheets.py", line 11, in <module>
sheet = client.open("[XO] developer test data").sheet1
File "C:\Python379\lib\site-packages\gspread\client.py", line 121, in open
self.list_spreadsheet_files(title),
File "C:\Python379\lib\site-packages\gspread\client.py", line 98, in list_spreadsheet_files
res = self.request('get', url, params=params).json()
File "C:\Python379\lib\site-packages\gspread\client.py", line 70, in request
headers=headers,
File "C:\Python379\lib\site-packages\requests\sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "C:\Python379\lib\site-packages\google\auth\transport\requests.py", line 478, in request
self.credentials.before_request(auth_request, method, url, request_headers)
File "C:\Python379\lib\site-packages\google\auth\credentials.py", line 133, in before_request
self.refresh(request)
File "C:\Python379\lib\site-packages\google\oauth2\service_account.py", line 377, in refresh
request, self._token_uri, assertion
File "C:\Python379\lib\site-packages\google\oauth2\_client.py", line 199, in jwt_grant
six.raise_from(new_exc, caught_exc)
File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: ('No access token in response.', {'id_token': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjE5ZmUyYTdiNjc5NTIzOTYwNmNhMGE3NTA3OTRhN2JkOWZkOTU5NjEiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOiJodHRwczovL2RvY3MuZ29vZ2xlLmNvbS9zcHJlYWRzaGVldHMvZC8xWWNnN3pUeGRzOURabkR2VHJGY3lOTkt1VFV4ZzZZeTZXRjBhOFdjMDJXUS9lZGl0I2dpZD0wLGh0dHBzOi8vZG9jcy5nb29nbGUuY29tL3NwcmVhZHNoZWV0cy9kLzFZY2c3elR4ZHM5RFpuRHZUckZjeU5OS3VUVXhnNll5NldGMGE4V2MwMldRL2VkaXQ_dXNwPXNoYXJpbmciLCJhenAiOiJhY2MxMC0yMjhAcHJvamVjdC0xMC0zMTc0MDIuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iLCJlbWFpbCI6ImFjYzEwLTIyOEBwcm9qZWN0LTEwLTMxNzQwMi5pYW0uZ3NlcnZpY2VhY2NvdW50LmNvbSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJleHAiOjE2MjQxNjA4NTIsImlhdCI6MTYyNDE1NzI1MiwiaXNzIjoiaHR0cHM6Ly9hY2NvdW50cy5nb29nbGUuY29tIiwic3ViIjoiMTA4OTUzMjg5ODAwOTQ4MzEyOTgxIn0.RYEt9eTvaCXSRPH-ddjB4s5wCE60tt2w-ZgL2aaeBT8ZTyKtlDbpMKV1zG8XRkxqCLs_xde-TEwTZoRQ8OKuJGQIZr6U8mSmqjW7xlhz5cvDLL03o-muks06FbLsgAFE2yEow5EdFcEL8hKt8LGuFoQWuiNWNT-sd-Z-vJMG0-XC6twrYgYp59tdmxCCG3MEprKbtgHpwAWHPoWjybKkETGXVnv17DSV99LIXY9FIzQVV4FV4reBn_4sgcT4y6IcbHqe6hpt4joEDYBsVxImzKk7IJjJfsRCg-kGpfc1yOC831qbupMlF45H75XMq9WWmnrn3Su4YFIBTugUxrMpdA'})

Issues:
You are not providing valid scopes. A scope is not the URL of a file you want to access. A scope determines what actions an API client can make, like reading user's spreadsheets, or reading and writing user's spreadsheets; it doesn't refer to specific files. Here you can see a list of the existing scopes in Sheets API: Authorizing requests with OAuth 2.0.
oauth2client is deprecated. Consider using google-auth instead.
Code sample:
import gspread
from google.oauth2.service_account import Credentials
scopes = [
'https://www.googleapis.com/auth/spreadsheets' # Change according to your preferences
]
credentials = Credentials.from_service_account_file(
'creds.json',
scopes=scopes
)
client = gspread.authorize(creds)
sheet = client.open("[XO] developer test data").sheet1
Notes:
gspread is not officially supported by Google. Here is the official library. And here is the corresponding quickstart for Sheets API.
I am assuming that creds.json refers to valid service account credentials.
If you store creds.json to ~/.config/gspread/creds.json, you should be able to do this, according to the gspread documentation:
import gspread
client = gspread.service_account()
sheet = client.open("[XO] developer test data").sheet1
Reference:
google.oauth2.service_account module

Unable to authenticate to ADLS using python due to AuthorizationPermissionMismatch

I am using below python authentication script to connect to ADLS using service principal details but it keeps throwing exception:azure.core.exceptions.HttpResponseError: (AuthorizationPermissionMismatch) This request is not authorized to perform this operation using this permission.
The role assigned to me is "Storage blob data owner" and not sure what is missing?
Python Code:
from azure.storage.filedatalake import DataLakeServiceClient
from azure.identity import ClientSecretCredential
TENANT_ID = 'XXXXXXXXXX'
CLIENT_ID = 'XXXXXXXXXX'
CLIENT_SECRET = 'XXXXXXXXXX'
STORAGE_ACCOUNT_NAME = 'XXXXXXXXXX'
credential = ClientSecretCredential(TENANT_ID, CLIENT_ID, CLIENT_SECRET)
service_client = DataLakeServiceClient(account_url="{}://{}.dfs.core.windows.net".format(
"https", STORAGE_ACCOUNT_NAME), credential=credential)
print(service_client.primary_endpoint) # Can see the primary endpoint.
file_system_client = service_client.get_file_system_client("my-container")
file_system_client.create_directory("test-dir") #Throwing the (AuthorizationPermissionMismatch) error.
print("test directory created.")
Complete Trace:
Traceback (most recent call last):
File "/home//lib/python3.5/site-packages/azure/storage/filedatalake/_path_client.py", line 200, in _create
return self._client.path.create(**options)
File "/home/lib/python3.5/site-packages/azure/storage/filedatalake/_generated/operations/_path_operations.py", line 248, in create
raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: (AuthorizationPermissionMismatch) This request is not authorized to perform this operation using this permission.
RequestId:80605399-e01f-0038-2cd2-0a4210000000
Time:2021-02-24T17:25:49.0969802Z
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "adls_client.py", line 30, in <module>
file_system_client.create_directory("test-dir")
File "/home/lib/python3.5/site-packages/azure/storage/filedatalake/_file_system_client.py", line 540, in create_directory
directory_client.create_directory(metadata=metadata, **kwargs)
File "/home/lib/python3.5/site-packages/azure/storage/filedatalake/_data_lake_directory_client.py", line 160, in create_directory
return self._create('directory', metadata=metadata, **kwargs)
File "/home/lib/python3.5/site-packages/azure/storage/filedatalake/_path_client.py", line 202, in _create
process_storage_error(error)
File "/home/lib/python3.5/site-packages/azure/storage/filedatalake/_deserialize.py", line 150, in process_storage_error
raise error
azure.core.exceptions.HttpResponseError: (AuthorizationPermissionMismatch) This request is not authorized to perform this operation using this permission.

I can reproduce your error:
I am pretty sure your code is no problem, and, Storage blob data owner is the right RBAC role of your AD app. I think maybe the problem comes from the RBAC role does not take effect immediately, you need to wait a while. And then it should work.

Trouble validating JWT access token in python cloud function

I am trying to add authorization to my python Cloud Functions. I created a service account in the GCP project and generated keys. The test client code (not in GCP) to call the Cloud Function looks like this:
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
SERVICE_ACCOUNT_FILE = '<my_project_key_file>.json'
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE,
scopes=['https://www.googleapis.com/auth/userinfo.email'])
authed_session = AuthorizedSession(credentials)
response = authed_session.get('https://<my_project>.cloudfunctions.net/authValidation')
I know this code correctly gets the JWT bearer token from Google and is added to the Authorization header in the call to my Cloud Function. I'm just having a hard time validating that token in the Cloud Function. The relevant part of that code looks like this:
from google.oauth2 import id_token
from google.auth.transport import requests
def hello_world(request):
# from https://developers.google.com/identity/sign-in/web/backend-auth#using-a-google-api-client-library
idinfo = id_token.verify_oauth2_token(request.headers.get('Authorization')[7:]), requests.Request())
I know the id token is correct because the manual validation (using https://www.googleapis.com/oauth2/v1/tokeninfo?access_token=xxx ) returns exactly what I would expect.
The error logging stacktrace I get is:
Traceback (most recent call last):
File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 346, in run_http_function
result = _function_handler.invoke_user_function(flask.request)
File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 217, in invoke_user_function
return call_user_function(request_or_event)
File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 210, in call_user_function
return self._user_function(request_or_event)
File "/user_code/main.py", line 17, in hello_world
idinfo = id_token.verify_oauth2_token(request.headers.get('Authorization')[7:], requests.Request())
File "/env/local/lib/python3.7/site-packages/google/oauth2/id_token.py", line 141, in verify_oauth2_token
certs_url=_GOOGLE_OAUTH2_CERTS_URL)
File "/env/local/lib/python3.7/site-packages/google/oauth2/id_token.py", line 122, in verify_token
return jwt.decode(id_token, certs=certs, audience=audience)
File "/env/local/lib/python3.7/site-packages/google/auth/jwt.py", line 219, in decode
header, payload, signed_section, signature = _unverified_decode(token)
File "/env/local/lib/python3.7/site-packages/google/auth/jwt.py", line 139, in _unverified_decode
header = _decode_jwt_segment(encoded_header)
File "/env/local/lib/python3.7/site-packages/google/auth/jwt.py", line 112, in _decode_jwt_segment
six.raise_from(new_exc, caught_exc)
File "<string>", line 3, in raise_from
ValueError: Can't parse segment: b'\xc9\xad\xbd'
What am I missing here? Thanks

By setting the GOOGLE_APPLICATION_CREDENTIAL environment variable in your local system, your client will run under the context of that service account without having to worry about auth. You don't need to code the path to the keyfile.
Also applies to deploying the Cloud Function, and testing it locally. When you deploy a Cloud Function, it runs as the AppEngine default service account, or the service account you specify with the --service-account parameter:
https://cloud.google.com/sdk/gcloud/reference/functions/deploy
Reference:
https://cloud.google.com/docs/authentication/production
This way you don't need to push the key to the server or worry about it in git, and you also don't need to make any code changes while running locally vs remotely.

Make sure the string passed to id_token.verify_oauth2_token() does not have the "Bearer " still at the start of it.

Authenticate a python application in VM with Managed Service Identity(MSI)

I am trying to use MSI example provided in below link :
https://learn.microsoft.com/en-us/python/azure/python-sdk-azure-authenticate?view=azure-python#mgmt-auth-msi
To do that, I created a linux VM , installed MSI extension on it and running above code in a python application and when I run that python application I get the following error:
[azureuser#vish-redhat ~]$ python msi-auth.py
No handlers could be found for logger "msrestazure.azure_active_directory"
Traceback (most recent call last):
File "msi-auth.py", line 10, in <module>
subscription = next(subscription_client.subscriptions.list())
File "/usr/lib/python2.7/site-packages/msrest/paging.py", line 121, in __next__
self.advance_page()
File "/usr/lib/python2.7/site-packages/msrest/paging.py", line 107, in advance_page
self._response = self._get_next(self.next_link)
File "/usr/lib/python2.7/site-packages/azure/mgmt/resource/subscriptions/v2016_06_01/operations/subscriptions_operations.py", line 207, in internal_paging
request, header_parameters, **operation_config)
File "/usr/lib/python2.7/site-packages/msrest/service_client.py", line 191, in send
session = self.creds.signed_session()
File "/usr/lib/python2.7/site-packages/msrestazure/azure_active_directory.py", line 685, in signed_session
self.set_token()
File "/usr/lib/python2.7/site-packages/msrestazure/azure_active_directory.py", line 681, in set_token
self.scheme, _, self.token = get_msi_token(self.resource, self.port, self.msi_conf)
File "/usr/lib/python2.7/site-packages/msrestazure/azure_active_directory.py", line 590, in get_msi_token
result = requests.post(request_uri, data=payload, headers={'Metadata': 'true'})
File "/usr/lib/python2.7/site-packages/requests/api.py", line 108, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(111, 'Connection refused'))
[azureuser#vish-redhat ~]$
Code:
from msrestazure.azure_active_directory import MSIAuthentication
from azure.mgmt.resource import ResourceManagementClient, SubscriptionClient
# Create MSI Authentication
credentials = MSIAuthentication()
# Create a Subscription Client
subscription_client = SubscriptionClient(credentials)
subscription = next(subscription_client.subscriptions.list())
subscription_id = subscription.subscription_id
# Create a Resource Management client
resource_client = ResourceManagementClient(credentials, subscription_id)
# List resource groups as an example. The only limit is what role and policy are assigned to this MSI token.
for resource_group in resource_client.resource_groups.list():
print(resource_group.name)

You need install Python SDK in your Linux VM. Please refer to this official document.
pip install azure
Also, you need give Owner role for your VM on subscription level.
More information about this please refer to this link.
Now, you could use this code to test on VM. I test in my lab, it works for me.
Note: You need modify resource_client = ResourceManagementClient(credentials, subscription_id) to resource_client = ResourceManagementClient(credentials, str(subscription_id)), it requires a string type.

A connection error is usually because the extension is not yet available. You can try if the extension is available using the CLI with az login --msi
https://learn.microsoft.com/en-us/azure/active-directory/managed-service-identity/how-to-use-vm-sign-in
If it works, your VM is created correctly with MSI support. It it doesn't, probably your extension is not configured correctly.
Note that we changed the way to get a token with MSI from inside a VM. We now use IMDS:
https://learn.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service
Starting with the next release of the CLI (the first one of April 2018), CLI will authenticate with IMDS directly and not use the VM extension anymore. This is already shipped in the underlying library msrestazure in its 0.4.25 version. This one will bypass completely your VM extension to use IMDS and is the prefered scenario now. Could you try with this version of msrestazure? If it works with 0.4.25 but not in 0.4.24, this likely means your VM extension is not installed correctly, but you don't care since it's a deprecated scenario :)
Note that in order to get a token, your VM doesn't need any special permissions or ownership of subscription. However, for this token to be useful you need it :). But since your error is related to the "get a token" part and not permission, I would just kindly suggest that you might need this complementary info for later if you have permissions issues:
https://learn.microsoft.com/en-us/azure/active-directory/managed-service-identity/howto-assign-access-cli
(full disclosure, I work at MS in the SDK/CLI team and wrote the MSI support)