Develop Reference

GSpread pass credentials from Python not JSON

Im using GSpread trying to pass the content on my JSON file (Google API Service Application credentials) as a python Dictionary on my script. Im trying to not to carry a json file wherever I take my script.
I get the following error when I tried to pass a dictionary instead of a json file on the following line:
credentials = ServiceAccountCredentials.from_json_keyfile_name(auth_gdrive(), scope)
TypeError: expected str, bytes or os.PathLike object, not set
### auth_gdrive() returns a dictionary like this:
def auth_gdrive():
dic = {
"type": "miauuuuuu",
"pass": "miauuuu"
}
Im not allow to show whats really in the dic.

Since I wanted to pass the credentials details from within my application , and not from a json file I couldn't use:
ServiceAccountCredentials.from_json_keyfile_name()
from_json_keyfile_name() expects a json file. But looking into the docs I found the following:
ServiceAccountCredentials.from_json_keyfile_dict()
This will expect an dict object , this is all I needed.
Link:
https://oauth2client.readthedocs.io/en/latest/source/oauth2client.service_account.html
Thank you everyone again

Additional tip: I am using Google API to read Google Drive files, but also using AWS. I stored the service account credentials in AWS Secrets Manager, so that I did not need a file. I copy-pasted each key-value pair from the downloaded JSON file into AWS Secrets Manager. But I kept getting the error:
Traceback (most recent call last):
File "./copy_from_google_drive_to_s3.py", line 301, in <module>
sys.exit(main())
File "./copy_from_google_drive_to_s3.py", line 96, in main
keyfile_dict=keyDict, scopes=scopes,
File "/usr/local/lib/python3.7/site-packages/oauth2client/service_account.py", line 253, in from_json_keyfile_dict
revoke_uri=revoke_uri)
File "/usr/local/lib/python3.7/site-packages/oauth2client/service_account.py", line 185, in _from_parsed_json_keyfile
signer = crypt.Signer.from_string(private_key_pkcs8_pem)
File "/usr/local/lib/python3.7/site-packages/oauth2client/_pure_python_crypt.py", line 182, in from_string
raise ValueError('No key could be detected.')
ValueError: No key could be detected.
I had to convert the string representation of newline back into newline:
# Last part of using AWS Secrets Manager, returns json string.
sa_creds = get_secret_value_response['SecretString']
# Convert JSON string to dict.
sa_creds = json.loads(sa_creds)
# In the private key, 1-char newline got replaced with 2-char '\n'
sa_creds['private_key'] = sa_creds['private_key'].replace('\\n', '\n')
credentials = ServiceAccountCredentials.from_json_keyfile_dict(
keyfile_dict=sa_creds,
scopes=['https://www.googleapis.com/auth/drive.readonly',]
)

My solution is close to Bob McCormick. The difference is that it's using the credentials method for using service account info instead of JSON file.
Here i'm using Googles Secret Manager to import service account information so that my code can connect to a different GCP project:
from google.cloud import secretmanager
from google.oauth2 import service_account
# Create the Secret Manager client.
secret_client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = f"projects/{project-id}/secrets/{very-secret-secret-name}/versions/latest"
# Access the secret version.
secret_response = secret_client.access_secret_version(request={"name": name})
# Getting the secret data
secret_payload = json.loads(secret_response.payload.data.decode("UTF-8"))
# Applying the credentials as INFO instead of JSON
credentials = service_account.Credentials.from_service_account_info(
secret_payload,
scopes=["https://www.googleapis.com/auth/cloud-platform"],
)

Since you're using ServiceAccountCredentials, I'm assuming you're using OAuth2 for authorization. You can skip the json file by using oauth2client.SignedJwtAssertionCredentials to create the appropriate credentials object and pass that to gspread.authorize.
import gspread
from oauth2client.client import SignedJwtAssertionCredentials
credentials = SignedJwtAssertionCredentials(service_account_name, private_key.encode(),
['https://spreadsheets.google.com/feeds'])
gclient = gspread.authorize(credentials)
UPDATE: It appears that oauth2client.SignedJwtAssertionCredentials has been deprecated in favor of oauth2client.service_account.ServiceAccountCredentials, which only supports json and p12 keyfiles.

Issue Deleting Google Compute Account through API using Python

I was able to piece together a Python script to interact with the Google API Library using information from here and here. The code below is working and I'm able to list all accounts within a particular Project. See below:
Code:
import os
from googleapiclient import discovery
from gcp import get_key
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
service = discovery.build('cloudresourcemanager', 'v1')
project_id = 'test-buckets'
get_iam = {}
request = service.projects().getIamPolicy(resource=project_id, body=get_iam)
response = request.execute()
bindings = response['bindings']
for binding in bindings:
for member in binding['members']:
print(member)
Output:
ssh://xxxx#x.x.x.x:22/home/xxxx/venv/bin/python3.4 -u /home/xxxx/scratch.py
serviceAccount:xxxxxxxxxxx-compute#developer.gserviceaccount.com
serviceAccount:xxxxxxxxxxx#xxxxxxxxxxx.iam.gserviceaccount.com
user:xxxx#.xxxxx.com
Now, I'm trying to leverage the API to delete specific accounts that are not needed (script below), however, I can't seem to get the the API to work. I'm using the Google API Library again here, but it doesn't seem to be working. I'm not sure what I'm doing wrong. Any help you can offer is greatly appreciated.
Code:
import os
from pprint import pprint
from googleapiclient import discovery
from gcp import get_key
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
service = discovery.build('cloudresourcemanager', 'v1')
project_id = 'test-buckets'
service_account = 'projects/xxxxxxx/serviceAccounts/test-delete#test-buckets.iam.gserviceaccount.com'
request = service.projects().delete(name=service_account, body={})
response = request.execute()
pprint(response)
Output:
ssh://xxxx#x.x.x.x:22/home/xxxx/venv/bin/python3.4 -u /home/xxxx/scratch2.py
Traceback (most recent call last):
File "/home/xxxx/scratch2.py", line 13, in <module>
request = service.projects().delete(name=service_account, body={})
File "/home/xxxx/venv/lib/python3.4/site-packages/googleapiclient/discovery.py", line 722, in method
raise TypeError('Got an unexpected keyword argument "%s"' % name)
TypeError: Got an unexpected keyword argument "body"
Process finished with exit code 1

Maybe it's a little too late for an answer, but I think the only error in your code is that this line of code:
request = service.projects().delete(name=service_account, body={})
should in fact be like this:
request = service.projects().delete(name=service_account)
As you can see in the error you shared, TypeError: Got an unexpected keyword argument "body", so body is not an accepted argument here. In any case, printing the response in pprint(response) will probably output nothing, as the response to a DELETE operation has no content.

I figured out that I was using the wrong API when building the service too. Below is the corrected code:
service = discovery.build('iam', 'v1')
request = service.projects().serviceAccounts().delete(name=project_name)
response = request.execute()

Trying to load data from google storage bucket results in a apiclient.errors.HttpError: <HttpError 403... returned "Forbidden">

so right now im trying to write a python script which loads a few files from our google storage. I'm using Windows, have installed the Google Cloud SDK and set everything up with gsutil so i can succesfully load a file in typing the following the the commandline:
gsutil cp gs://pubsite_prod_rev_****/financial-stats/subscriptions/***.csv .
This loads a correct csv file to my folder. Cool.
Now i tried the same in python, following this documentation from google: https://support.google.com/googleplay/android-developer/answer/6135870?hl=en (you have to click on "Download reports using a client library & service account" and then on "Python example" to show the python code)
I used the exact same code
import json
from httplib2 import Http
from oauth2client.client import SignedJwtAssertionCredentials
from apiclient.discovery import build
from urllib import quote
client_email = 'serviceAccountEmail'
json_file = 'pathToMyServiceAccountPrivateKey'
cloud_storage_bucket = 'pubsite_prod_rev_******'
report_to_download = 'financial-stats/subscriptions/fileName.csv'
print report_to_download
private_key = json.loads(open(json_file).read())['private_key']
credentials = SignedJwtAssertionCredentials(client_email, private_key, 'https://www.googleapis.com/auth/devstorage.read_only')
storage = build('storage', 'v1', http=credentials.authorize(Http()))
print storage.objects().get(bucket=cloud_storage_bucket, object=report_to_download).execute()
to load the csv file.Unfortunetaley this result in the following message:
Traceback (most recent call last):
File "C:/Office/Financial Reports/Python Reporter/Test.py", line 66, in <module>
print storage.objects().get(bucket=cloud_storage_bucket, object=report_to_download).execute()
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\bq\third_party\oauth2client\util.py", line 137, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\bq\third_party\apiclient\http.py", line 724, in execute
raise HttpError(resp, content, uri=self.uri)
apiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/storage/v1/b/pubsite_prod_rev_17179175168453545219/o/financial-stats%2Fsubscriptions%2Fsubscriptions_de.kaasahealth.namagi_cody_monthly_sub_201701_device.csv?alt=json returned "Forbidden">
So obviously there is something wrong with the URL it builds. But i dont know what exactly, as it is the same thing google tells us to do on the doc page.
Hope someone can help me out here :/
EDIT: I changed the traceback, as it changed from error 404 to error 403 now..

Alright so the reason why it did not work was pretty... easy.
You have to enable access and add Admin rights or less (but i didn try out the other possibilities) inside the google dev console though Settings -> API Acess and selct the service account you are using.

It means you do now have permissions to download reports from
Google Play Console -> Download Reports -> Financial (or something related to financial reports).
There you can download the same files you're trying to download via gsutil.
Once you have permissions to download reports from Google Play Console you will be able to download it with gsutil or programmatically

You will need admin access or at least account-level permission for "View app information and download bulk reports (read-only)"
Once the permissions are updated, it'd take about 24 hours for the downloads to work.

Invalid credentials - 401 when trying to access Google Datastore from a GCE instance

I'm trying the adams.py example found on this URL
https://cloud.google.com/datastore/docs/getstarted/start_python/
As it's explained in this URL:
If you are running this code on Google Compute Engine, you do not have to create new service account credentials because the default service account for your project is used when you create an instance with the datastore and userinfo-email scopes.
While creating my instance, I checked the option:
Allow API access to all Google Cloud services in the same project.
When I run
python adams.py my-instance
I get:
ERROR:root:Error while doing datastore operation
ERROR:root:RPCError: beginTransaction Invalid Credentials
ERROR:root:HTTPError: 401 Unauthorized
In fact I get the same message even if I use a wrong name for the instance.
My problem seems similar to this thread from one year ago: Connect google datastore from existing google compute engine in python
In my own code, it works fine accessing BigQuery and then when it comes to Datastore it raises the same error, but at least is more specific addressing the line, which is the last one: resp = datastore.commit(req). Here is the section of the code use Datastore.
req = datastore.CommitRequest()
req.mode = datastore.CommitRequest.NON_TRANSACTIONAL
dsDashboard = req.mutation.insert_auto_id.add()
path_element = dsDashboard.key.path_element.add()
path_element.kind = 'Dashboard'
fmt = '%Y-%m-%d'
dashboardDate = dsDashboard.property.add()
dashboardDate.name = 'Date'
dashboardDate.value.string_value = dtDay.strftime(fmt)
dashboardValue = dsDashboard.property.add()
dashboardValue.name = 'Value'
dashboardValue.value.indexed = False
dashboardValue.value.string_value = encode(jDashboard)
resp = datastore.commit(req)
The error for my code:
Traceback (most recent call last):
File "code.py", line 262, in <module>
main()
File "code.py", line 256, in main
resp = datastore.commit(req)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/__init__.py", line 90, in commit
return get_default_connection().commit(request)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 135, in commit
datastore_v1_pb2.CommitResponse)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 195, in _call_method
raise RPCError(method, response, content)
googledatastore.connection.RPCError: commit RPC client failure with HTTP(401) Unauthorized: Invalid Credentials
In short, it seems something simple to do. Anyone here had the same problem and has found a workaround?

The issue here is:
While creating my instance, I checked the option:
Allow API access to all Google Cloud services in the same project.
One would assume that this would make your instance get all scopes but It seems it is not working so. I've tried it and if you inspect a newly created instance with:
gcloud compute instances describe my-instance
you'll see that it only gets one scope
scopes:
- https://www.googleapis.com/auth/cloud-platform
The solution here is to either
manually enable "User info" and "Cloud Datastore" ("Management, disk, networking, access & security options" > "Access & security") when creating your instance via Developers console
make sure you use correct scopes flag (--scopes datastore,userinfo-email) when using gcloud command as explained in Getting started with Google Cloud Datastore and Python/Protobuf article

AccessTokenRefreshError: Google Spreadsheet API with oAuth 2.0 Service Account of App Engine App

I'm trying to access a Google Spreadsheet via the GData API using oAuth 2.0 service-account credentials created for a Python 2.7 app hosted on Google App Engine.
The app uses the recent gdata-python-client from Google, v. 2.0.18 (gdata and atom).
The app uses the recent google-api-python-client-gae, v. 1.2.
In the Google Developer Console for this project (in this example referred to as "my-gae-app"), I have created a Service Account and delegated domain-wide authority to the service account as described here
The desired spreadsheet in Google Drive belongs to a Google Apps for Work domain, here referred to as "mygoogleappsdomain.com".
I have granted read+write access for the spreadsheet to my-gae-app#appspot.gserviceaccount.com and to the email address shown for this service account and which is assigned to clientEmail variable in the code below. Not sure which of the two email addresses would be actually needed, so I assigned both. The user with the impersonateUser email address also has read+write access to this spreadsheet.
With Google API Python Client's AppAssertionCredentials I can access the meta-data of the desired spreadsheet via the Google Drive API. However, if I try to access the spreadsheet's content using gdata, I'm getting errors. The best result I could get so far with the service account is using SignedJwtAssertionCredentials, as suggested here. However, I'm stuck with this AccessRefreshTokenError: access denied and I don't understand what's going wrong.
import os
import httplib2
from google.appengine.api import memcache
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
import gdata.spreadsheets.client
import gdata.spreadsheet.service
# AppAssertionCredentials is not supported in gdata python client library,
# so we use SignedJwtAssertionCredentials with the credential
# file of this service account.
# Load the key in PKCS 12 format that you downloaded from the Google API
# Console when you created your Service account.
clientEmail = '10346........-g3dp......................3m1em8#developer.gserviceaccount.com'
p12File = 'app.p12'
path = os.path.join(ROOT_DIR, 'data', 'oAuth2', p12File)
impersonateUser = 'user#mygoogleappsdomain.com'
spreadsheetKey = '1mcJHJ...................................juQMw' # ID copied from URL of desired spreadsheet in Google Drive
with open(path) as f:
privateKey = f.read()
f.close()
# Getting credentials with AppAssertionCredentials only worked successfully
# for Google API Client Library for Python, e.g. accessing file's meta-data.
# So we use SignedJwtAssertionCredentials, as suggested in
# https://stackoverflow.com/questions/16026286/using-oauth2-with-service-account-on-gdata-in-python
credentials = SignedJwtAssertionCredentials(
clientEmail,
privateKey,
scope=(
'https://www.googleapis.com/auth/drive.file ',
# added the scope above as suggested somewhere else,
# but error occurs with and with-out this scope
'https://www.googleapis.com/auth/drive',
'https://spreadsheets.google.com/feeds',
'https://docs.google.com/feeds'
),
sub=impersonateUser
)
http = httplib2.Http()
http = credentials.authorize(http)
auth2token = gdata.gauth.OAuth2TokenFromCredentials(credentials)
# error will occur, wether using SpreadsheetsService() or SpreadsheetsClient()
#srv = gdata.spreadsheet.service.SpreadsheetsService()
#srv = auth2token.authorize(srv)
clt = gdata.spreadsheets.client.SpreadsheetsClient()
clt = auth2token.authorize(clt)
# Until here no errors
wks = clt.get_worksheets(spreadsheetKey)
# AccessTokenRefreshError: access_denied
This is the error I get in the remote shell:
s~my-gae-app> wks = clt.get_worksheets(spreadsheetKey)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "gdata/spreadsheets/client.py", line 108, in get_worksheets
**kwargs)
File "gdata/client.py", line 640, in get_feed
**kwargs)
File "gdata/client.py", line 267, in request
uri=uri, auth_token=auth_token, http_request=http_request, **kwargs)
File "atom/client.py", line 122, in request
return self.http_client.request(http_request)
File "gdata/gauth.py", line 1344, in new_request
refresh_response = self._refresh(request_orig)
File "gdata/gauth.py", line 1485, in _refresh
self.credentials._refresh(httplib2.Http().request)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 653, in _refresh
self._do_refresh_request(http_request)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 710, in _do_refresh_request
raise AccessTokenRefreshError(error_msg)
AccessTokenRefreshError: access_denied
I'm not sure if this indicates that this service-account is denied access to the spreadsheet, or if there was an error with refreshing the access token. Do you know what's wrong with this code or setup?

I've figured out that calling SignedJwtAssertionCredentials with-out the sub parameter (for the "impersonated" user) will not produce AccessTokenRefreshError: access_denied
import os
import httplib2
from google.appengine.api import memcache
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
import gdata.spreadsheets.client
import gdata.spreadsheet.service
# AppAssertionCredentials is not supported in gdata python client library,
# so we use SignedJwtAssertionCredentials with the credential
# file of this service account.
# Load the key in PKCS 12 format that you downloaded from the Google API
# Console when you created your Service account.
clientEmail = '10346........-g3dp......................3m1em8#developer.gserviceaccount.com'
p12File = 'app.p12'
path = os.path.join(ROOT_DIR, 'data', 'oAuth2', p12File)
impersonateUser = 'user#mygoogleappsdomain.com'
spreadsheetKey = '1mcJHJ...................................juQMw' # ID copied from URL of desired spreadsheet in Google Drive
with open(path) as f:
privateKey = f.read()
f.close()
# Getting credentials with AppAssertionCredentials only worked successfully
# for Google API Client Library for Python, e.g. accessing file's meta-data.
# So we use SignedJwtAssertionCredentials, as suggested in
# http://stackoverflow.com/questions/16026286/using-oauth2-with-service-account-on-gdata-in-python
# but with-out the sub parameter!
credentials = SignedJwtAssertionCredentials(
clientEmail,
privateKey,
scope=(
'https://www.googleapis.com/auth/drive.file ',
# added the scope above as suggested somewhere else,
# but error occurs with and with-out this scope
'https://www.googleapis.com/auth/drive',
'https://spreadsheets.google.com/feeds',
'https://docs.google.com/feeds'
),
# sub=impersonateUser
)
http = httplib2.Http()
http = credentials.authorize(http)
auth2token = gdata.gauth.OAuth2TokenFromCredentials(credentials)
# this pattern would eventually also work using SpreadsheetsService()
# SpreadsheetsService methods are different from SpreadsheetsClient, though
#srv = gdata.spreadsheet.service.SpreadsheetsService()
#srv = auth2token.authorize(srv)
clt = gdata.spreadsheets.client.SpreadsheetsClient()
clt = auth2token.authorize(clt)
wks = clt.get_worksheets(spreadsheetKey)
# works now!