I am attempting to use Python to access some files stored on a Google Team Drive. I have figured out the functionality to download files, but am running into a mysterious issue when attempting to get metadata
If I execute the following:
myfileid = 'thegooglefileid'
self.service = build('drive', 'v3', http=creds.authorize(Http()))
data = self.service.files().get_media(fileId=myfileid).execute()
meta = self.service.files().get(fileId=myfileid,fields="*").execute()
"data" returns as expected allowing me the info to download my files as I expect. "meta" returns with a HttpError 404 indicating it can not find the file (which it in fact found in the line above).
I know this issue can occur with incorrectly set authorization, but my authorization is set such that I expect this to work
SCOPES = ['https://www.googleapis.com/auth/drive.metadata.readonly',
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/drive.file']
Any ideas why the file is visible to one part of the API but not the other ?
In this case, an important point is this is accessing TeamDrive. In the specific case of the get call in which "meta" is retrieved, the API needs to be informed that it is using a TeamDrive
The following code works once I figured this out
myfileid = 'thegooglefileid'
self.service = build('drive', 'v3', http=creds.authorize(Http()))
data = self.service.files().get_media(fileId=myfileid).execute()
meta = self.service.files().get(fileId=myfileid,fields="*",supportsTeamDrives=True).execute()
Interestingly, get requires this parameter while get_media functions fine without it
Related
I'm trying to call a Google Cloud function from within Python using the following:
import requests
url = "MY_CLOUD_FUNCTON_URL"
data = {'name': 'example'}
response = requests.post(url, data = data)
but I get back the error: Your client does not have permission to get URL MY_CLOUD_FUNCTON from this server
Does anyone know how I can avoid this error? I am assuming I should be passing credentials as part of the request somehow?
Also note that if I instead try to call the function via gcloud from the command line like the below then it works, but i want to do this from within python
gcloud functions call MY_CLOUD_FUNCTON --data '{"name": "example"}'
Any help would be really appreciated!
Given a working Cloud Function in HTTP mode which requires authentication in order to be triggered.
You need to generate an authentication token and insert it in the header as shown below:
import os
import json
import requests
import google.oauth2.id_token
import google.auth.transport.requests
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = './my-service-account.json'
request = google.auth.transport.requests.Request()
audience = 'https://mylocation-myprojectname.cloudfunctions.net/MyFunctionName'
TOKEN = google.oauth2.id_token.fetch_id_token(request, audience)
r = requests.post(
'https://mylocation-myprojectname.cloudfunctions.net/MyFunctionName',
headers={'Authorization': f"Bearer {TOKEN}", "Content-Type": "application/json"},
data=json.dumps({"key": "value"}) # possible request parameters
)
r.status_code, r.reason
You have a few options here. Either open the function to the public so that anyone can call it or take the more secure route, albeit necessitating a bit more steps. I will cover the 2nd option since it's the one I would suggest for security reasons, but should you be satisfied with simply opening the function to the public ( which is especially useful if you are trying to create a public endpoint after all ), see this documentation.
If you want to limit who can invoke your GCF however, you would have to perform a few more steps.
Create a service account and give it the Cloud Functions Invoker role ( if you simply want to restrict it's permissions to only invoke the GCF )
After you assign the Service Account a role(s), the next page will give you the option to create a key
After creating the Service Account Key and downloading it as credentials.json, the next step is straightforward. You would simply populate the environment variable GOOGLE_APPLICATION_CREDENTIALS with the path to the credentials.json file.
Once these steps are done, you can simply invoke the GCF as you did before, only this time, it will invoke it as the service account that you created, which contained all the permissions necessary to invoke a GCF.
This may be obvious to many, but to add to Marco's answer (I can't comment yet):
Make sure to install the google-auth package, not the google package. More details in the documentation and the requirements.txt for the code on GitHub.
I have a situation where I am trying to create two Cloud Functions namely CF1 & CF2 and I have one Cloud Scheduler. Both cloud functions are having authenticated invocation enabled. My flow is Cloud Scheduler will trigger CF1. On completion of CF1, the CF1 will trigger CF2 as a http call. I have referred Cannot invoke Google Cloud Function from GCP Scheduler to access authenticated CF1 from Cloud Scheduler and able to access CF1. But I am getting problem when accessing CF2 from CF1. The CF1 does not trigger CF2 and also not giving any error message. Do we need to follow any other technique when accessing authenticated Cloud Function from another authenticated Cloud Function.
CF1 code:
import json
import logging
from requests_futures.sessions import FuturesSession
def main(request):
# To read parameter values from request (url arguments or Json body).
raw_request_data = request.data
string_request_data = raw_request_data.decode("utf-8")
request_json: dict = json.loads(string_request_data)
request_args = request.args
if request_json and 'cf2_endpoint' in request_json:
cf2_endpoint = request_json['cf2_endpoint']
elif request_args and 'cf2_endpoint' in request_args:
cf2_endpoint = request_args['cf2_endpoint']
else:
cf2_endpoint = 'Invalid endpoint for CF2'
logger = logging.getLogger('test')
try:
session = FuturesSession()
session.get("{}".format(cf2_endpoint))
logger.info("First cloud function executed successfully.")
except RuntimeError:
logger.error("Exception occurred {}".format(RuntimeError))
CF2 code:
import logging
def main(request):
logger = logging.getLogger('test')
logger.info("second cloud function executed successfully.")
Current output logs:
First cloud function executed successfully.
Expected output logs:
First cloud function executed successfully.
second cloud function executed successfully.
Note: Same flow is working if I use unauthenticated access to the both cloud functions.
Two things are happening here:
You're not using request-futures entirely correctly. Since the request is made asynchronously, you need to block on the result before the function implicitly returns, otherwise it might return before your HTTP request completes (although it probably is in this example):
session = FuturesSession()
future = session.get("{}".format(cf2_endpoint))
resp = future.result() # Block on the request completing
The request you're making to the second function is not actually an authenticated request. Outbound requests from a Cloud Function are not authenticated by default. If you looked at what the actual response is above, you would see:
>>> resp.status_code
403
>>> resp.content
b'\n<html><head>\n<meta http-equiv="content-type" content="text/html;charset=utf-8">\n<title>403 Forbidden</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Forbidden</h1>\n<h2>Your client does not have permission to get URL <code>/function_two</code> from this server.</h2>\n<h2></h2>\n</body></html>\n'
You could jump through a lot of hoops to properly authenticate this request, as detailed in the docs: https://cloud.google.com/functions/docs/securing/authenticating#function-to-function
However, a better alternative would be to make your second function a "background" function and invoke it via a PubSub message published from the first function instead:
from google.cloud import pubsub
publisher = pubsub.PublisherClient()
topic_name = 'projects/{project_id}/topics/{topic}'.format(
project_id=<your project id>,
topic='MY_TOPIC_NAME', # Set this to something appropriate.
)
def function_one(request):
message = b'My first message!'
publisher.publish(topic_name, message)
def function_two(event, context):
message = event['data'].decode('utf-8')
print(message)
As long as your functions have the permissions to publish PubSub messages, this avoids the need to add authorization to the HTTP requests, and also ensures at-least-once delivery.
Google Cloud Function provide REST API interface what include call method that can be used in another Cloud Function HTTP invokation.
Although the documentation mention using Google-provided client libraries there is still non one for Cloud Function on Python.
And instead you need to use general Google API Client Libraries. [This is the python one].3
Probably, the main difficulties while using this approach is an understanding of authentification process.
Generally you need provide two things to build a client service:
credentials ans scopes.
The simpliest way to get credentials is relay on Application Default Credentials (ADC) library. The rigth documentation about that are:
https://cloud.google.com/docs/authentication/production
https://github.com/googleapis/google-api-python-client/blob/master/docs/auth.md
The place where to get scopes is the each REST API function documentation page.
Like, OAuth scope: https://www.googleapis.com/auth/cloud-platform
The complete code example of calling 'hello-world' clound fucntion is below.
Before run:
Create default Cloud Function on GCP in your project.
Keep and notice the default service account to use
Keep the default body.
Notice the project_id, function name, location where you deploy function.
If you will call function outside Cloud Function environment (locally for instance) setup the environment variable GOOGLE_APPLICATION_CREDENTIALS according the doc mentioned above
If you will call actualy from another Cloud Function you don't need to configure credentials at all.
from googleapiclient.discovery import build
from googleapiclient.discovery_cache.base import Cache
import google.auth
import pprint as pp
def get_cloud_function_api_service():
class MemoryCache(Cache):
_CACHE = {}
def get(self, url):
return MemoryCache._CACHE.get(url)
def set(self, url, content):
MemoryCache._CACHE[url] = content
scopes = ['https://www.googleapis.com/auth/cloud-platform']
# If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set,
# ADC uses the service account file that the variable points to.
#
# If the environment variable GOOGLE_APPLICATION_CREDENTIALS isn't set,
# ADC uses the default service account that Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run,
# and Cloud Functions provide
#
# see more on https://cloud.google.com/docs/authentication/production
credentials, project_id = google.auth.default(scopes)
service = build('cloudfunctions', 'v1', credentials=credentials, cache=MemoryCache())
return service
google_api_service = get_cloud_function_api_service()
name = 'projects/{project_id}/locations/us-central1/functions/function-1'
body = {
'data': '{ "message": "It is awesome, you are develop on Stack Overflow language!"}' # json passed as a string
}
result_call = google_api_service.projects().locations().functions().call(name=name, body=body).execute()
pp.pprint(result_call)
# expected out out is:
# {'executionId': '3h4c8cb1kwe2', 'result': 'It is awesome, you are develop on Stack Overflow language!'}
I'm trying to get a list of folders that I created in Google Drive (Python, Flask):
SCOPES = ['https://www.googleapis.com/auth/drive']
drive_credentials = service_account.Credentials.from_service_account_file(
json_url, scopes=SCOPES)
drive_service = build('drive', 'v3', credentials = drive_credentials)
results = drive_service.files().list(
q="mimeType='application/vnd.google-apps.folder'",
spaces='drive').execute()
results.files is an empty array.
I can not figure out what is wrong. If I try the same query here https://developers.google.com/drive/api/v3/reference/files/list?apix_params=%7B%22q%22%3A%22mimeType%20%3D%20%27application%2Fvnd.google-apps.folder%27%22%7D I see all my folders.
Also, if I remove query I can see all my files but not folders.
UPD. I found it doesn't see any other files except Getting started pdf. I created just couple of test files and the query has still only one result.
I found what was the problem. Even although the scope gave "the full access" it actually did not. Only after I gave permission to my own email it started working.
What I did: in the Google drive interface I selected folders, then click "Share" and entered the email from credentials: xxx#yyy.iam.gserviceaccount.com
That's weird but worked.
I am having some issues with the Google Reports API. I have no issues running the sample code provided in the documentation to get the reports.activities.list data, but when I change to program to to try and pull full domain data (reports.customerUsageReports.get) I get an error stating that "'Resource' object has no attribute 'usage'".
I have no issue with auth and have changed the api scopes in the program to https://www.googleapis.com/auth/admin.reports.usage.readonly as required.
I am running the following snippet to try and access the data
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
service = discovery.build('admin', 'reports_v1', http=http)
print('Getting data')
results = service.usage().list(date='2016-01-01').execute()
usage = results.get('items', [])
if not usage:
print('No data found.')
else:
print('Data:')
for use in usageReports:
print(use)
Make sure you follow these steps:
According to the Python quickstart, if you modify the scopes, you need to delete the previously saved credentials at ~/.credentials/admin-reports_v1-python-quickstart.json
For the Scopes, in requesting access using OAuth 2.0, your application needs the scope information, as well as information that Google supplies when you register your application (such as the client ID and the client secret).
For more information check this link and the documentation about CustomerUsageReports: get to know how to use these parameter correctly.
The method you want to call is:
service.customerUsageReports()
You are calling:
service.usage()
This does not exist.
I'm trying the adams.py example found on this URL
https://cloud.google.com/datastore/docs/getstarted/start_python/
As it's explained in this URL:
If you are running this code on Google Compute Engine, you do not have to create new service account credentials because the default service account for your project is used when you create an instance with the datastore and userinfo-email scopes.
While creating my instance, I checked the option:
Allow API access to all Google Cloud services in the same project.
When I run
python adams.py my-instance
I get:
ERROR:root:Error while doing datastore operation
ERROR:root:RPCError: beginTransaction Invalid Credentials
ERROR:root:HTTPError: 401 Unauthorized
In fact I get the same message even if I use a wrong name for the instance.
My problem seems similar to this thread from one year ago: Connect google datastore from existing google compute engine in python
In my own code, it works fine accessing BigQuery and then when it comes to Datastore it raises the same error, but at least is more specific addressing the line, which is the last one: resp = datastore.commit(req). Here is the section of the code use Datastore.
req = datastore.CommitRequest()
req.mode = datastore.CommitRequest.NON_TRANSACTIONAL
dsDashboard = req.mutation.insert_auto_id.add()
path_element = dsDashboard.key.path_element.add()
path_element.kind = 'Dashboard'
fmt = '%Y-%m-%d'
dashboardDate = dsDashboard.property.add()
dashboardDate.name = 'Date'
dashboardDate.value.string_value = dtDay.strftime(fmt)
dashboardValue = dsDashboard.property.add()
dashboardValue.name = 'Value'
dashboardValue.value.indexed = False
dashboardValue.value.string_value = encode(jDashboard)
resp = datastore.commit(req)
The error for my code:
Traceback (most recent call last):
File "code.py", line 262, in <module>
main()
File "code.py", line 256, in main
resp = datastore.commit(req)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/__init__.py", line 90, in commit
return get_default_connection().commit(request)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 135, in commit
datastore_v1_pb2.CommitResponse)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 195, in _call_method
raise RPCError(method, response, content)
googledatastore.connection.RPCError: commit RPC client failure with HTTP(401) Unauthorized: Invalid Credentials
In short, it seems something simple to do. Anyone here had the same problem and has found a workaround?
The issue here is:
While creating my instance, I checked the option:
Allow API access to all Google Cloud services in the same project.
One would assume that this would make your instance get all scopes but It seems it is not working so. I've tried it and if you inspect a newly created instance with:
gcloud compute instances describe my-instance
you'll see that it only gets one scope
scopes:
- https://www.googleapis.com/auth/cloud-platform
The solution here is to either
manually enable "User info" and "Cloud Datastore" ("Management, disk, networking, access & security options" > "Access & security") when creating your instance via Developers console
make sure you use correct scopes flag (--scopes datastore,userinfo-email) when using gcloud command as explained in Getting started with Google Cloud Datastore and Python/Protobuf article