Can someone please give me a clear explanation of how to get the Google Calendar API v3 working with the Python Client? Specifically, the initial OAuth stage is greatly confusing me. All I need to do is access my own calendar, read it, and make changes to it. Google provides this code for configuring my app:
import gflags
import httplib2
from apiclient.discovery import build
from oauth2client.file import Storage
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run
FLAGS = gflags.FLAGS
# Set up a Flow object to be used if we need to authenticate. This
# sample uses OAuth 2.0, and we set up the OAuth2WebServerFlow with
# the information it needs to authenticate. Note that it is called
# the Web Server Flow, but it can also handle the flow for native
# applications
# The client_id and client_secret are copied from the API Access tab on
# the Google APIs Console
FLOW = OAuth2WebServerFlow(
client_id='YOUR_CLIENT_ID',
client_secret='YOUR_CLIENT_SECRET',
scope='https://www.googleapis.com/auth/calendar',
user_agent='YOUR_APPLICATION_NAME/YOUR_APPLICATION_VERSION')
# To disable the local server feature, uncomment the following line:
# FLAGS.auth_local_webserver = False
# If the Credentials don't exist or are invalid, run through the native client
# flow. The Storage object will ensure that if successful the good
# Credentials will get written back to a file.
storage = Storage('calendar.dat')
credentials = storage.get()
if credentials is None or credentials.invalid == True:
credentials = run(FLOW, storage)
# Create an httplib2.Http object to handle our HTTP requests and authorize it
# with our good Credentials.
http = httplib2.Http()
http = credentials.authorize(http)
# Build a service object for interacting with the API. Visit
# the Google APIs Console
# to get a developerKey for your own application.
service = build(serviceName='calendar', version='v3', http=http,
developerKey='YOUR_DEVELOPER_KEY')
But (a) it makes absolutely no sense to me; the comment explanations are terrible, and (b) I don't know what to put in the variables. I've registered my program with Google and signed up for a Service Account key. But all that gave me was an encrypted key file to download, and a client ID. I have no idea what a "developerKey" is, or what a "client_secret" is? Is that the key? If it is, how do I get it, since it is actually contained in an encrypted file? Finally, given the relatively simple goals of my API use (i.e., it's not a multi-user, multi-access operation), is there a simpler way to be doing this? Thanks.
A simple (read: way I've done it) way to do this is to create a web application instead of a service account. This may sound weird since you don't need any sort of web application, but I use this in the same way you do - make some queries to my own calendar/add events/etc. - all from the command line and without any sort of web-app interaction. There are ways to do it with a service account (I'll tinker around if you do in fact want to go on that route), but this has worked for me thus far.
After you create a web application, you will then have all of the information indicated above (side note: the sample code above is based on a web application - to use a service account your FLOW needs to call flow_from_clientsecrets and further adjustments need to be made - see here). Therefore you will be able to fill out this section:
FLOW = OAuth2WebServerFlow(
client_id='YOUR_CLIENT_ID',
client_secret='YOUR_CLIENT_SECRET',
scope='https://www.googleapis.com/auth/calendar',
user_agent='YOUR_APPLICATION_NAME/YOUR_APPLICATION_VERSION')
You can now fill out with the values you see in the API console (client_id = the entire Client ID string, client_secret = the client secret, scope is the same and the user_agent can be whatever you want). As for the service line, developerKey is the API key you can find under the Simple API Access section in the API console (label is API key):
service = build(serviceName='calendar', version='v3', http=http,
developerKey='<your_API_key>')
You can then add in a simple check like the following to see if it worked:
events = service.events().list(calendarId='<your_email_here>').execute()
print events
Now when you run this, a browser window will pop up that will allow you to complete the authentication flow. What this means is that all authentication will be handled by Google, and the authentication response info will be stored in calendar.dat. That file (which will be stored in the same directory as your script) will contain the authentication info that the service will now use. That is what is going here:
storage = Storage('calendar.dat')
credentials = storage.get()
if credentials is None or credentials.invalid == True:
credentials = run(FLOW, storage)
It checks for the existence of valid credentials by looking for that file and verifying the contents (this is all abstracted away from you to make it easier to implement). After you authenticate, the if statement will evaluate False and you will be able to access your data without needing to authenticate again.
Hopefully that shines a bit more light on the process - long story short, make a web application and use the parameters from that, authenticate once and then forget about it. I'm sure there are various points I'm overlooking, but hopefully it will work for your situation.
Google now has a good sample application that gets you up and running without too much fuss. It is available as the "5 minute experience - Quickstart" on their
Getting Started page.
It will give you a URL to visit directly if you are working on a remote server without a browser.
Related
Currently, I am building the async frontend to my TF2 model. Now it works as two services, 1st service is a twisted service, and 2nd service is a TensorFlow serving.
The async web client is being used to query the model asynchronously. For practical reasons, I've deployed the model into the GCP AI Platform, and I can get data from it using the python code from examples, and it is okay.
But the thing is that the Google API client is synchronous, and I would like to use the asynchronous client. Since, AFAIK, there are no actively supported async clients for GCP, I tried to get straightforward and use REST. The model input is the same on TensorFlow serving (GCP AI Platform uses TensorFlow serving internally, I believe).
To perform the async call, I need to have:
Model URL. (I have it)
Input data. (I also have it)
Access token.
I saw some examples that are:
import googleapiclient.discovery
credentials = service_account.Credentials.from_service_account_file(
'/path/to/key.json',
scopes=['https://www.googleapis.com/auth/cloud-platform'])
But the issue is that credential.token is None, so I can't use it.
So I have a question: how could I get the access token to use in the rest request then?
Or maybe there is another but better way of doing that?
I already saw the following question: How to get access token from instance of google.oauth2.service_account.Credentials object?
but I am think that it is slightly irrelevant.
The following code sets up the data structures for managing credentials (OAuth tokens) from a service account. No tokens are requested at this point.
credentials = service_account.Credentials.from_service_account_file(
'/path/to/key.json',
scopes=['https://www.googleapis.com/auth/cloud-platform'])
Tokens are not requested from the Google auth server until required. There are several reasons: a) network calls take time - a significant amount of time for network failures; b) tokens expire; c) tokens are cached until they (almost) expire.
To generate a token, call the refresh() method:
import google.auth.transport.requests
request = google.auth.transport.requests.Request()
credentials.refresh(request)
credential.token will now contain an OAuth Access Token else an exception will be thrown (network error, etc.).
Just focusing on "How do I get an access token?", you will need to:
Create a Service Account
Use the Google Auth library
Run code
NOTE If you're running the code off-GCP (i.e. not on App Engine, Compute Engine, GKE etc.), then you will need to create a Key for the Service Account and you will need to export GOOGLE_APPLICATION_CREDENTIALS=path/to/your/key.json. Application Default Credentials (see below) simplify auth.
See: Authenticating as a Service Account
And:
import google.auth
from googleapiclient.discovery import build
SCOPES = ["https://www.googleapis.com/auth/cloud-platform"]
creds, project_id = google.auth.default(scopes=SCOPES)
service = build(GOOGLE_API, GOOGLE_API_VERSION, credentials=creds)
What is the python programmatic alternative to the gcloud command line gcloud auth print-identity-token?
I am trying to invoke Google Cloud Function by http trigger (only for auth users) and i need to pass the identity token in the Authentication header. I have method which works great when i run the code on GCP app engine. However, i struggle to find a way to get this identity token when i run the program on my own machine (where i can create the token with gcloud command line gcloud auth print-identity-token)
I found how to create access-token according to this answer but i didn't managed to understand how can i create identity-token.
Thank you in advance!
Great topic! And it's a long long way, and months of tests and discussion with Google.
TL;DR: you can't generate an identity token with your user credential, you need to have a service account (or to impersonate a service) to generate an identity token.
If you have a service account key file, I can share a piece of code to generate an identity token, but generating and having a service account key file is globally a bad practice.
I released an article on this and 2 merge requests to implement an evolution in the Java Google auth library (I'm more Java developer that python developer even if I also contribute to python OSS project) here and here. You can read them if you want to understand what is missing and how works the gcloud command today.
On the latest merge request, I understood that something is coming from google, internally, but up to now, I didn't see anything...
If you have a service account you can impersonate this is one way to get an ID token in Python from a local/dev machine.
import google.auth
from google.auth.transport.requests import AuthorizedSession
def impersonated_id_token():
credentials, project = google.auth.default(scopes=['https://www.googleapis.com/auth/cloud-platform'])
authed_session = AuthorizedSession(credentials)
sa_to_impersonate = "<SA_NAME>#<GCP_PROJECT>.iam.gserviceaccount.com"
request_body = {"audience": "<SOME_URL>"}
response = authed_session.post( f'https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{sa_to_impersonate}:generateIdToken',request_body)
return response
if __name__ == "__main__":
print(impersonated_id_token().json())
I am developing an application that is supposed to help a friend of mine better organize his YouTube channels. He has multiple channels on different Google accounts. I'm developing this in Python and I currently don't have too much experience with the YouTube Data API, which I'm planning on using, since it seems like the only option.
The application itself isn't very complicated. The only things it needs to be able to do is upload videos, with a specified title, description and other properties and it should also be possible to write comments on videos. I started a simple application in the Google Developers Console, enabled the YouTube Data API and created an API key and an OAUTH-Client ID.
So far I've managed to post comments on videos, but it seems like every time I run the Python script (currently its just a simple script that posts a single comment) Google wants me to explicitly choose which account I want to use and I have to give permission to the script every time I run it.
Is there a way I can just run the script once and tell Google which account I want to use to post the comment, give all the permissions and Google then remembers that so I don't have to explicitly give permissions every time?
Also how would I be able to then switch accounts and make uploads with that one, because currently I always need to choose one, when the Google client pops up, when running the script.
I've heard you can get an application authorized by Google, would that help with this or is it fine if I just keep my app in test and not in production?
If you have N accounts and want to upload videos on each of them, then you'll have to run to successful completion N OAuth 2 authorization/authentication flows.
For each of these N OAuth flows, upon completing each one successfully, you'll have to make persistent the obtained credentials data to a separate file within your computer local storage.
This could well be conceived as an initialization step of your app (although, at any later stage, you may well repeat it for any additional channel that you need your app be aware of). Your code would look like:
# run an OAuth flow; then obtain credentials data
# relative to the channel the app's user had chosen
# during that OAuth flow
from google_auth_oauthlib.flow import InstalledAppFlow
scopes = ['https://www.googleapis.com/auth/youtube']
flow = InstalledAppFlow.from_client_secrets_file(
client_secret_file, scopes)
cred = flow.run_console()
# build an YouTube service object such that to
# be able to retrieve the ID of the channel that
# the app's user had chosen during the OAuth flow
from googleapiclient.discovery import build
youtube = build('youtube', 'v3', credentials = cred)
response = youtube.channels().list(
part = 'id',
mine = True
).execute()
channel_id = response['items'][0]['id']
# save the credentials data to a JSON text file
cred_file = f"/path/to/credentials/data/dir/{channel_id}.json"
with open(cred_file, 'w', encoding = 'UTF-8') as json_file:
json_file.write(cred.to_json())
Above, client_secret_file is the full path to the file containing your app's client secret JSON file that you've obtained from Google Developers Console.
Subsequently, each time you'll want to upload a video, you'll have to choose from within the app to which channel to upload that video. From the perspective of the logic of your program that would imply the following thing -- say you've chosen the channel of which ID is channel_id: do read in the credentials data file associated to channel_id for to pass its content to your YouTube service object youtube constructed as shown below:
# read in the credentials data associated to
# the channel identified by its ID 'channel_id'
from google.oauth2.credentials import Credentials
cred_file = f"/path/to/credentials/data/dir/{channel_id}.json"
cred = Credentials.from_authorized_user_file(cred_file)
# the access token need be refreshed when
# the previously saved one expired already
from google.auth.transport.requests import Request
assert cred and cred.valid and cred.refresh_token
if cred.expired:
cred.refresh(Request())
# save credentials data upon it got refreshed
with open(cred_file, 'w', encoding = 'UTF-8') as json_file:
json_file.write(cred.to_json())
# construct an YouTube service object through
# which any API invocations are authorized on
# behalf of the channel with ID 'channel_id'
from googleapiclient.discovery import build
youtube = build('youtube', 'v3', credentials = cred)
Upon running this code, the YouTube service object youtube will be initialized such a way that each an every API endpoint call that is issued through this object will accomplish an authorized request on behalf of the channel identified by channel_id.
An important note: you need to have installed the package Google Authentication Library for Python, google-auth, version >= 1.21.3 (google-auth v1.3.0 introduced Credentials.from_authorized_user_file, v1.8.0 introduced Credentials.to_json and v1.21.3 fixed this latter function w.r.t. its class' expiry member), for the credentials object cred to be saved to and loaded from JSON text files.
Also an important note: the code above is simplified as much as possible. Error conditions are not handled at all. For example, the code above does not handle the error situation when cred_file already exists at the time of writing out a new credentials data file or when cred_file does not exist at the time of reading in credentials data that's supposed to already exist.
I'm developing a Cloud Run Service that accesses different Google APIs using a service account's secrets file with the following python 3 code:
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file(SECRETS_FILE_PATH, scopes=SCOPES)
In order to deploy it, I upload the secrets file during the build/deploy process (via gcloud builds submit and gcloud run deploy commands).
How can I avoid uploading the secrets file like this?
Edit 1:
I think it is important to note that I need to impersonate user accounts from GSuite/Workspace (with domain wide delegation). The way I deal with this is by using the above credentials followed by:
delegated_credentials = credentials.with_subject(USER_EMAIL)
Using the Secret Manager might help you, as you can manage the multiple secrets you have and not have them stored as files, as you are doing right now. I would recommend you to take a look at this article here, so you can get more information on how to use it with Cloud Run, to improve the way you manage your secrets.
In addition to that, as clarified in this similar case here, you have two options: use default service account that comes with it or deploy another one with the Service Admin role. This way, you won't need to specify keys with variables - as clarified by a Google developer in this specific answer.
To improve the security, the best way is to never use service account key file, locally or on GCP (I wrote an article on this). To achieve this, Google Cloud service have an automatically loaded service account, either this one by default or, when possible, a custom one.
On Cloud Run, the default service account is the Compute Engine default service account (I recommend you to never use it, it has editor role on the project, it's too wide!), or you can specify the service account to use (--service-account= parameter)
Then, in your code, simply use the ADC mechanism (Application Default Credential) to get your credentials, like this in Python
import google.auth
credentials, project_id = google.auth.default(scopes=SCOPES)
I've found one way to solve the problem.
First, as suggested by guillaume blaquiere answer, I used google.auth ADC mechanism:
import google.auth
credentials, project_id = google.auth.default(scopes=SCOPES)
However, as I need to impersonate GSuite's (now Workspace) accounts, this method is not enough, as the credentials object generated from this method does not have the with_subject property. This led me to this similar post and specific answer which works a way to convert google.auth.credentials into the Credential object returned by service_account.Credentials.from_service_account_file. There was one problem with his solution, as it seemed that an authentication scope was missing.
All I had to do is add the https://www.googleapis.com/auth/cloud-platform scope to the following places:
The SCOPES variable in the code
Google Admin > Security > API Controls > Set client ID and scope for the service account I am deploying with
At the OAuth Consent Screen of my project
After that, my Cloud Run had access to credentials that were able to impersonate user's accounts without using key files.
I am building a command line tool using python that interfaces with an RESTful api. The API uses oauth2 for authentication. Rather than asking for access_token every time user runs the python tool. Can I store the access_token in some way so that I can use it till its lifespan? If it is then how safe it is.
You can store the access token in a file on your user's desktop.
You can do so using a storage. Assuming you use oauth2client:
# Reading credentials
store = oauth2client.file.Storage(cred_path)
credentials = store.get()
# Writing credentials
creds = client.AccessTokenCredentials(access_token, user_agent)
creds.access_token = access_token
creds.refresh_token = refresh_token
creds.client_id = client_id
creds.client_secret = client_secret
# For some reason it does not save all the credentials,
# so write them to a json file manually instead
with open(credential_path, "w") as f:
f.write(creds.to_json)
In terms of security, I would not see much of a threat here as these access tokens will be on a user's desktop. If someone wants to get their access token, they would need to have read access to that file for that time frame. However, if they can already do that, they most likely also can use your script to send them a copy of the user's access token every time it is authenticated. But take my word lightly as I'm not a professional in that area. See information security stack exchange.
A post in information security stack exchange did talk about this:
these tokens give access to some fairly privileged information about your users.
However, the question was addressed to a database instead.
In conclusion, you can keep it in a file. (But take my word with a grain of salt)
Do you want to store it on the service side or locally?
Since your tool interfaces RESTful API, which is stateless, meaning that no information is stored between different requests to API, you actually need to provide access token every time your client accesses any of the REST endpoints. I am maybe missing some of the details in your design, but access tokens should be used only for authorization, since your user is already authenticated if he has a token. This is why tokens are valid only for a certain amount of time, usually 1 hour.
So you need to provide a state either by using cookie (web interface) or storing the token locally (Which is what you meant). However, you should trigger the entire oauth flow every time a user logs in to your client (authenticating user and providing a new auth token) otherwise you are not utilizing the benefits of oauth.