I have looked at the other question regarding this topic and it doesn't seem to match my error. I'm getting the error when running Google Sheets APIv4:
raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
HttpAccessTokenRefreshError: invalid_grant
Error occurs on the line service.spreadsheets().values().get(spreadsheetId=key, range=ranges).execute()
This error only pops up sometimes. If I don't do anything and just run the code again. It will take me through the authentication flow process again and I get
Authentication successful.
Storing credentials to C:\Users\jason\.credentials\sheets.googleapis.com-python-quickstart.json
After which, I can run any code for a while until the same HttpAccessTokenRefreshError: invalid_grant pops up again and I have to reauthenticate again.
How do I prevent this?
I'm using the code found developers.google.com/sheets/api/quickstart/python.
I've tried to use ntp to sync time with the following
import time
import os
try:
import ntplib
client = ntplib.NTPClient()
response = client.request('pool.ntp.org')
os.system('date ' + time.strftime('%m%d%H%M%Y.%S',time.localtime(response.tx_time)))
except:
print('Could not sync with time server.')
print('Done.')
but getting:
The system cannot accept the date entered.
Enter the new data: (mm-dd-yy)
After I enter the current date, nothing happens.
I have also looked at this page. https://blog.timekit.io/google-oauth-invalid-grant-nightmare-and-how-to-fix-it-9f4efaf1da35#.5utz2vcn6
This problem also arises when I run code that is longer than 1 hour to finish. On the refresh token. It always bomb.
Now I'm thinking, tokens granted only lasts one hour and on refreshes, it always bombs.
I have posted the code for connecting:
class APIv4:
def __init__(self):
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
discoveryUrl = ('https://sheets.googleapis.com/$discovery/rest?'
'version=v4')
self.service = discovery.build('sheets', 'v4', http=http,
discoveryServiceUrl=discoveryUrl)
def get_credentials():
"""Gets valid user credentials from storage.
If nothing has been stored, or if the stored credentials are invalid,
the OAuth2 flow is completed to obtain the new credentials.
Returns:
Credentials, the obtained credential.
"""
home_dir = os.path.expanduser('~')
credential_dir = os.path.join(home_dir, '.credentials')
if not os.path.exists(credential_dir):
os.makedirs(credential_dir)
credential_path = os.path.join(credential_dir,
'sheets.googleapis.com-python-quickstart.json')
store = Storage(credential_path)
credentials = store.get()
if not credentials or credentials.invalid:
flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
flow.user_agent = APPLICATION_NAME
if flags:
credentials = tools.run_flow(flow, store, flags)
else: # Needed only for compatibility with Python 2.6
credentials = tools.run(flow, store)
print('Storing credentials to ' + credential_path)
return credentials
As a general idea, there seems to be a problem with the refresh of your access token in between calls.
That either means that some of your credentials are not passed correctly or there is some problem with your local machine's time (although seems less likely than the first option)
I suggest researching the possibilities stated in this issue: https://github.com/google/oauth2client/issues/451:
(less likely) Why don't you try to force a clock update with ntpdate. Install ntp and give it a try, because a user stated that worked for him
Ok. After a loong research I guess I found out the problem. In fact, refresh_token was missing from the user credential, but the issue was tricky.
The refresh token is given for the FIRST time when the application asks the user for permissions. The refresh token is given ONLY IF the flow's step 1 includes the parameters approval_prompt="force"
For some reason the user (me) hadn't got refresh_token in user's credentials, so I revoked permissions from the application on My Account -> Security -> Applications, and restarted the OAuth dance again. Now I got refresh_token.
Update for #2 option:
Following this guide and the recommendation stated above, I believe that you must add to this code snippet (taken directly from the guide):
# Create a state token to prevent request forgery.
# Store it in the session for later validation.
state = hashlib.sha256(os.urandom(1024)).hexdigest()
session['state'] = state
# Set the client ID, token state, and application name in the HTML while
# serving it.
response = make_response(
render_template('index.html',
CLIENT_ID=CLIENT_ID,
STATE=state,
APPLICATION_NAME=APPLICATION_NAME))
the prompt=consent line and then execute the third step of the quoted response above (option 2).
Another option is to use approval_prompt=force but you must choose between the two, because they don't work well together.
Good luck :)
Some thoughts ....
The timekit.io blog link you posted is quite good in running through the alternatives. In your case, it doesn't sound like it's time related and ntp is overkill. As long as your time is more or less current, you'll be fine.
Your code runs, then fails after an hour with "invalid grant". That means that something is trying to use a Refresh Token to generate a new Access Token and failing. If you are doing your own refresh, obviously check that you are correctly retrieving and using the Refresh Token. My guess is that you're relying on the Google python library to do this (ugh I hate libraries lol).
So, for me the most likely causes are:-
The Refresh Token isn't being correctly saved and restored to use for the refresh
The Refresh Token is stale (ie. more than 25 Refresh Tokens old). This can happen if you run repeated tests during development. Always make sure you are using the most recent RT.
The Refresh Token is null because it is only provided the first time a user authorizes your application. Try going into https://myaccount.google.com/permissions?pli=1 and removing permission to your app, then start again.
If you can capture an http trace, it will help a lot in debugging. See if the python library has any debug/log features you can turn on (I know the Java library does).
Related
I have a python script that sends emails with attachments using GMAIL's API. Each time(mostly after a day) I run the script, I get an error that the token's invalid.
The only solution I have identified so far is to download the json file each time I run the script but I was expecting this to be done only once as I intend to convert the script to a desktop application.
Google sends you an authToken and a RefreshToken, who need to be stored to refresh your token when he is no longer valid.
Check that :
https://developers.google.com/identity/protocols/oauth2
There are two types of tokens access tokens and refresh tokens.
Access tokens are only good for an hour. Refresh tokens are long lived and should work until the access has been removed. Or if your application is still in the testing phase then the token will only work for seven days. There is one other thing, if you are creating your tokens from google oauth2 playground I bleave they are only good for three hours give or take.
The best solution for all of the above is to ensure that your app is first off set to prodctuion, and second that you are properly storing your token after you have created it.
In the sample below the token is stored in the token.json for later use.
def Authorize(credentials_file_path, token_file_path):
"""Shows basic usage of authorization"""
try:
credentials = None
# The file token.json stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists(token_file_path):
try:
credentials = Credentials.from_authorized_user_file(token_file_path, SCOPES)
credentials.refresh(Request())
except google.auth.exceptions.RefreshError as error:
# if refresh token fails, reset creds to none.
credentials = None
print(f'An refresh authorization error occurred: {error}')
# If there are no (valid) credentials available, let the user log in.
if not credentials or not credentials.valid:
if credentials and credentials.expired and credentials.refresh_token:
credentials.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
credentials_file_path, SCOPES)
credentials = flow.run_local_server(port=0)
# Save the credentials for the next run
with open(token_file_path, 'w') as token:
token.write(credentials.to_json())
except HttpError as error:
# Todo handle error
print(f'An authorization error occurred: {error}')
return credentials
if __name__ == '__main__':
creds = Authorize('C:\\YouTube\\dev\\credentials.json', "token.json")
I'm trying to create a python script that continuously reads mail from a service account in my organization. I'm attempting to use the Microsoft Graph API, but the more I read, the more confused I get. I have registered an app in Azure Portal and have my client id, client secret, etc, then it's my understanding you have to use those, call the API that requires you to paste a url into your browser to log in to consent access, and that provides a token that only lasts an hour? How can I do this programmatically?
I guess my question is, has anyone had any luck doing this with the graph api? How can I do this without having to do the browser handshake every hour? I would like to be able to just run this script and let it run without worrying about needing to refresh a token ever so often. Am I just dumb, or is this way too complicated lol. Any python examples on how people are authenticating to the graph api and staying authenticated would be greatly appreciated!
I was just working on something similar today. (Microsoft recently deprecated basic authentication for exchange, and I can no longer send mail using a simple username/password from a web application I support.)
Using the microsoft msal python library https://github.com/AzureAD/microsoft-authentication-library-for-python, and the example in sample/device_flow_sample.py, I was able to build a user-based login that retrieves an access token and refresh token in order to stay logged in (using "device flow authentication"). The msal library handles storing and reloading the token cache, as well as refreshing the token whenever necessary.
Below is the code for logging in the first time
#see https://github.com/AzureAD/microsoft-authentication-library-for-python/blob/dev/sample/device_flow_sample.py
import sys
import json
import logging
import os
import atexit
import requests
import msal
# logging
logging.basicConfig(level=logging.DEBUG) # Enable DEBUG log for entire script
logging.getLogger("msal").setLevel(logging.INFO) # Optionally disable MSAL DEBUG logs
# config
config = dict(
authority = "https://login.microsoftonline.com/common",
client_id = 'YOUR CLIENT ID',
scope = ["User.Read"],
username = 'user#domain',
cache_file = 'token.cache',
endpoint = 'https://graph.microsoft.com/v1.0/me'
)
# cache
cache = msal.SerializableTokenCache()
if os.path.exists(config["cache_file"]):
cache.deserialize(open(config["cache_file"], "r").read())
atexit.register(lambda:
open(config["cache_file"], "w").write(cache.serialize())
if cache.has_state_changed else None)
# app
app = msal.PublicClientApplication(
config["client_id"], authority=config["authority"],
token_cache=cache)
# exists?
result = None
accounts = app.get_accounts()
if accounts:
logging.info("found accounts in the app")
for a in accounts:
print(a)
if a["username"] == config["username"]:
result = app.acquire_token_silent(config["scope"], account=a)
break
else:
logging.info("no accounts in the app")
# initiate
if result:
logging.info("found a token in the cache")
else:
logging.info("No suitable token exists in cache. Let's get a new one from AAD.")
flow = app.initiate_device_flow(scopes=config["scope"])
if "user_code" not in flow:
raise ValueError(
"Fail to create device flow. Err: %s" % json.dumps(flow, indent=4))
print(flow["message"])
sys.stdout.flush() # Some terminal needs this to ensure the message is shown
# Ideally you should wait here, in order to save some unnecessary polling
input("Press Enter after signing in from another device to proceed, CTRL+C to abort.")
result = app.acquire_token_by_device_flow(flow) # By default it will block
# You can follow this instruction to shorten the block time
# https://msal-python.readthedocs.io/en/latest/#msal.PublicClientApplication.acquire_token_by_device_flow
# or you may even turn off the blocking behavior,
# and then keep calling acquire_token_by_device_flow(flow) in your own customized loop.
if result and "access_token" in result:
# Calling graph using the access token
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
print("Graph API call result: %s" % json.dumps(graph_data, indent=2))
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
You'll need to fix up the config, and update the scope for the appropriate privileges.
All the magic is in here:
result = app.acquire_token_silent(config["scope"], account=a)
and putting the Authorization access_token in the requests headers:
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
As long as you call acquire_token_silent before you invoke any graph APIs, the tokens will stay up to date. The refresh token is good for 90 days or something, and automatically updates. Once you login, the tokens will be updated and stored in the cache (and persisted to a file), and will stay alive more-or-less indefinitely (there are some things that can invalidate it on the server side).
Unfortunately, I'm still having problems because it's an unverified multi-tenant application. I successfully added the user as a guest in my tenant, and the login works, but as soon as I try to get more interesting privileges in scope, the user can't log in - I'll either have to get my mpn verified, or get my client's 3rd party IT guys admin to grant permission for this app in their tenant. If I had admin privileges for their tenant, I'd probably be looking at the daemon authentication method instead of user-based.
(to be clear, the code above is the msal example almost verbatim, with config and persistence tweaks)
I'm making a desktop app in Python that sends mail from Gmail. The problem is that after receiving consent (OAuth 2) through the browser, the user for whom the software receives the consent, continues to be logged in to the browser in Gmail. Is there a way to go through the authorization process without staying logged in to Gmail in your browser?
What you are referring to is Oauth2. Oauth2 gives users the ability to grant applications like yours consent to access their private data. Private data is data that is owned by someone. My gmail data is mine your application can not use it unless I grant you access.
Is there a way to go through the authorization process without staying logged in to Gmail in your browser?
Lets clear up some confusion in this statement you mention authorization which is correct a user is authorizing your application to access their data. Yet you also mention logged in which has nothing to do with authorization. Logging in a user is authentication and is not part of Oauth2. It is part of something else called openid connect.
As for how to request authorization of a user without using the browser. Once the user has consented to your application accessing my data once then your application should have what its called a refresh token, this refresh token can be used at a latter time for your application to request a new access token. Granting you access to may data without using the browser to access my data again. So you could store this refresh token in the backend some where and use that to continue to access the users data without needing to use the browser again.
storing user credentials in an installed application
It is hard to know exactly what you are doing since you did not include any code in your question, and your question is a little unclear.
In the following example please note how the users credentials are stored in gmail.dat using this code in an installed application will cause it to load the refresh token the next time the user runs the app meaning that the consent screen should not be shown, as the credentials are already stored for that user.
def initialize_gmail():
"""Initializes the gmail service object.
Returns:
analytics an authorized gmail service object.
"""
# Parse command-line arguments.
parser = argparse.ArgumentParser(
formatter_class=argparse.RawDescriptionHelpFormatter,
parents=[tools.argparser])
flags = parser.parse_args([])
# Set up a Flow object to be used if we need to authenticate.
flow = client.flow_from_clientsecrets(
CLIENT_SECRETS_PATH, scope=SCOPES,
message=tools.message_if_missing(CLIENT_SECRETS_PATH))
# Prepare credentials, and authorize HTTP object with them.
# If the credentials don't exist or are invalid run through the native client
# flow. The Storage object will ensure that if successful the good
# credentials will get written back to a file.
storage = file.Storage('gmail.dat')
credentials = storage.get()
if credentials is None or credentials.invalid:
credentials = tools.run_flow(flow, storage, flags)
http = credentials.authorize(http=httplib2.Http())
# Build the service object.
service = build('gmail ', 'v1', http=http)
return service
I'm building a Python Flask app that gathers some information about the user's Google Tag Manager (GTM) account. To do this, I need them to authorize read only access to their GTM account, and then I go and retrieve the data I need.
The ideal flow here is:
User clicks "log in with Google"
A new tab opens up asking them to select their Google account
The next page tells them what permissions I'm looking for, and to allow access
They accept, the auth flow is complete and the tab closes.
I've managed to get the app running locally with the desired flow, no problem.
However, once I deployed it to Heroku, I hit a roadblock, in that the auth flow simply doesn't work. Heroku's logs read:
If your browser is on a different machine then exit and re-run this
application with the command-line parameter
--noauth_local_webserver
I dug a bit, added args.noauth_local_webserver = True as a parameter to the Oauth Flow object.
However, that creates a new problem, where on page load, this shows up in Heroku's logs:
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?client_id=629932623162-nguh8ssud6her0e6bbc5vloe1k3aq95q.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Ftagmanager.readonly&access_type=offline&response_type=code&prompt=select_account
Enter verification code:
This works in my local environment as I can visit that page and paste the verification code into my console, but isn't really useful on Heroku. I'm not sure where I'd paste the verification code, and this certainly doesn't make sense if someone other than me is going to try and log on.
Does anyone know how I can get the user's web browser to open the auth flow automatically, rather than having it show up in the console?
If it helps, here's the GetService() function I'm using (pulled from Google's docs):
def GetService():
# Parse command-line arguments.
parser = argparse.ArgumentParser(
formatter_class=argparse.RawDescriptionHelpFormatter,
parents=[tools.argparser])
flags = parser.parse_args([])
flags.noauth_local_webserver = True
# Set up a Flow object to be used if we need to authenticate.
flow = client.flow_from_clientsecrets(
client_secrets_path, scope=scope,
prompt = 'consent',
message=tools.message_if_missing(client_secrets_path))
storage = file.Storage(api_name + '.dat')
credentials = storage.get()
if credentials is None or credentials.invalid:
credentials = tools.run_flow(flow, storage, flags)
http = credentials.authorize(http=httplib2.Http())
# Build the service object.
service = build(api_name, api_version, http=http)
return service
I then retrieve data from the service object using calls like this:
def retrieveAcctList():
accountObj = GetService().accounts().list(
access_token = token).execute()
which triggers the flow on my local machine.
Any help is appreciated!
I am having trouble generating a refresh token using Python for the AdWords API & need some help. Here is the situation:
I have a client on AdWords that I want to pull reports for through the AdWords API (we have a developer token now for this). Let's say that, in AdWords, the clients account is 521-314-0974 (making this up). Here is where I am confused:
Below is the following code snippet needed to generate a refresh token that I am trying to get working:
"""Generates a refresh token for use with AdWords."""
__author__ = 'Nathaniel Payne'
import sys
import urllib2
from oauthlib import oauth2
# Your OAuth 2.0 Client ID and Secret. If you do not have an ID and Secret yet,
# please go to https://console.developers.google.com and create a set.
CLIENT_ID = 'INSERT_CLIENT_ID_HERE'
CLIENT_SECRET = 'INSERT_CLIENT_SECRET_HERE'
# You may optionally provide an HTTPS proxy.
HTTPS_PROXY = None
# The AdWords API OAuth 2.0 scope.
SCOPE = u'https://adwords.google.com/api/adwords'
# This callback URL will allow you to copy the token from the success screen.
CALLBACK_URL = 'urn:ietf:wg:oauth:2.0:oob'
# The HTTP headers needed on OAuth 2.0 refresh requests.
OAUTH2_REFRESH_HEADERS = {'content-type':
'application/x-www-form-urlencoded'}
# The web address for generating new OAuth 2.0 credentials at Google.
GOOGLE_OAUTH2_AUTH_ENDPOINT = 'https://accounts.google.com/o/oauth2/auth'
GOOGLE_OAUTH2_GEN_ENDPOINT = 'https://accounts.google.com/o/oauth2/token'
def main():
oauthlib_client = oauth2.WebApplicationClient(CLIENT_ID)
authorize_url = oauthlib_client.prepare_request_uri(
GOOGLE_OAUTH2_AUTH_ENDPOINT, redirect_uri=CALLBACK_URL, scope=SCOPE)
print ('Log in to your AdWords account and open the following URL: \n%s\n' %
authorize_url)
print 'After approving the token enter the verification code (if specified).'
code = raw_input('Code: ').strip()
post_body = oauthlib_client.prepare_request_body(
client_secret=CLIENT_SECRET, code=code, redirect_uri=CALLBACK_URL)
if sys.version_info[0] == 3:
post_body = bytes(post_body, 'utf8')
request = urllib2.Request(GOOGLE_OAUTH2_GEN_ENDPOINT, post_body,
OAUTH2_REFRESH_HEADERS)
if HTTPS_PROXY:
request.set_proxy(HTTPS_PROXY, 'https')
raw_response = urllib2.urlopen(request).read().decode()
oauth2_credentials = oauthlib_client.parse_request_body_response(raw_response)
print ('Your access token is %s and your refresh token is %s'
% (oauth2_credentials['access_token'],
oauth2_credentials['refresh_token']))
print ('You can cache these credentials into a yaml file with the '
'following keys:\nadwords:\n client_id: %s\n client_secret: %s\n'
' refresh_token: %s\n'
% (CLIENT_ID, CLIENT_SECRET, oauth2_credentials['refresh_token']))
if __name__ == '__main__':
main()
Questions:
1) Do I need to have a special project set-up for every AdWords customer in the console.developers.google.com, in order to pull from the AdWords Reporting API? Or, can I simply provide the client secret and ID for a generic account in the console?
2) Following from this, can someone please confirm what should go in place of the client_ID & Client_Secret in order to make the Python code block below work. What I mean is, I was using the client ID and client secret from https://console.developers.google.com ... for the analytics account that we have billing set-up on (and which I have used for BigQuery API access previously). Is that correct? I am not seeing clearly how this will be linked to the AdWords account for this client.
2) In the consent screen, I put my own e-mail, since I am owner of the project,. That said, when I run the code, I get the link to the URL that I need to run to generate the code. That said, when I sun this snippet:
print ('Log in to your AdWords account and open the following URL: \n%s\n' %
authorize_url)
print 'After approving the token enter the verification code (if specified).'
code = raw_input('Code: ').strip()
I get an error. This is the message that I get in error:
Error: redirect_uri_mismatch
The redirect URI in the request: urn:ietf:wg:oauth:2.0:oob did not match a registered redirect URI
Learn more
Request Details
cookie_policy_enforce=false
scope=https://adwords.google.com/api/adwords
response_type=code
access_type=online
redirect_uri=urn:ietf:wg:oauth:2.0:oob
display=page
client_id=XXXXXXXXX.apps.googleusercontent.com
I am puzzled here. Some folks suggested changing the e-mail address in the consent screen (which I did ... but was unsuccessful). Again, my simple goal is to be able to pull one report from tis clients through the AdWords API (which I will expand once I get there). Any help would be appreciated. Cheers.
After some work, I was able to successfully navigate through this issue. Here are the detailed steps that I took to get to the point where I could successfully pull data through the API. In my situation, I manage an AdWords MCC with multiple accounts. Thus, I went back to the beginning of many of the help manuals and did the following:
Create a new project called AdWords-API-XXXX.
In the credentials screen on the console, I created a new "Client ID for native application". This allowed me to generate my CLIENT_ID and the CLIENT_SECRET that I needed. Critically, it also generated a re-direct URI which was the source of my problem.
I took both of these values, added them to the main script, and ran the generate_refresh_token.py script. This allowed me to generate a working refresh token. I had to be signed into my AdWords account MCC, in order to make sure that OAuth2 provided me the ability to access all potential AdWord clientsinside my MCC. I got an authentication screen generated by URL for this process which asked me to confirm that permission was being granted for AdWords access.
Following this, I created a new googleads.yaml script and placed this in my c:\gsutil directory. This is the code in most Python programs where the program looks for the file googleads.yaml:
adwords_client = adwords.AdWordsClient.LoadFromStorage()
Once this was done, I was able to successfully run the script from my command line to generate the final output. The script was:
python download_criteria_report.py
Note of course that I have changed my path variable previously in order to run Python 2.7 from the command line. This script was run inside the directory of the download_criteria_report.py file. This script ran successfully and enabled me to pull data from the AdWords API for one of my test clients.
The next challenge will be working with the returned output from the API and putting it into a format that I can quickly use for analysis & storage.