I'm trying to create a python script that continuously reads mail from a service account in my organization. I'm attempting to use the Microsoft Graph API, but the more I read, the more confused I get. I have registered an app in Azure Portal and have my client id, client secret, etc, then it's my understanding you have to use those, call the API that requires you to paste a url into your browser to log in to consent access, and that provides a token that only lasts an hour? How can I do this programmatically?
I guess my question is, has anyone had any luck doing this with the graph api? How can I do this without having to do the browser handshake every hour? I would like to be able to just run this script and let it run without worrying about needing to refresh a token ever so often. Am I just dumb, or is this way too complicated lol. Any python examples on how people are authenticating to the graph api and staying authenticated would be greatly appreciated!
I was just working on something similar today. (Microsoft recently deprecated basic authentication for exchange, and I can no longer send mail using a simple username/password from a web application I support.)
Using the microsoft msal python library https://github.com/AzureAD/microsoft-authentication-library-for-python, and the example in sample/device_flow_sample.py, I was able to build a user-based login that retrieves an access token and refresh token in order to stay logged in (using "device flow authentication"). The msal library handles storing and reloading the token cache, as well as refreshing the token whenever necessary.
Below is the code for logging in the first time
#see https://github.com/AzureAD/microsoft-authentication-library-for-python/blob/dev/sample/device_flow_sample.py
import sys
import json
import logging
import os
import atexit
import requests
import msal
# logging
logging.basicConfig(level=logging.DEBUG) # Enable DEBUG log for entire script
logging.getLogger("msal").setLevel(logging.INFO) # Optionally disable MSAL DEBUG logs
# config
config = dict(
authority = "https://login.microsoftonline.com/common",
client_id = 'YOUR CLIENT ID',
scope = ["User.Read"],
username = 'user#domain',
cache_file = 'token.cache',
endpoint = 'https://graph.microsoft.com/v1.0/me'
)
# cache
cache = msal.SerializableTokenCache()
if os.path.exists(config["cache_file"]):
cache.deserialize(open(config["cache_file"], "r").read())
atexit.register(lambda:
open(config["cache_file"], "w").write(cache.serialize())
if cache.has_state_changed else None)
# app
app = msal.PublicClientApplication(
config["client_id"], authority=config["authority"],
token_cache=cache)
# exists?
result = None
accounts = app.get_accounts()
if accounts:
logging.info("found accounts in the app")
for a in accounts:
print(a)
if a["username"] == config["username"]:
result = app.acquire_token_silent(config["scope"], account=a)
break
else:
logging.info("no accounts in the app")
# initiate
if result:
logging.info("found a token in the cache")
else:
logging.info("No suitable token exists in cache. Let's get a new one from AAD.")
flow = app.initiate_device_flow(scopes=config["scope"])
if "user_code" not in flow:
raise ValueError(
"Fail to create device flow. Err: %s" % json.dumps(flow, indent=4))
print(flow["message"])
sys.stdout.flush() # Some terminal needs this to ensure the message is shown
# Ideally you should wait here, in order to save some unnecessary polling
input("Press Enter after signing in from another device to proceed, CTRL+C to abort.")
result = app.acquire_token_by_device_flow(flow) # By default it will block
# You can follow this instruction to shorten the block time
# https://msal-python.readthedocs.io/en/latest/#msal.PublicClientApplication.acquire_token_by_device_flow
# or you may even turn off the blocking behavior,
# and then keep calling acquire_token_by_device_flow(flow) in your own customized loop.
if result and "access_token" in result:
# Calling graph using the access token
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
print("Graph API call result: %s" % json.dumps(graph_data, indent=2))
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
You'll need to fix up the config, and update the scope for the appropriate privileges.
All the magic is in here:
result = app.acquire_token_silent(config["scope"], account=a)
and putting the Authorization access_token in the requests headers:
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
As long as you call acquire_token_silent before you invoke any graph APIs, the tokens will stay up to date. The refresh token is good for 90 days or something, and automatically updates. Once you login, the tokens will be updated and stored in the cache (and persisted to a file), and will stay alive more-or-less indefinitely (there are some things that can invalidate it on the server side).
Unfortunately, I'm still having problems because it's an unverified multi-tenant application. I successfully added the user as a guest in my tenant, and the login works, but as soon as I try to get more interesting privileges in scope, the user can't log in - I'll either have to get my mpn verified, or get my client's 3rd party IT guys admin to grant permission for this app in their tenant. If I had admin privileges for their tenant, I'd probably be looking at the daemon authentication method instead of user-based.
(to be clear, the code above is the msal example almost verbatim, with config and persistence tweaks)
Related
I've seen lots of related questions to this one, but none of the answers have helped me.
First, I went to the Twitter Developer Portal and set up my OAuth2.0 Client ID and Secret:
Then, I used Authlib to set up a OAuth2Session with an Authorization Code flow, documented here:
authorization_url = 'https://api.twitter.com/2/oauth2/authorize'
scopes = ['tweet.read', 'tweet.write', 'users.read']
scope = ' '.join(scopes)
Then I set up an OAuth2 client using the ID and secret, create the authorization URL, and print the URI for use:
client = OAuth2Session(client_id, client_secret, scope=scope)
uri, state = client.create_authorization_url(authorization_url)
print("> Open this in your browser: " + uri)
In the terminal, it prints
> Open this in your browser: https://api.twitter.com/2/oauth2/authorize?response_type=code&client_id=[redacted]&scope=tweet.read+tweet.write+users.read&state=L21JiPkirR8awYs7kcDjFC4jrPj68x
But opening the link in a browser displays the Twitter-produced error
{"errors":[{"message":"Bad Authentication data","code":215}]}
which is all over Stack Overflow. So I tried
logging out of Twitter again,
opening the link on various browsers,
removing the scope parameter from the OAuth2Session object,
changing the scope join character from ' ' to '%20' and '+',
double-checking and regenerating Client ID + Secret,
removing Authlib and implementing the same code using just Python requests
I believe another answer (and others like these) fail to help in my situation because Authlib's OAuth2Session should be correctly set up and authenticating via OAuth2 (unless there's a problem with the library).
So, after that, I tried adding the callback URI through the client. I used a local callback URI with a Flask app. This is the same Callback URI I set up in the Twitter developer portal. Although this probably isn't done correctly, I don't believe it's causing Error 215. I thought I'd include this just in case:
callback_uri = 'http://127.0.0.1:5000/oauth/callback'
client = OAuth2Session(client_id, client_secret, scope=scope, redirect_uri=callback_uri)
from flask import Flask
app = Flask(__name__)
#app.route("/oauth/callback", methods=["GET"])
def callback():
print('Callback')
I could be missing something, or perhaps my data is improperly formatted in some way. This has proven to be a frustrating issue.
If the type of library is not important to you and you only want the output, I recommend you to use the Tweepy library, because it has a mode that does not require a callback and only works with pin. Pay attention to the following example:
import tweepy
auth = tweepy.OAuthHandler(
"CONSUMER_KEY",
"CONSUMER_SECRET",
callback='oob'
)
auth_url = auth.get_authorization_url()
print("\n", auth_url, "\n")
verifier_value = input("Enter Authorization Pin: ")
auth.get_access_token(verifier_value)
text = f"Consumer key: {CONSUMER_KEY}\n"
text += f"Consumer secret: {CONSUMER_SECRET}\n"
text += f"Access token: {auth.access_token}\n"
text += f"Access token secret: {auth.access_token_secret}\n"
print(text)
Whenever you set the value of the callback parameter to oob, Twitter authenticates will be changed
Here's my problem:
I have a 365 Family OneDrive subscription with 3 members, my account being the admin.
I am trying to build a python application to read/extract the content of the files I have on this onedrive space based on specific criterias. I want to build it as a command line application, running locally on my PC. I am aware some tools may exist for this but I'd like to code my own solution.
After going through tons of different documentation, I ended up doing the following
Registered my application on the Azure portal
Granted some permission on the Microsoft Graph API (User.read, Files.Read and Files.ReadAll)
Created a secret
Grabbed the sample code provided by Microsoft
Replaces some variables with my Client_Id and Secret
Ran the code
The code returns an access token but the authorization requests fails with 401 - Unauthorized: Access is denied due to invalid credentials.
Here's the Python code I'm using.
import msal
config = {
"authority": "https://login.microsoftonline.com/consumers",
"client_id": "<my client ID>",
"scope": ["https://graph.microsoft.com/.default"],
"secret": "<My secret stuff>",
"endpoint": "https://graph.microsoft.com/v1.0/users"
}
# Create a preferably long-lived app instance which maintains a token cache.
app = msal.ConfidentialClientApplication(
config["client_id"], authority=config["authority"],
client_credential=config["secret"],
)
result = None
result = app.acquire_token_silent(config["scope"], account=None)
if not result:
result = app.acquire_token_for_client(scopes=config["scope"])
if "access_token" in result:
# Calling graph using the access token
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']}, ).json()
print("Graph API call result: ")
print(json.dumps(graph_data, indent=2))
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
According to the error message, I'm obviously missing something in the authorization process but can't tell what. I'm not even sure about the Authority and Endpoints I should use. My account being a personal one, I have no tenant.
Do I need to set-up / configure some URI somewhere?
Any help would be welcome.
Thank you in advance.
In your client app you need to store the token that you are getting from the MSAL. and then send the token with an authorized request.
For OneDrive, download the OneDrive for python. You can see the different option for Authentication.
The reason you are getting an access token, ID token, and a refresh token is because of the flow you're using. My suggestion is to review the flows for a better understanding of how the authentication process works and what will be returned accordingly. You can use this MSAL library for python.
I have looked at the other question regarding this topic and it doesn't seem to match my error. I'm getting the error when running Google Sheets APIv4:
raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
HttpAccessTokenRefreshError: invalid_grant
Error occurs on the line service.spreadsheets().values().get(spreadsheetId=key, range=ranges).execute()
This error only pops up sometimes. If I don't do anything and just run the code again. It will take me through the authentication flow process again and I get
Authentication successful.
Storing credentials to C:\Users\jason\.credentials\sheets.googleapis.com-python-quickstart.json
After which, I can run any code for a while until the same HttpAccessTokenRefreshError: invalid_grant pops up again and I have to reauthenticate again.
How do I prevent this?
I'm using the code found developers.google.com/sheets/api/quickstart/python.
I've tried to use ntp to sync time with the following
import time
import os
try:
import ntplib
client = ntplib.NTPClient()
response = client.request('pool.ntp.org')
os.system('date ' + time.strftime('%m%d%H%M%Y.%S',time.localtime(response.tx_time)))
except:
print('Could not sync with time server.')
print('Done.')
but getting:
The system cannot accept the date entered.
Enter the new data: (mm-dd-yy)
After I enter the current date, nothing happens.
I have also looked at this page. https://blog.timekit.io/google-oauth-invalid-grant-nightmare-and-how-to-fix-it-9f4efaf1da35#.5utz2vcn6
This problem also arises when I run code that is longer than 1 hour to finish. On the refresh token. It always bomb.
Now I'm thinking, tokens granted only lasts one hour and on refreshes, it always bombs.
I have posted the code for connecting:
class APIv4:
def __init__(self):
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
discoveryUrl = ('https://sheets.googleapis.com/$discovery/rest?'
'version=v4')
self.service = discovery.build('sheets', 'v4', http=http,
discoveryServiceUrl=discoveryUrl)
def get_credentials():
"""Gets valid user credentials from storage.
If nothing has been stored, or if the stored credentials are invalid,
the OAuth2 flow is completed to obtain the new credentials.
Returns:
Credentials, the obtained credential.
"""
home_dir = os.path.expanduser('~')
credential_dir = os.path.join(home_dir, '.credentials')
if not os.path.exists(credential_dir):
os.makedirs(credential_dir)
credential_path = os.path.join(credential_dir,
'sheets.googleapis.com-python-quickstart.json')
store = Storage(credential_path)
credentials = store.get()
if not credentials or credentials.invalid:
flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
flow.user_agent = APPLICATION_NAME
if flags:
credentials = tools.run_flow(flow, store, flags)
else: # Needed only for compatibility with Python 2.6
credentials = tools.run(flow, store)
print('Storing credentials to ' + credential_path)
return credentials
As a general idea, there seems to be a problem with the refresh of your access token in between calls.
That either means that some of your credentials are not passed correctly or there is some problem with your local machine's time (although seems less likely than the first option)
I suggest researching the possibilities stated in this issue: https://github.com/google/oauth2client/issues/451:
(less likely) Why don't you try to force a clock update with ntpdate. Install ntp and give it a try, because a user stated that worked for him
Ok. After a loong research I guess I found out the problem. In fact, refresh_token was missing from the user credential, but the issue was tricky.
The refresh token is given for the FIRST time when the application asks the user for permissions. The refresh token is given ONLY IF the flow's step 1 includes the parameters approval_prompt="force"
For some reason the user (me) hadn't got refresh_token in user's credentials, so I revoked permissions from the application on My Account -> Security -> Applications, and restarted the OAuth dance again. Now I got refresh_token.
Update for #2 option:
Following this guide and the recommendation stated above, I believe that you must add to this code snippet (taken directly from the guide):
# Create a state token to prevent request forgery.
# Store it in the session for later validation.
state = hashlib.sha256(os.urandom(1024)).hexdigest()
session['state'] = state
# Set the client ID, token state, and application name in the HTML while
# serving it.
response = make_response(
render_template('index.html',
CLIENT_ID=CLIENT_ID,
STATE=state,
APPLICATION_NAME=APPLICATION_NAME))
the prompt=consent line and then execute the third step of the quoted response above (option 2).
Another option is to use approval_prompt=force but you must choose between the two, because they don't work well together.
Good luck :)
Some thoughts ....
The timekit.io blog link you posted is quite good in running through the alternatives. In your case, it doesn't sound like it's time related and ntp is overkill. As long as your time is more or less current, you'll be fine.
Your code runs, then fails after an hour with "invalid grant". That means that something is trying to use a Refresh Token to generate a new Access Token and failing. If you are doing your own refresh, obviously check that you are correctly retrieving and using the Refresh Token. My guess is that you're relying on the Google python library to do this (ugh I hate libraries lol).
So, for me the most likely causes are:-
The Refresh Token isn't being correctly saved and restored to use for the refresh
The Refresh Token is stale (ie. more than 25 Refresh Tokens old). This can happen if you run repeated tests during development. Always make sure you are using the most recent RT.
The Refresh Token is null because it is only provided the first time a user authorizes your application. Try going into https://myaccount.google.com/permissions?pli=1 and removing permission to your app, then start again.
If you can capture an http trace, it will help a lot in debugging. See if the python library has any debug/log features you can turn on (I know the Java library does).
I am having trouble generating a refresh token using Python for the AdWords API & need some help. Here is the situation:
I have a client on AdWords that I want to pull reports for through the AdWords API (we have a developer token now for this). Let's say that, in AdWords, the clients account is 521-314-0974 (making this up). Here is where I am confused:
Below is the following code snippet needed to generate a refresh token that I am trying to get working:
"""Generates a refresh token for use with AdWords."""
__author__ = 'Nathaniel Payne'
import sys
import urllib2
from oauthlib import oauth2
# Your OAuth 2.0 Client ID and Secret. If you do not have an ID and Secret yet,
# please go to https://console.developers.google.com and create a set.
CLIENT_ID = 'INSERT_CLIENT_ID_HERE'
CLIENT_SECRET = 'INSERT_CLIENT_SECRET_HERE'
# You may optionally provide an HTTPS proxy.
HTTPS_PROXY = None
# The AdWords API OAuth 2.0 scope.
SCOPE = u'https://adwords.google.com/api/adwords'
# This callback URL will allow you to copy the token from the success screen.
CALLBACK_URL = 'urn:ietf:wg:oauth:2.0:oob'
# The HTTP headers needed on OAuth 2.0 refresh requests.
OAUTH2_REFRESH_HEADERS = {'content-type':
'application/x-www-form-urlencoded'}
# The web address for generating new OAuth 2.0 credentials at Google.
GOOGLE_OAUTH2_AUTH_ENDPOINT = 'https://accounts.google.com/o/oauth2/auth'
GOOGLE_OAUTH2_GEN_ENDPOINT = 'https://accounts.google.com/o/oauth2/token'
def main():
oauthlib_client = oauth2.WebApplicationClient(CLIENT_ID)
authorize_url = oauthlib_client.prepare_request_uri(
GOOGLE_OAUTH2_AUTH_ENDPOINT, redirect_uri=CALLBACK_URL, scope=SCOPE)
print ('Log in to your AdWords account and open the following URL: \n%s\n' %
authorize_url)
print 'After approving the token enter the verification code (if specified).'
code = raw_input('Code: ').strip()
post_body = oauthlib_client.prepare_request_body(
client_secret=CLIENT_SECRET, code=code, redirect_uri=CALLBACK_URL)
if sys.version_info[0] == 3:
post_body = bytes(post_body, 'utf8')
request = urllib2.Request(GOOGLE_OAUTH2_GEN_ENDPOINT, post_body,
OAUTH2_REFRESH_HEADERS)
if HTTPS_PROXY:
request.set_proxy(HTTPS_PROXY, 'https')
raw_response = urllib2.urlopen(request).read().decode()
oauth2_credentials = oauthlib_client.parse_request_body_response(raw_response)
print ('Your access token is %s and your refresh token is %s'
% (oauth2_credentials['access_token'],
oauth2_credentials['refresh_token']))
print ('You can cache these credentials into a yaml file with the '
'following keys:\nadwords:\n client_id: %s\n client_secret: %s\n'
' refresh_token: %s\n'
% (CLIENT_ID, CLIENT_SECRET, oauth2_credentials['refresh_token']))
if __name__ == '__main__':
main()
Questions:
1) Do I need to have a special project set-up for every AdWords customer in the console.developers.google.com, in order to pull from the AdWords Reporting API? Or, can I simply provide the client secret and ID for a generic account in the console?
2) Following from this, can someone please confirm what should go in place of the client_ID & Client_Secret in order to make the Python code block below work. What I mean is, I was using the client ID and client secret from https://console.developers.google.com ... for the analytics account that we have billing set-up on (and which I have used for BigQuery API access previously). Is that correct? I am not seeing clearly how this will be linked to the AdWords account for this client.
2) In the consent screen, I put my own e-mail, since I am owner of the project,. That said, when I run the code, I get the link to the URL that I need to run to generate the code. That said, when I sun this snippet:
print ('Log in to your AdWords account and open the following URL: \n%s\n' %
authorize_url)
print 'After approving the token enter the verification code (if specified).'
code = raw_input('Code: ').strip()
I get an error. This is the message that I get in error:
Error: redirect_uri_mismatch
The redirect URI in the request: urn:ietf:wg:oauth:2.0:oob did not match a registered redirect URI
Learn more
Request Details
cookie_policy_enforce=false
scope=https://adwords.google.com/api/adwords
response_type=code
access_type=online
redirect_uri=urn:ietf:wg:oauth:2.0:oob
display=page
client_id=XXXXXXXXX.apps.googleusercontent.com
I am puzzled here. Some folks suggested changing the e-mail address in the consent screen (which I did ... but was unsuccessful). Again, my simple goal is to be able to pull one report from tis clients through the AdWords API (which I will expand once I get there). Any help would be appreciated. Cheers.
After some work, I was able to successfully navigate through this issue. Here are the detailed steps that I took to get to the point where I could successfully pull data through the API. In my situation, I manage an AdWords MCC with multiple accounts. Thus, I went back to the beginning of many of the help manuals and did the following:
Create a new project called AdWords-API-XXXX.
In the credentials screen on the console, I created a new "Client ID for native application". This allowed me to generate my CLIENT_ID and the CLIENT_SECRET that I needed. Critically, it also generated a re-direct URI which was the source of my problem.
I took both of these values, added them to the main script, and ran the generate_refresh_token.py script. This allowed me to generate a working refresh token. I had to be signed into my AdWords account MCC, in order to make sure that OAuth2 provided me the ability to access all potential AdWord clientsinside my MCC. I got an authentication screen generated by URL for this process which asked me to confirm that permission was being granted for AdWords access.
Following this, I created a new googleads.yaml script and placed this in my c:\gsutil directory. This is the code in most Python programs where the program looks for the file googleads.yaml:
adwords_client = adwords.AdWordsClient.LoadFromStorage()
Once this was done, I was able to successfully run the script from my command line to generate the final output. The script was:
python download_criteria_report.py
Note of course that I have changed my path variable previously in order to run Python 2.7 from the command line. This script was run inside the directory of the download_criteria_report.py file. This script ran successfully and enabled me to pull data from the AdWords API for one of my test clients.
The next challenge will be working with the returned output from the API and putting it into a format that I can quickly use for analysis & storage.
I have used the Python Starter project, and I can add time line cards that then show up on my Glass.
What I would like to do is call the endpoints from a standalone application running on my Mac to trigger the Python logic to insert entries into the timeline.
Any ideas on where I should start?
Edit: Not sure why this was down voted. I basically wanted to insert cards to my time line from Objective C. After digging around for a while, I was able to figure this out using the Objective C libraries that Google provides for interacting with their services.
Your code which inserts the timeline items will be largely the same, but you will need to use a different flow to acquire your access token. You probably want to use the OAuth 2.0 flow for installed applications which is also document in the Python API Client Library docs.
Your Glassware might work something like this:
Create a new flow
from oauth2client.client import OAuth2WebServerFlow
...
flow = OAuth2WebServerFlow(client_id='your_client_id',
client_secret='your_client_secret',
scope='https://www.googleapis.com/auth/glass.timeline',
redirect_uri='urn:ietf:wg:oauth:2.0:oob')
Create an Auth URL and instruct the user to access it in a web browser
auth_uri = flow.step1_get_authorize_url()
print 'Please navigate here ' + auth_uri
This will yield a code. Have the user paste that code to you.
Exchange the code for a credentials
credentials = flow.step2_exchange(code)
Store those credentials for later use in a file, database, or some other persistent storage. This is how you'll insert items into your user's timeline.
Using the credentials, insert an item into their timeline
http = httplib2.Http()
http = credentials.authorize(http)
mirror_service = build("mirror", "v1", http=http)
body = {
'notification': {'level': 'DEFAULT'},
'text':'Hello world!'
}
timeline_item = mirror_service.timeline().insert(body=body).execute()