I have a Json service file, and a service account that already accesses translate and sheets, but it will not access user lists. The result is either 400 showing its confused or 401 saying its not authorized. Examples are usually about client involved OAuth processes, where I need server to server. I have enabled that "Enable G Suite domain-wide delegation" feature on the service account too.
I read and tried the JWT method, but I get the same error responses.
https://developers.google.com/identity/protocols/oauth2/service-account#python_2
My goal is to call either one of these end points
https://www.googleapis.com/admin/directory/v1/users
https://www.googleapis.com/admin/directory/v1/users.readonly
Any direction would be greatly appreciated.
UPDATE1:
This is using the Jwt token approach which yields error 401.
with open(CLIENT_SECRET_FILE, "r+") as fh:
config = json.load(fh)
iat = time.time()
exp = iat + 3600
payload = {'iss': config['client_email'],
'sub': config['client_email'],
'aud': 'https://www.googleapis.com/',
'iat': iat,
'exp': exp}
additional_headers = {'kid': config['private_key_id']}
signed_jwt = jwt.encode(payload, config['private_key'], headers=additional_headers,
algorithm='RS256')
url = 'https://www.googleapis.com/admin/directory/v1/users'
headers = {"Authorization": "Bearer " + signed_jwt}
r = requests.get(url, headers=headers)
I have also tried
scopes = ['https://www.googleapis.com/auth/admin.directory.user']
credentials = ServiceAccountCredentials.from_json_keyfile_name(CLIENT_SECRET_FILE, scopes=scopes)
service = build('admin', 'directory_v1', credentials=credentials)
results = service.users().list().execute()
UPDATE2:
This link has great information and simple code to review. As much as I tried to avoid impersonation, the AdminSDK requires it. That makes integrations a bit awkward in my view. In addition, the issue I also faced was the Domain-Wide-Delegation screen in the Google Workspace Admin can get messed up. Deleting the entry and recreating it fixed the forever 403 error I kept getting no matter what I had tried.
https://gist.github.com/lewisrodgers/fa766ebd37c1397d3a014eb73805edd1
You need to incorporate into your code impersonation, so that the service account acts on behalf of the admin
Because only admins have authroization to access Resource: users.
For impersonation in Python you need to implement the additional line
delegated_credentials = credentials.with_subject('admin#example.org')
The link below has great information and simple code to review. As much as I tried to avoid impersonation, the AdminSDK requires it. That makes integrations a bit awkward in my view.
In addition, the issue I also faced was the Domain-Wide-Delegation screen in the Google Workspace Admin that messed up. After much digging over weeks, I found a simple solution of deleting and recreating the client entry in that screen. It fixed the never ending 403 error that hit every test I tried that should have worked and did for many others.
This seems to be the only API set by Google that requires impersonation, and is annoying when attempting to create a SaaS solution.
Really basic, trimmed examples, and decent article references.
https://gist.github.com/lewisrodgers/fa766ebd37c1397d3a014eb73805edd1
Related
I have built a python application to access read only Power BI Rest API’s. I am automating the collection of tenant activity. However despite configuring my Azure App and using the service principal to generate an access token, the response I receive from the API request is one of an unauthorised response:
{"error": {"code": "PowerBINotAuthorizedException", "pbi.error": {"code":
"PowerBINotAuthorizedException", "parameters": {}, "details": [], "exceptionCulprit": 1}}}
I have found a number of similar issues posted online, however feel that I have done everything that is suggested but am still not able to get it working. I would appreciate any guidance.
The steps that I have taken are:
Configured an Azure App, adding the Application Permission for Power Bi Service-Tenant.Read.All
Screenshot of App Settings in Azure Portal
Requested my access token based upon the Client Credentials Flow using my app's client_ID and client_Secret as documented in the below link:
https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow
I successfully receive a token using the script below:
import requests
azureTenantID = "xxxxxxxxxxxxxxxxx"
azureClientId = "xxxxxxxxxxxxxxxxx"
azureClientSecret = "xxxxxxxxxxxxxxxxxx"
url = f"https://login.microsoftonline.com/{azureTenantID}/oauth2/v2.0/token"
payload = {
"grant_type": "client_credentials",
"client_id": azureClientId,
"client_secret": azureClientSecret,
"scope": "https://analysis.windows.net/powerbi/api/.default"
}
# Header HAS to be x-www-form-urlencoded for MS to accept it.
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
# Return POST content as JSON.
r = requests.post(url, data=payload, headers=headers).json()
# Grab the access token.
response = r.get("access_token")
# Concatenate with Bearer string
access_token = "Bearer {r['access_token']}"
Configured my Power BI Tenant Settings to enable Service Principals to use API's.
Screenshot of Admin API Setting
Screenshot of Developer API Setting
Note that I added the Service Principal as a member of the Security Group for which both of these settings are enabled
Execute my Get request to the API
The followings script returns a good response when I take an access token from the Power BI REST API Documentation's 'Try it out' feature, but not when I generate the token as above.
import requests
# Initialise parameters.
url = "https://api.powerbi.com/v1.0/myorg/admin/groups?$top=1000&$expand=datasets,dataflows,reports,users,dashboards"
headers = {'Authorization': get_access_token2()}
# Get response.
response = requests.get(url, headers=headers)
response = response.json()
Any assistance would be appreciated !
I just went through this exact scenario that you described, and in the end we had to engage Microsoft support to solve it.
Although extremely counter intuitive, if the app that you create for your service principal authentication has any Power BI permissions assigned to it then the access token that is generated (when passed to Power BI REST Admin API) will return an error response that reports PowerBINotAuthorizedException.
To be even more specific, if the access token that you pass to the Power BI API has a roles key/value pair, then you will get a PowerBINotAuthorizedException.
In your case, the issue is easier because you have listed out what permissions you granted. You mentioned that you Configured an Azure App, adding the Application Permission for Power Bi Service-Tenant.Read.All. In order to resolve this issue, you will need to remove that permission.
For future readers, you can troubleshoot this by decoding your access token using a JWT token decoder like one found at jstoolset.com. If your app has permissions allocated to the scope that you have requested (https://analysis.windows.net/powerbi/api/.default is the typical Power BI scope that you request in your authorization) and you decode your JWT token then you will see a roles key/value pair. The presence of this roles is essentially the issue. It does not matter that the values there might match up to the Required Scope in the Power BI REST Admin API documentation. It was described to us as if there is a roles value in your access token then when the token is presented to the Power BI API the roles that are granted are attempted to be used, which ultimately results in a PowerBINotAuthorizedException because service principals are not allowed to use a certain role.
If you have an app that you have removed all permissions from, but still has a value coming through in your access token for the roles key/value pair, then I would suggest starting with a new app with no permissions allocated to it, and simply add the new app to the existing security group that you originally created. This is how we realized that this truly was the issue, and were then able to reconcile from there.
EDIT: Microsoft has now updated their API documentation on the relevant endpoints to reflect this information. For example, in Admin - Groups GetGroupUsersAsAdmin the Required Scope now reads:
Tenant.Read.All or Tenant.ReadWrite.All
Relevant only when authenticating via a standard delegated admin access token. Must not be present when authentication via a service principal is used.
I have a Python script that is running periodically on an AWS EC2 Ubuntu machine.
This script reads data from some files and sometimes changes data in them.
I want to download these files from OneDrive, do my own thing with them, and upload them back to OneDrive.
I want this to be done automatically, without the need for a user to approve any login or credentials. I'm ok with doing it once (i.e. approving the login on the first run) but the rest has to run automatically, without asking ever again for approvals (unless the permissions change, of course).
What is the best way to do this?
I've been reading the documentation on Microsoft Graph API but I'm struggling with the authentication part. I've created an application in Azure AAD, gave the sample permissions (to test) and created a secret credential.
I managed to do it. I'm not sure if it's the best way but it is working now. It's running automatically every hour and I don't need to touch it.
I followed the information on https://learn.microsoft.com/en-gb/azure/active-directory/develop/v2-oauth2-auth-code-flow
This is what I did.
Azure Portal
Create an application. Azure Active Directory -> App Registrations -> Applications from personal account
In Supported account types, choose the one that has personal Microsoft accounts.
In Redirect URI, choose Public client/native. We'll add the specific URI later.
In the application details, in the section Overview, take note of the Application (client) ID. We'll need this later.
In the section Authentication, click Add a Platform and choose Desktop + devices. You can use your own, I chose one of the suggested: https://login.microsoftonline.com/common/oauth2/nativeclient
In the section API permissions, you have to add all the permissions that your app will use. I added User.Read, Files.ReadWrite and offline_access. The offline_access is to be able to get the refresh token, which will be crucial to keep the app running without asking the user to login.
I did not create any Certificate or Secret.
Web
Looks like to get a token for the first time we have to use a browser or emulate something like that.
There must be a programmatic way to do this, but I had no idea how to do it. I also thought about using Selenium for this, but since it's only one time and my app will request tokens every hour (keeping the tokens fresh), I dropped that idea.
If we add new permissions, the tokens that we have will become invalid and we have to do this manual part again.
Open a browser and go to the URL below. Use the Scopes and the Redirect URI that you set up in Azure Portal.
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=your_app_client_id&response_type=code&redirect_uri=https%3A%2F%2Flogin.microsoftonline.com%2Fcommon%2Foauth2%2Fnativeclient&response_mode=query&scope=User.Read%20offline_access%20Files.ReadWrite
That URL will redirect you to the Redirect URI that you set up and with a code=something in the URL. Copy that something.
Do a POST request with type FORM URL Encoded. I used https://reqbin.com/ for this.
Endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token
Form URL: grant_type=authorization_code&client_id=your_app_client_id&code=use_the_code_returned_on_previous_step
This will return an Access Token and a Refresh Token. Store the Refresh Token somewhere. I'm saving it in a file.
Python
# Build the POST parameters
params = {
'grant_type': 'refresh_token',
'client_id': your_app_client_id,
'refresh_token': refresh_token_that_you_got_in_the_previous_step
}
response = requests.post('https://login.microsoftonline.com/common/oauth2/v2.0/token', data=params)
access_token = response.json()['access_token']
new_refresh_token = response.json()['refresh_token']
# ^ Save somewhere the new refresh token.
# I just overwrite the file with the new one.
# This new one will be used next time.
header = {'Authorization': 'Bearer ' + access_token}
# Download the file
response = requests.get('https://graph.microsoft.com/v1.0/me/drive/root:' +
PATH_TO_FILE + '/' + FILE_NAME + ':/content', headers=header)
# Save the file in the disk
with open(file_name, 'wb') as file:
file.write(response.content)
So basically, I have the Refresh Token always updated.
I call the Token endpoint using that Refresh Token, and the API gives me an Access Token to use during the current session and a new Refresh Token.
I use this new Refresh Token the next time I run the program, and so on.
I've just published a repo which does this. Contributions and pull requests welcome:
https://github.com/stevemurch/onedrive-download
I'm trying to get this example to work from https://github.com/ozgur/python-linkedin. I'm using his example. When I run this code. I don't get the RETURN_URL and authorization_code talked about in the example. I'm not sure why, I think it is because I'm not setting up the HTTP API example correctly. I can't find http_api.py, and when I visit http://localhost:8080, I get a "this site can't be reached".
from linkedin import linkedin
API_KEY = 'wFNJekVpDCJtRPFX812pQsJee-gt0zO4X5XmG6wcfSOSlLocxodAXNMbl0_hw3Vl'
API_SECRET = 'daJDa6_8UcnGMw1yuq9TjoO_PMKukXMo8vEMo7Qv5J-G3SPgrAV0FqFCd0TNjQyG'
RETURN_URL = 'http://localhost:8000'
authentication = linkedin.LinkedInAuthentication(API_KEY, API_SECRET, RETURN_URL, linkedin.PERMISSIONS.enums.values())
# Optionally one can send custom "state" value that will be returned from OAuth server
# It can be used to track your user state or something else (it's up to you)
# Be aware that this value is sent to OAuth server AS IS - make sure to encode or hash it
#authorization.state = 'your_encoded_message'
print authentication.authorization_url # open this url on your browser
application = linkedin.LinkedInApplication(authentication)
http_api.py is one of the examples provided in the package. This is an HTTP server that will handle the response from LinkedIn's OAuth end point, so you'll need to boot it up for the example to work.
As stated in the guide, you'll need to execute that example file to get the server working. Note you'll also need to supply the following environment variables: LINKEDIN_API_KEY and LINKEDIN_API_SECRET.
You can run the example file by downloading the repo and calling LINKEDIN_API_KEY=yourkey LINKEDIN_API_SECRET=yoursecret python examples/http_api.py. Note you'll need Python 3.4 for it to work.
I created 2 applications in my Azure directory, 1 for my API Server and one for my API client. I am using the Python ADAL Library and can successfully obtain a token using the following code:
tenant_id = "abc123-abc123-abc123"
context = adal.AuthenticationContext('https://login.microsoftonline.com/' + tenant_id)
token = context.acquire_token_with_username_password(
'https://myapiserver.azurewebsites.net/',
'myuser',
'mypassword',
'my_apiclient_client_id'
)
I then try to send a request to my API app using the following method but keep getting 'unauthorized':
at = token['accessToken']
id_token = "Bearer {0}".format(at)
response = requests.get('https://myapiserver.azurewebsites.net/', headers={"Authorization": id_token})
I am able to successfully login using myuser/mypass from the loginurl. I have also given the client app access to the server app in Azure AD.
Although the question was posted a long time ago, I'll try to provide an answer. I stumbled across the question because we had the exact same problem here. We could successfully obtain a token with the adal library but then we were not able to access the resource I obtained the token for.
To make things worse, we sat up a simple console app in .Net, used the exact same parameters, and it was working. We could also copy the token obtained through the .Net app and use it in our Python request and it worked (this one is kind of obvious, but made us confident that the problem was not related to how I assemble the request).
The source of the problem was in the end in the oauth2_client of the adal python package. When I compared the actual HTTP requests sent by the .Net and the python app, a subtle difference was that the python app sent a POST request explicitly asking for api-version=1.0.
POST https://login.microsoftonline.com/common//oauth2/token?api-version=1.0
Once I changed the following line in oauth2_client.py in the adal library, I could access my resource.
Changed
return urlparse('{}?{}'.format(self._token_endpoint, urlencode(parameters)))
in the method _create_token_url, to
return urlparse(self._token_endpoint)
We are working on a pull request to patch the library in github.
For the current release of Azure Python SDK, it support authentication with a service principal. It does not support authentication using an ADAL library yet. Maybe it will in future releases.
See https://azure-sdk-for-python.readthedocs.io/en/latest/resourcemanagement.html#authentication for details.
See also Azure Active Directory Authentication Libraries for the platforms ADAL is available on.
#Derek,
Could you set your Issue URL on Azure Portal? If I set the wrong Issue URL, I could get the same error with you. It seems that your code is right.
Base on my experience, you need add your application into Azure AD and get a client ID.(I am sure you have done this.) And then you can get the tenant ID and input into Issue URL textbox on Azure portal.
NOTE:
On old portal(manage.windowsazure.com),in the bottom command bar, click View Endpoints, and then copy the Federation Metadata Document URL and download that document or navigate to it in a browser.
Within the root EntityDescriptor element, there should be an entityID attribute of the form https://sts.windows.net/ followed by a GUID specific to your tenant (called a "tenant ID"). Copy this value - it will serve as your Issuer URL. You will configure your application to use this later.
My demo is as following:
import adal
import requests
TenantURL='https://login.microsoftonline.com/*******'
context = adal.AuthenticationContext(TenantURL)
RESOURCE = 'http://wi****.azurewebsites.net'
ClientID='****'
ClientSect='7****'
token_response = context.acquire_token_with_client_credentials(
RESOURCE,
ClientID,
ClientSect
)
access_token = token_response.get('accessToken')
print(access_token)
id_token = "Bearer {0}".format(access_token)
response = requests.get(RESOURCE, headers={"Authorization": id_token})
print(response)
Please try to modified it. Any updates, please let me know.
I am trying to complete a story assignment system for my school newspaper in Google App Engine. It'll track deadlines for writers, allow writers to pick up stories, and give an "at a glance" view of the weeks stories. My partner and I are trying to fully integrate it with our newspapers Google Apps installation. Oh, and we have to use 3 legged Oauth because we don't have Google Apps Premier.
In that endeavor, I stumbled upon Aeoid and was able to follow the instructions to make federated login work. It's very cool!
Where I'm running into trouble is using Oauth to get a list of the users google documents. I have a test page set up here: mustrun.cornellsun.com/test. It is giving me errors - I've copied them at the bottom of this mail. I don't know if this has to do with my consumer secret (should I be using the key I get from google marketplace? or should I be using the key I get from the manage domains page?). Right now I'm using the key I got from the manage domains page
Also complicating this is that the actual appspot domain is mustrun2sun [].appspot[too new can't post more than one link].com, but I set it up in google apps so that only users from my domain can log in and also so that the app is deployed on my domain. (app is deployed as must[]run[].corn[]ellsun[].[]com & everything refers to it as such, even in the manage domains thing.)
I'm using GDClient 2.0 classes so I'm fairly sure that everything should work as planned... i.e. I'm not using the old service stuff or anything. I've used htt[]p:/[]/k[]ing[]yo-bachi.blog[]spot.c[]om/2010/05/gaego[]ogleoauth.ht[]ml as a bit of a template for my Oauth "dance" because the Google examples are out of date & use the old Google data 1.0 library - I think.
The error that I'm getting when I go to my test page is
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 511, in __call__
handler.get(*groups)
File "/base/data/home/apps/mustrun2sun/1.341947133742569880/main.py", line 170, in get
feed = client.GetDocList(auth_token=gdata.gauth.AeLoad(users.get_current_user().user_id())) #auth_token=TOKEN
File "/base/data/home/apps/mustrun2sun/1.341947133742569880/gdata/docs/client.py", line 141, in get_doclist
auth_token=auth_token, **kwargs)
File "/base/data/home/apps/mustrun2sun/1.341947133742569880/gdata/client.py", line 635, in get_feed
**kwargs)
File "/base/data/home/apps/mustrun2sun/1.341947133742569880/gdata/client.py", line 308, in request
response, Unauthorized)
Unauthorized: Unauthorized - Server responded with: 401, <HTML>
<HEAD>
<TITLE>Token invalid - Invalid AuthSub token.</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Token invalid - Invalid AuthSub token.</H1>
<H2>Error 401</H2>
</BODY>
</HTML>
Also, since this is hard w/o any source code, below is the relevant code:
import gdata.auth
import gdata.gauth
import gdata.docs.client
import gdata.docs.data
import gdata.docs.service
import gdata.alt.appengine
from aeoid import middleware, users
class GetOauthToken(webapp.RequestHandler):
def get(self):
user_id = users.get_current_user().user_id()
saved_request_token = gdata.gauth.AeLoad("tmp_"+user_id)
gdata.gauth.AeDelete ("tmp_" + user_id)
request_token = gdata.gauth.AuthorizeRequestToken(saved_request_token, self.request.uri)
#upgrade the token
access_token = client.GetAccessToken(request_token)
#save the upgraded token
gdata.gauth.AeSave(access_token, user_id)
self.redirect('/test')
class Test(webapp.RequestHandler):
def get(self):
TOKEN = gdata.gauth.AeLoad(users.get_current_user().user_id())
if TOKEN:
client = gdata.docs.client.DocsClient(source=SETTINGS['APP_NAME'])
client.auth_token = gdata.gauth.AeLoad(users.get_current_user().user_id()) #could try to put back as TOKEN?
self.response.out.write('moo baby')
client.ssl = True
feed = client.GetDocList(auth_token=gdata.gauth.AeLoad(users.get_current_user().user_id())) #auth_token=TOKEN
self.response.out.write(feed)
self.response.out.write('moo boobob')
self.response.headers['Content-Type'] = 'text/plain'
for entry in feed.entry:
self.response.out.writeln(entry.title.text)
else:
# Get unauthorized request token
gdata.gauth.AeDelete(users.get_current_user().user_id())
client = gdata.docs.client.DocsClient(source=SETTINGS['APP_NAME'])
client.ssl = True # Force communication through HTTPS
oauth_callback_url = ('http://%s/get_oauth_token' %
self.request.host)
request_token = client.GetOAuthToken(
SETTINGS['SCOPES'], oauth_callback_url, SETTINGS['CONSUMER_KEY'],
consumer_secret=SETTINGS['CONSUMER_SECRET'])
gdata.gauth.AeSave(request_token, "tmp_"+users.get_current_user().user_id())
# Authorize request token
domain = None#'cornellsun.com'
self.redirect(str(request_token.generate_authorization_url(google_apps_domain=domain)))
I've been looking high and low on the web for an answer & I have not been able to find one.
I have a working python App Engine app that uses OpenID, and OAuth to get your google contacts:
http://github.com/sje397/Chess
It is running at:
http://your-move.appspot.com
Note that Aeoid is not needed anymore, since App Engine has built-in OpenID support.
I just found out wasting a couple of hours, that you get a 401 also if the URL is not correct.
In my example, I was doing
.../buzz/v1/activities/#me/#self**?&**alt=json
Instead of
.../buzz/v1/activities/#me/#self**?**alt=json
I have personally not worked with OAuth, but a few things I noticed that may (or may not) help:
The 401 error is likely an HTTP 401 error, which means that the url was valid but required authentication. This obviously is explained by the failed OAuth attempt, but it also might be important to redirect users who are not logged in to another page.
The error is occurring when you assign your feed variable. Is the auth_token parameter simply supposed to be a username?
3.You are using the line.
gdata.gauth.AeLoad(users.get_current_user().user_id())
frequently. Even though it might not be related to your auth problems, you would probably be better off making this query once and storing it in a variable. Then when you need it again, access it that way. It will improve the speed of your application.
Again, I apologize that I have had no specific OAuth experience. I just tried to scan and find some things that may spark you onto the right path.