I am trying to fetch captions from YouTube video using YouTube Data API (v3)
https://developers.google.com/youtube/v3/guides/implementation/captions
So, first I tried to retrieve a captions list using this url:
https://www.googleapis.com/youtube/v3/captions?part=snippet&videoId=KK9bwTlAvgo&key={My API KEY}
I could retrieve the caption id that I'd like to download (jEDP-pmNCIqoB8QGlXWQf4Rh3faalD_l) from the above link.
Then, I followed this instruction to download the caption:
https://developers.google.com/youtube/v3/docs/captions/download
However, even though I input the caption id and my api key correctly, it shows "Login Required" error.
I suppose I need OAuth authentication, but what I am trying to do is not related to my users's account, but simply downloading public caption data automatically.
My question is: Is there any way to process OAuth authentication just once to get an access token of my own YouTube account and then reuse it whenever I need it in my application?
I can't speak to the permissions needed for the captions API in particular, but in general, yes, you can OAuth to your app once using your own account and use the access and refresh tokens to make subsequent OAuth'd requests to the API. You can find the details of generating tokens here:
https://developers.google.com/youtube/v3/guides/auth/server-side-web-apps#Obtaining_Access_Tokens
To perform the steps manually (fortunately, you only need to do this once):
If access has already been granted for an app, it needs to be removed so that new auth credentials can be established. Go to https://security.google.com/settings/security/permissions (while logged into your account) and remove access to the app. If the client ID or secret change (or you need to create one), find them at https://console.developers.google.com under API Manager.
To grant access and receive a temporary code, enter this URL in a browser:
https://accounts.google.com/o/oauth2/auth?
client_id=<client_id>&
redirect_uri=http://www.google.com&
scope=https://www.googleapis.com/auth/youtube.force-ssl&
response_type=code&
access_type=offline&
approval_prompt=force
Follow the prompt to grant access to the app.
This will redirect to google.com with a code parameter (e.g.,
https://www.google.com/?code=4/ux5gNj-_mIu4DOD_gNZdjX9EtOFf&gws_rd=ssl#). Save the code.
Send a POST request (e.g., via Postman Chrome plugin) to https://accounts.google.com/o/oauth2/token with the following in the request body:
code=<code>&
client_id=<client_id>&
client_secret=<client_secret>&
redirect_uri=http://www.google.com&
grant_type=authorization_code
The response will contain both an access token and refresh token. Save both, but particularly the refresh token (because the access token will expire in 1 hour).
You can then use the access token to send an OAuth'd request manually, following one of the options here, essentially:
curl -H "Authorization: Bearer ACCESS_TOKEN" https://www.googleapis.com/youtube/v3/captions/<id>
or
curl https://www.googleapis.com/youtube/v3/captions/<id>?access_token=ACCESS_TOKEN
(When I tried the second option for captions, however, I got the message: "The OAuth token was received in the query string, which this API forbids for response formats other than JSON or XML. If possible, try sending the OAuth token in the Authorization header instead.")
You can also use the refresh token in your code to create the credential needed when building your YouTube object. In Java, this looks like the following:
String clientId = <your client ID>
String clientSecret = <your client secret>
String refreshToken = <refresh token>
HttpTransport transport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(transport)
.setJsonFactory(jsonFactory)
.setClientSecrets(clientId, clientSecret)
.build()
.setRefreshToken(refreshToken);
try {
credential.refreshToken();
} catch (IOException e) {
e.printStackTrace();
}
youtube = new YouTube.Builder(transport, jsonFactory, credential).build();
I imagine you can do something similar in Python with the API Client Libraries, although I haven't tried Python.
Related
I have built a python application to access read only Power BI Rest API’s. I am automating the collection of tenant activity. However despite configuring my Azure App and using the service principal to generate an access token, the response I receive from the API request is one of an unauthorised response:
{"error": {"code": "PowerBINotAuthorizedException", "pbi.error": {"code":
"PowerBINotAuthorizedException", "parameters": {}, "details": [], "exceptionCulprit": 1}}}
I have found a number of similar issues posted online, however feel that I have done everything that is suggested but am still not able to get it working. I would appreciate any guidance.
The steps that I have taken are:
Configured an Azure App, adding the Application Permission for Power Bi Service-Tenant.Read.All
Screenshot of App Settings in Azure Portal
Requested my access token based upon the Client Credentials Flow using my app's client_ID and client_Secret as documented in the below link:
https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow
I successfully receive a token using the script below:
import requests
azureTenantID = "xxxxxxxxxxxxxxxxx"
azureClientId = "xxxxxxxxxxxxxxxxx"
azureClientSecret = "xxxxxxxxxxxxxxxxxx"
url = f"https://login.microsoftonline.com/{azureTenantID}/oauth2/v2.0/token"
payload = {
"grant_type": "client_credentials",
"client_id": azureClientId,
"client_secret": azureClientSecret,
"scope": "https://analysis.windows.net/powerbi/api/.default"
}
# Header HAS to be x-www-form-urlencoded for MS to accept it.
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
# Return POST content as JSON.
r = requests.post(url, data=payload, headers=headers).json()
# Grab the access token.
response = r.get("access_token")
# Concatenate with Bearer string
access_token = "Bearer {r['access_token']}"
Configured my Power BI Tenant Settings to enable Service Principals to use API's.
Screenshot of Admin API Setting
Screenshot of Developer API Setting
Note that I added the Service Principal as a member of the Security Group for which both of these settings are enabled
Execute my Get request to the API
The followings script returns a good response when I take an access token from the Power BI REST API Documentation's 'Try it out' feature, but not when I generate the token as above.
import requests
# Initialise parameters.
url = "https://api.powerbi.com/v1.0/myorg/admin/groups?$top=1000&$expand=datasets,dataflows,reports,users,dashboards"
headers = {'Authorization': get_access_token2()}
# Get response.
response = requests.get(url, headers=headers)
response = response.json()
Any assistance would be appreciated !
I just went through this exact scenario that you described, and in the end we had to engage Microsoft support to solve it.
Although extremely counter intuitive, if the app that you create for your service principal authentication has any Power BI permissions assigned to it then the access token that is generated (when passed to Power BI REST Admin API) will return an error response that reports PowerBINotAuthorizedException.
To be even more specific, if the access token that you pass to the Power BI API has a roles key/value pair, then you will get a PowerBINotAuthorizedException.
In your case, the issue is easier because you have listed out what permissions you granted. You mentioned that you Configured an Azure App, adding the Application Permission for Power Bi Service-Tenant.Read.All. In order to resolve this issue, you will need to remove that permission.
For future readers, you can troubleshoot this by decoding your access token using a JWT token decoder like one found at jstoolset.com. If your app has permissions allocated to the scope that you have requested (https://analysis.windows.net/powerbi/api/.default is the typical Power BI scope that you request in your authorization) and you decode your JWT token then you will see a roles key/value pair. The presence of this roles is essentially the issue. It does not matter that the values there might match up to the Required Scope in the Power BI REST Admin API documentation. It was described to us as if there is a roles value in your access token then when the token is presented to the Power BI API the roles that are granted are attempted to be used, which ultimately results in a PowerBINotAuthorizedException because service principals are not allowed to use a certain role.
If you have an app that you have removed all permissions from, but still has a value coming through in your access token for the roles key/value pair, then I would suggest starting with a new app with no permissions allocated to it, and simply add the new app to the existing security group that you originally created. This is how we realized that this truly was the issue, and were then able to reconcile from there.
EDIT: Microsoft has now updated their API documentation on the relevant endpoints to reflect this information. For example, in Admin - Groups GetGroupUsersAsAdmin the Required Scope now reads:
Tenant.Read.All or Tenant.ReadWrite.All
Relevant only when authenticating via a standard delegated admin access token. Must not be present when authentication via a service principal is used.
This is the code I use to retrieve idToken in my flutter app
Future<void> _handleSignIn() async {
try {
final result = await _googleSignIn.signIn();
final ggAuth = await result!.authentication;
print(ggAuth.idToken); // this is the one that I use as token value
print(ggAuth.accessToken);
} catch (error) {
print(error);
}
}
And this is the code I use to access user info in my backend.
from google.oauth2 import id_token
from google.auth.transport import requests
try:
idinfo = id_token.verify_oauth2_token(token, requests.Request(), CLIENT_ID)
userid = idinfo['sub']
print(userid)
except ValueError:
print('Invalid token)
But after replacing the token variable with the token received from flutter app and replacing the CLIENT_ID with the id generated for my app (available in the console) , it still throws the error of invalid token. What am I doing wrong?
EDIT- When I use https://jwt.io/ to decode the token it works as expected and I get all the details.
Authentication Think of the id token as your birth certificate. It just identifies you as a person.
Authorization Think of the access token as your drivers license its what contains the permissions that you have to drive a car.
An Id token does not out write give you permission to access any data on google servers. With an access token you have been authorized to access some data in this case some profile information.
The issue you are having is that you are using sign-in in. Sign in is Open id connect think of it as a birth certificate (Id token). There is a user their who is signing in to your app. They are in fount of their computer. By default with Google signin gives you get basic profile information. What you are doing with Jwt.io is checking the claims that are returned in the Id token. Id tokens are just the token that allows your application to identify that the user is logged in. There should also be a sub claim there you should use this id to map the user to your integral user system. There may be a user name and email address in the claims but you can not guarantee that they will be there every time google has stated they do not always return these claims.
To get user profile information after the user has logged in you should be using the people api the people.get method when passed the access token will return to you the profile information of the currently authenticated user.
beyond that you should IMO not be passing tokens to your backend if you want the token in the backend you should log the user in using your backend language.
Okay so the idToken that I had was correct but flutter terminal was not able to print its full length and because of which I was copying only half the token and was getting a signature error when checking manually.
I have a Python script that is running periodically on an AWS EC2 Ubuntu machine.
This script reads data from some files and sometimes changes data in them.
I want to download these files from OneDrive, do my own thing with them, and upload them back to OneDrive.
I want this to be done automatically, without the need for a user to approve any login or credentials. I'm ok with doing it once (i.e. approving the login on the first run) but the rest has to run automatically, without asking ever again for approvals (unless the permissions change, of course).
What is the best way to do this?
I've been reading the documentation on Microsoft Graph API but I'm struggling with the authentication part. I've created an application in Azure AAD, gave the sample permissions (to test) and created a secret credential.
I managed to do it. I'm not sure if it's the best way but it is working now. It's running automatically every hour and I don't need to touch it.
I followed the information on https://learn.microsoft.com/en-gb/azure/active-directory/develop/v2-oauth2-auth-code-flow
This is what I did.
Azure Portal
Create an application. Azure Active Directory -> App Registrations -> Applications from personal account
In Supported account types, choose the one that has personal Microsoft accounts.
In Redirect URI, choose Public client/native. We'll add the specific URI later.
In the application details, in the section Overview, take note of the Application (client) ID. We'll need this later.
In the section Authentication, click Add a Platform and choose Desktop + devices. You can use your own, I chose one of the suggested: https://login.microsoftonline.com/common/oauth2/nativeclient
In the section API permissions, you have to add all the permissions that your app will use. I added User.Read, Files.ReadWrite and offline_access. The offline_access is to be able to get the refresh token, which will be crucial to keep the app running without asking the user to login.
I did not create any Certificate or Secret.
Web
Looks like to get a token for the first time we have to use a browser or emulate something like that.
There must be a programmatic way to do this, but I had no idea how to do it. I also thought about using Selenium for this, but since it's only one time and my app will request tokens every hour (keeping the tokens fresh), I dropped that idea.
If we add new permissions, the tokens that we have will become invalid and we have to do this manual part again.
Open a browser and go to the URL below. Use the Scopes and the Redirect URI that you set up in Azure Portal.
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=your_app_client_id&response_type=code&redirect_uri=https%3A%2F%2Flogin.microsoftonline.com%2Fcommon%2Foauth2%2Fnativeclient&response_mode=query&scope=User.Read%20offline_access%20Files.ReadWrite
That URL will redirect you to the Redirect URI that you set up and with a code=something in the URL. Copy that something.
Do a POST request with type FORM URL Encoded. I used https://reqbin.com/ for this.
Endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token
Form URL: grant_type=authorization_code&client_id=your_app_client_id&code=use_the_code_returned_on_previous_step
This will return an Access Token and a Refresh Token. Store the Refresh Token somewhere. I'm saving it in a file.
Python
# Build the POST parameters
params = {
'grant_type': 'refresh_token',
'client_id': your_app_client_id,
'refresh_token': refresh_token_that_you_got_in_the_previous_step
}
response = requests.post('https://login.microsoftonline.com/common/oauth2/v2.0/token', data=params)
access_token = response.json()['access_token']
new_refresh_token = response.json()['refresh_token']
# ^ Save somewhere the new refresh token.
# I just overwrite the file with the new one.
# This new one will be used next time.
header = {'Authorization': 'Bearer ' + access_token}
# Download the file
response = requests.get('https://graph.microsoft.com/v1.0/me/drive/root:' +
PATH_TO_FILE + '/' + FILE_NAME + ':/content', headers=header)
# Save the file in the disk
with open(file_name, 'wb') as file:
file.write(response.content)
So basically, I have the Refresh Token always updated.
I call the Token endpoint using that Refresh Token, and the API gives me an Access Token to use during the current session and a new Refresh Token.
I use this new Refresh Token the next time I run the program, and so on.
I've just published a repo which does this. Contributions and pull requests welcome:
https://github.com/stevemurch/onedrive-download
I have added Firebase to allow clients to authenticate directly from the web app client (browser). I am using the firebase-web JS package and it works great. I can see in my browser that I receive a user object with information about the user, including an idToken.
I need to then authenticate this user on my server backend, which is python django. In the Firebase docs I found a how-to for exactly what I am trying to do, which is to verify the id token.
Since they don't have the supported Firebase sdk for python, I need to use a third party solution. I have come to the python-jose package after finding it listed on the jwt.io site. The example looks simple enough:
jwt.decode(token, 'secret', algorithms=['RS256'])
This is my first time using JWT. I don't know what to use for the 'secret'. I tried pasting my id token as token, and the web API key from the Firebase console for secret, but got this error:
jose.exceptions.JWKError: RSA key format is not supported
I also tried the JWT debugger, which seems to be reading most of my id token correctly, but the signature verification is looking for a public and/or a private keys, which like the 'secret' are escaping me.
I am really at a loss for how to find this secret, and how to verify the JWT id token in general. The information on the Firebase docs (third-party section) is:
Finally, ensure that the ID token was signed by the private key
corresponding to the token's kid claim. Grab the public key from
https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com
and use a JWT library to verify the signature. Use the value of
max-age in the Cache-Control header of the response from that endpoint
to know when to refresh the public keys.
I have tried pasting the whole json blob from that googleapis url into the JWT debugger, but still getting an "invalid signature" alert. I don't understand how to use that public key.
Should python-jose work for this approach? If so, what should I use for the secret? If not, can someone point me in the right direction?
Thanks.
I finally found the answer I was looking for in this post: Migrating Python backend from Gitkit to to Firebase-Auth with python-jose for token verification
Since the time of the post there have been updates made to the python-jose package, which gives better support for firebase id tokens. Here is some working code ( jose version 1.3.1 ) on how to use python to decode the firebase id token:
import urllib, json
from jose import jwt
idtoken = "<id token passed to server from firebase auth>"
target_audience = "<firebase app id>"
certificate_url = 'https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com'
response = urllib.urlopen(certificate_url)
certs = response.read()
certs = json.loads(certs)
#will throw error if not valid
user = jwt.decode(idtoken, certs, algorithms='RS256', audience=target_audience)
print user
I am having a problem while using the access token retrieved using the refresh token. I am using python api client for Google doc. I have offline access mode and stored my refresh token in database in the userconsent part. Now using the refresh token, I am also able to retrieve the access token using requests.post. Now when I feed the access token thus retrieved to authenticate in the Flowexchange
self.flow = OAuth2WebServerFlow(CLIENT_ID, CLIENT_SECRET .... )
self.flow.step2_exchange(accesstoken)
I am getting FlowExchangeError(u'invalid_grant',). Anything I m missing ?
Thanks