I have a Python script that is running periodically on an AWS EC2 Ubuntu machine.
This script reads data from some files and sometimes changes data in them.
I want to download these files from OneDrive, do my own thing with them, and upload them back to OneDrive.
I want this to be done automatically, without the need for a user to approve any login or credentials. I'm ok with doing it once (i.e. approving the login on the first run) but the rest has to run automatically, without asking ever again for approvals (unless the permissions change, of course).
What is the best way to do this?
I've been reading the documentation on Microsoft Graph API but I'm struggling with the authentication part. I've created an application in Azure AAD, gave the sample permissions (to test) and created a secret credential.
I managed to do it. I'm not sure if it's the best way but it is working now. It's running automatically every hour and I don't need to touch it.
I followed the information on https://learn.microsoft.com/en-gb/azure/active-directory/develop/v2-oauth2-auth-code-flow
This is what I did.
Azure Portal
Create an application. Azure Active Directory -> App Registrations -> Applications from personal account
In Supported account types, choose the one that has personal Microsoft accounts.
In Redirect URI, choose Public client/native. We'll add the specific URI later.
In the application details, in the section Overview, take note of the Application (client) ID. We'll need this later.
In the section Authentication, click Add a Platform and choose Desktop + devices. You can use your own, I chose one of the suggested: https://login.microsoftonline.com/common/oauth2/nativeclient
In the section API permissions, you have to add all the permissions that your app will use. I added User.Read, Files.ReadWrite and offline_access. The offline_access is to be able to get the refresh token, which will be crucial to keep the app running without asking the user to login.
I did not create any Certificate or Secret.
Web
Looks like to get a token for the first time we have to use a browser or emulate something like that.
There must be a programmatic way to do this, but I had no idea how to do it. I also thought about using Selenium for this, but since it's only one time and my app will request tokens every hour (keeping the tokens fresh), I dropped that idea.
If we add new permissions, the tokens that we have will become invalid and we have to do this manual part again.
Open a browser and go to the URL below. Use the Scopes and the Redirect URI that you set up in Azure Portal.
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=your_app_client_id&response_type=code&redirect_uri=https%3A%2F%2Flogin.microsoftonline.com%2Fcommon%2Foauth2%2Fnativeclient&response_mode=query&scope=User.Read%20offline_access%20Files.ReadWrite
That URL will redirect you to the Redirect URI that you set up and with a code=something in the URL. Copy that something.
Do a POST request with type FORM URL Encoded. I used https://reqbin.com/ for this.
Endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token
Form URL: grant_type=authorization_code&client_id=your_app_client_id&code=use_the_code_returned_on_previous_step
This will return an Access Token and a Refresh Token. Store the Refresh Token somewhere. I'm saving it in a file.
Python
# Build the POST parameters
params = {
'grant_type': 'refresh_token',
'client_id': your_app_client_id,
'refresh_token': refresh_token_that_you_got_in_the_previous_step
}
response = requests.post('https://login.microsoftonline.com/common/oauth2/v2.0/token', data=params)
access_token = response.json()['access_token']
new_refresh_token = response.json()['refresh_token']
# ^ Save somewhere the new refresh token.
# I just overwrite the file with the new one.
# This new one will be used next time.
header = {'Authorization': 'Bearer ' + access_token}
# Download the file
response = requests.get('https://graph.microsoft.com/v1.0/me/drive/root:' +
PATH_TO_FILE + '/' + FILE_NAME + ':/content', headers=header)
# Save the file in the disk
with open(file_name, 'wb') as file:
file.write(response.content)
So basically, I have the Refresh Token always updated.
I call the Token endpoint using that Refresh Token, and the API gives me an Access Token to use during the current session and a new Refresh Token.
I use this new Refresh Token the next time I run the program, and so on.
I've just published a repo which does this. Contributions and pull requests welcome:
https://github.com/stevemurch/onedrive-download
I am trying to fetch captions from YouTube video using YouTube Data API (v3)
https://developers.google.com/youtube/v3/guides/implementation/captions
So, first I tried to retrieve a captions list using this url:
https://www.googleapis.com/youtube/v3/captions?part=snippet&videoId=KK9bwTlAvgo&key={My API KEY}
I could retrieve the caption id that I'd like to download (jEDP-pmNCIqoB8QGlXWQf4Rh3faalD_l) from the above link.
Then, I followed this instruction to download the caption:
https://developers.google.com/youtube/v3/docs/captions/download
However, even though I input the caption id and my api key correctly, it shows "Login Required" error.
I suppose I need OAuth authentication, but what I am trying to do is not related to my users's account, but simply downloading public caption data automatically.
My question is: Is there any way to process OAuth authentication just once to get an access token of my own YouTube account and then reuse it whenever I need it in my application?
I can't speak to the permissions needed for the captions API in particular, but in general, yes, you can OAuth to your app once using your own account and use the access and refresh tokens to make subsequent OAuth'd requests to the API. You can find the details of generating tokens here:
https://developers.google.com/youtube/v3/guides/auth/server-side-web-apps#Obtaining_Access_Tokens
To perform the steps manually (fortunately, you only need to do this once):
If access has already been granted for an app, it needs to be removed so that new auth credentials can be established. Go to https://security.google.com/settings/security/permissions (while logged into your account) and remove access to the app. If the client ID or secret change (or you need to create one), find them at https://console.developers.google.com under API Manager.
To grant access and receive a temporary code, enter this URL in a browser:
https://accounts.google.com/o/oauth2/auth?
client_id=<client_id>&
redirect_uri=http://www.google.com&
scope=https://www.googleapis.com/auth/youtube.force-ssl&
response_type=code&
access_type=offline&
approval_prompt=force
Follow the prompt to grant access to the app.
This will redirect to google.com with a code parameter (e.g.,
https://www.google.com/?code=4/ux5gNj-_mIu4DOD_gNZdjX9EtOFf&gws_rd=ssl#). Save the code.
Send a POST request (e.g., via Postman Chrome plugin) to https://accounts.google.com/o/oauth2/token with the following in the request body:
code=<code>&
client_id=<client_id>&
client_secret=<client_secret>&
redirect_uri=http://www.google.com&
grant_type=authorization_code
The response will contain both an access token and refresh token. Save both, but particularly the refresh token (because the access token will expire in 1 hour).
You can then use the access token to send an OAuth'd request manually, following one of the options here, essentially:
curl -H "Authorization: Bearer ACCESS_TOKEN" https://www.googleapis.com/youtube/v3/captions/<id>
or
curl https://www.googleapis.com/youtube/v3/captions/<id>?access_token=ACCESS_TOKEN
(When I tried the second option for captions, however, I got the message: "The OAuth token was received in the query string, which this API forbids for response formats other than JSON or XML. If possible, try sending the OAuth token in the Authorization header instead.")
You can also use the refresh token in your code to create the credential needed when building your YouTube object. In Java, this looks like the following:
String clientId = <your client ID>
String clientSecret = <your client secret>
String refreshToken = <refresh token>
HttpTransport transport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(transport)
.setJsonFactory(jsonFactory)
.setClientSecrets(clientId, clientSecret)
.build()
.setRefreshToken(refreshToken);
try {
credential.refreshToken();
} catch (IOException e) {
e.printStackTrace();
}
youtube = new YouTube.Builder(transport, jsonFactory, credential).build();
I imagine you can do something similar in Python with the API Client Libraries, although I haven't tried Python.
I am trying to access my own docs & spreadsheets via onedrive's api. I have:
import requests
client_id = 'my_id'
client_secret = 'my_secret'
scopes = 'wl.offline_access%20wl.signin%20wl.basic'
response_type = 'token' # also have tried "code"
redirect_uri = 'https://login.live.com/oauth20_desktop.srf'
base_url = 'https://apis.live.net/v5.0/'
r = requests.get('https://login.live.com/oauth20_authorize.srf?client_id=%s&scope=%s&response_type=%s&redirect_uri=%s' % (client_id, scopes, response_type, redirect_uri))
print r.text
(For my client I've also tried both "Mobile or desktop client app:" set to "Yes" and "No")
This will return the html for the user to manually click on. Since the user is me and it's my account how do I access the API without user interaction?
EDIT #1:
For those confused on what I'm looking for it would be the equivalent of Google's Service Account (OAuth2): https://console.developers.google.com/project
You cannot "bypass" the user interaction.
However you are very close to getting it to work. If you want to gain an access token in python you have to do it through the browser. You can use the web browser library to open the default web browser. It will look something like this (your app must be a desktop app):
import webbrowser
webbrowser.open("https://login.live.com/oauth20_authorize.srf?client_id=foo&scope=bar&response_type=code&redirect_uri=https://login.live.com/oauth20_desktop.srf")
This will bring you to the auth page, sign in and agree to the terms (it will differ depending on scope). It will direct you to a page where the url looks like:
https://login.live.com/oauth20_desktop.srf?code=<THISISTHECODEYOUWANT>&lc=foo
Copy this code from the browser and have your python script take it as input.
You can then make a request as described here using the code you received from the browser.
You will receive a response described here
I want use django-s3direct and I want upload many image in admin panel.
1) All time when I try upload image/file get error "Oops, file upload failed, please try again" ?
When I refresh page. Name file is in input. But my input "Save" are disabled :/
edit
I remove from settings:
AWS_SECRET_ACCESS_KEY = ''
AWS_ACCESS_KEY_ID = ''
AWS_STORAGE_BUCKET_NAME = ''
and now I don't get error but file no upload :/ All time black progress bar..
2) How upload multiple image? No inline.. Please help me and give some advice? Im newbie..
I have Django 1.5.5. Now i use inline and I don't know what's next.
You will need to edit some of the permissions properties of the target S3 bucket so that the final request has sufficient privileges to write to the bucket. Sign in to the AWS console and select the S3 section. Select the appropriate bucket and click the ‘Properties’ tab. Select the Permissions section and three options are provided (Add more permissions, Edit bucket policy and Edit CORS configuration).
CORS (Cross-Origin Resource Sharing) will allow your application to access content in the S3 bucket. Each rule should specify a set of domains from which access to the bucket is granted and also the methods and headers permitted from those domains.
For this to work in your application, click ‘Add CORS Configuration’ and enter the following XML:
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>yourdomain.com</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
Click ‘Save’ in the CORS window and then ‘Save’ again in the bucket’s ‘Properties’ tab.
This tells S3 to allow any domain access to the bucket and that requests can contain any headers. For security, you can change the ‘AllowedOrigin’ to only accept requests from your domain.
If you wish to use S3 credentials specifically for this application, then more keys can be generated in the AWS account pages. This provides further security, since you can designate a very specific set of requests that this set of keys are able to perform. If this is preferable to you, then you will need to also set up an IAM user in the Edit bucket policy option in your S3 bucket. There are various guides on AWS’s web pages detailing how this can be accomplished.
Setting up the client-side code
This setup does not require any additional, non-standard Python libraries, but some scripts are necessary to complete the implementation on the client-side.
This article covers the use of the s3upload.js script. Obtain this script from the project’s repo (using Git or otherwise) and store it somewhere appropriate in your application’s static directory. This script currently depends on both the JQuery and Lo-Dash libraries. Inclusion of these in your application will be covered later on in this guide.
The HTML and JavaScript can now be created to handle the file selection, obtain the request and signature from your Python application, and then finally make the upload request.
Firstly, create a file called account.html in your application’s templates directory and populate the head and other necessary HTML tags appropriately for your application. In the body of this HTML file, include a file input and an element that will contain status updates on the upload progress.
<input type="file" id="file" onchange="s3_upload();"/>
<p id="status">Please select a file</p>
<div id="preview"><img src="/static/default.png" /></div>
<form method="POST" action="/submit_form/">
<input type="hidden" id="" name="" value="/static/default.png" />
<input type="text" name="example" placeholder="" /><br />
<input type="text" name="example2" placeholder="" /><br /><br />
<input type="submit" value="" />
</form>
The preview element initially holds a default image. Both of these are updated by the JavaScript, discussed below, when the user selects a new image.
Thus when the user finally clicks the submit button, the URL of the image is submitted, along with the other details of the user, to your desired endpoint for server-side handling. The JavaScript method, s3_upload(), is called when a file is selected by the user. The creation and population of this method is covered below.
Next, include the three dependency scripts in your HTML file,account.html. You may need to adjust the src attribute for the files3upload.js if you put this file in a directory other than /static:
<script type="text/javascript" src="http://code.jquery.com/jquery-1.9.1.js"></script>
<script type="text/javascript" src="https://raw.github.com/bestiejs/lodash/v1.1.1/dist/lodash.min.js"></script>
<script type="text/javascript" src="/static/s3upload.js"></script>
The ordering of the scripts is important as the dependencies need to be satisfied in this sequence. If you desire to host your own versions of JQuery and Lo-Dash, then adjust thesrc attribute accordingly.
Finally, in a block, declare a JavaScript function,s3_upload(), in the same file again to process the file upload. This block will need to exist below the inclusion of the three dependencies:
function s3_upload(){
var s3upload = new S3Upload({
file_dom_selector: 'file',
s3_sign_put_url: '/sign_s3_upload/',
onProgress: function(percent, message) {
$('#status').html('Upload progress: ' + percent + '%' + message);
},
onFinishS3Put: function(url) {
$('#status').html('Upload completed. Uploaded to: '+ url);
$("#image_url").val(url);
$("#preview").html('<img src="'+url+'" style="width:300px;" />');
},
onError: function(status) {
$('#status').html('Upload error: ' + status);
}
});
}
This function creates a new instance of S3Upload, to which is passed the file input element, the URL from which to retrieve the signed request and three functions.
Initially, the function makes a request to the URL denoted by thes3_sign_put_url argument, passing the file name and mime type as GET parameters. The server-side code (covered in the next section) interprets the request and responds with a preview of the URL of the file to be uploaded to S3 and the signed request, which this function then uses to asynchronously upload the file to your bucket.
The function will post upload updates to the onProgress() function and , if the upload is successful, onFinishS3Put() is called and the URL returned by the Python application view is received as an argument. If, for any reason, the upload should fail, onError() will be called and thestatus parameter will describe the error.
If you find that the page isn’t working as you intend after implementing the system, then consider using console.log()to record any errors that occur inside the onError() callback and use your browser’s error console to help diagnose the problem.
If successful, the preview div will now be updated with the user’s chosen image, and the hidden input field will contain the URL for the image. Now, once the user has completed the rest of the form and clicked submit, all pieces of information can be posted to the same endpoint.
It is good practice to inform the user of any prolonged activity in any form of application (web- or device-based) and to display updates on changes. Thus the status methods could be used, for example, to show a loading GIF to indicate that an upload is in progress, which can then be hidden when the upload has finished. Without this sort of information, users may suspect that the page has crashed, and could try to refresh the page or otherwise disrupt the upload process.
Setting up the server-side Python code
To generate a temporary signature with which the upload request can be signed. This temporary signature uses the account details (the AWS access key and secret access key) as a basis for the signature, but users will not have direct access to this information. After the signature has expired, then upload requests with the same signature will not be successful.
As mentioned previously, this article covers the production of an application for the Flask framework, although the steps for other Python frameworks will be similar. Readers using Python 3 should consider therelevant information on Flask’s website before continuing.
Start by creating your main application file, application.py, and set up your skeleton application appropriately:
from flask import Flask, render_template, request
from hashlib import sha1
import time, os, json, base64, hmac, urllib
app = Flask(__name__)
if __name__ == '__main__':
port = int(os.environ.get('PORT', 5000))
app.run(host='0.0.0.0', port=port)
The currently-unused import statements will be necessary later on.
Readers using Python 3 should import urllib.parse in place of urllib.
Next, in the same file, you will need to create the views responsible for returning the correct information back to the user’s browser when requests are made to various URLs. First define view for requests to/account to return the page account.html, which contains the form for the user to complete:
#app.route("/account/")
def account():
return render_template('account.html')
Please note that the views for the application will need to be placed between the app = Flask(__name__) and if __name__ == '__main__': lines in application.py.
Now create the view, in the same Python file, that is responsible for generating and returning the signature with which the client-side JavaScript can upload the image. This is the first request made by the client before attempting an upload to S3. This view responds with requests to /sign_s3/:
#app.route('/sign_s3/')
def sign_s3():
AWS_ACCESS_KEY = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
S3_BUCKET = os.environ.get('S3_BUCKET')
object_name = request.args.get('s3_object_name')
mime_type = request.args.get('s3_object_type')
expires = int(time.time()+10)
amz_headers = "x-amz-acl:public-read"
put_request = "PUT\n\n%s\n%d\n%s\n/%s/%s" % (mime_type, expires, amz_headers, S3_BUCKET, object_name)
signature = base64.encodestring(hmac.new(AWS_SECRET_KEY, put_request, sha1).digest())
signature = urllib.quote_plus(signature.strip())
url = 'https://%s.s3.amazonaws.com/%s' % (S3_BUCKET, object_name)
return json.dumps({
'signed_request': '%s?AWSAccessKeyId=%s&Expires=%d&Signature=%s' % (url, AWS_ACCESS_KEY, expires, signature),
'url': url
})
Readers using Python 3 should useurllib.parse.quote_plus() to quote the signature.
This code performs the following steps:
• The request is received to /sign_s3/ and the AWS keys and S3 bucket name are loaded from the environment.
• The name and mime type of the object to be uploaded are extracted from the GET parameters of the request (this stage may differ in other frameworks).
• The expiry time of the signature is set and forms the basis of the temporary nature of the signature. As shown, this is best used as a function relative to the current UNIX time. In this example, the signature will expire 10 seconds after Python has executed that line of code.
• The headers line tells S3 what access permissions to grant. In this case, the object will be publicly available for download.
• Now the PUT request can be constructed from the object information, headers and expiry time.
• The signature is generated as an SHA hash of the compiled AWS secret key and the actual PUT request.
• In addition, surrounding whitespace is stripped from the signature and special characters are escaped (using quote_plus) for safer transmission through HTTP.
• The prospective URL of the object to be uploaded is produced as a combination of the S3 bucket name and the object name.
• Finally, the signed request can be returned, along with the prospective URL, to the browser in JSON format.
You may wish to assign another, customised name to the object instead of using the one that the file is already named with, which is useful for preventing accidental overwrites in the S3 bucket. This name could be related to the ID of the user’s account, for example. If not, you should provide some method for properly quoting the name in case there are spaces or other awkward characters present. In addition, this is the stage at which you could provide checks on the uploaded file in order to restrict access to certain file types. For example, a simple check could be implemented to allow only .png files to proceed beyond this point.
It is sometimes possible for S3 to respond with 403 (forbidden) errors for requests which are signed by temporary signatures containing special characters. Therefore, it is important to appropriately quote the signature as demonstrated above.
Finally, in application.py, create the view responsible for receiving the account information after the user has uploaded an image, filled in the form, and clicked submit. Since this will be a POST request, this will also need to be defined as an ‘allowed access method’. This method will respond to requests to the URL /submit_form/:
#app.route("/submit_form/", methods=["POST"])
def submit_form():
example = request.form[""]
example2 = request.form[""]
image_url = request.form["image_url"]
update_account(example, example2, image_url)
return redirect(url_for('profile'))
In this example, an update_account() function has been called, but creation of this method is not covered in this article. In your application, you should provide some functionality, at this stage, to allow the app to store these account details in some form of database and correctly associate the information with the rest of the user’s account details.
In addition, the URL for the profile page has not been defined in this article (or companion code). Ideally, for example, after updating the account, the user would be redirected back to their own profile so that they can see the updated information.
For more information http://www.tivix.com/blog/easy-user-uploads-with-direct-s3-uploading/
I have an app engine web app that would like to automatically write a post to the wall of a facebook application I control (i.e. every time a particular event occurs on the website I would like to update the wall of my facebook application).
This code will be called from a deferred task on the server.
I have been unable to find anything addressing this. Your help would be appreciated.
First thing I did was get my access token with the following code:
https://graph.facebook.com/oauth/access_token?client_id=FACEBOOK_APP_ID&client_secret=FACEBOOK_APP_SECRET&grant_type=client_credentials&scope=manage_pages,offline_access
Using the returned access token this is what I'm running on the server:
form_fields = {
"access_token": FACEBOOK_ACCESS_TOKEN,
"message": tgText
};
form_data = urllib.urlencode(form_fields)
result = urlfetch.fetch(url="https://graph.facebook.com/MYAPP_FACEBOOK_ID/feed",
payload=form_data,
method=urlfetch.POST,
validate_certificate=False,
headers={'Content-Type': 'application/x-www-form-urlencoded'})
But calling this results in:
{"error":{"type":"OAuthException","message":"(#200) The user hasn't authorized the application to perform this action"}}
As an administrator you can grant access to third party apps (e.g. your python app) to post onto your App's Profile Page (http://www.facebook.com/apps/application.php?id=YOUR_APP_ID) using OAuth:
http://developers.facebook.com/docs/authentication/ (Section Page Login)
Once you received an access token you should be able to post to App Profile Page as described here:
http://developers.facebook.com/docs/reference/api/post/ (Section Publishing)
I have a similar app. Facebook can change the code that you're meant to submit as part of the process for gaining an access token. I wrote a simple web page that creates a form with hidden input fields that contain the data required to get Facebook to authorise an app with a user (see http://developers.facebook.com/docs/reference/dialogs/oauth/).
When the user clicks the submit button, the browser does an HTTP GET to Facebook, with an appropriate query string, where the Facebook user is prompted to login (if needed) and authorise the app. If authorised Facebook calls back your redirect_url with the code which you can store in the DataStore to retrieve when needed as part of the "give me an access token" flow.