I am using Django with requests_oauthlib to access the Xero API using the xero-python SDK.
Xero Starter Demo
requests-oauthlib
I am trying to use request.session to store the oauth token information. This works well on the initial call and callback, but in subsequent views when I want to load the correct token from the request.session I am hitting a problem.
The Xero ApiClient (xero_python.api_client) wants me to configure two functions it can call to get and save the token in the future (oauth2_token_getter, oauth2_token_saver) (decorators are used in the demo). Those functions are called by the Xero API and take no parameters. I can create the functions no problem, and they get called correctly, but once inside the functions I have no request object so I can't access request.session to get or save the token to the session.
The Xero demonstration projects use Flask which imports a root level session object (from flask import session) which can be accessed in any function, so this works fine for Flask. What is the correct way to solve this problem in Django?
I have tried creating a SessionStore object but it was empty despite me knowing the data is in the request.session
I have also tried importing requests.session but this doesn't seem to be what I am looking for as I can't get it to read/write any data.
Some Code:
Saving the token in the callback works as I have a request.session:
def xero_callback(request):
...
request.session["oauth_token"] = token
...
The Xero ApiClient is configured with the methods for getting and saving:
api_client = ApiClient(
Configuration(
debug=True,
oauth2_token=OAuth2Token(client_id=config.XERO_CLIENT_ID, client_secret=config.XERO_CLIENT_SECRET),
),
pool_threads=1,
oauth2_token_getter=oauth2_token_getter,
oauth2_token_saver=oauth2_token_saver)
This doesn't work as I have no request object and can't pass one in:
def oauth2_token_getter():
return request.session.get("token") #! FAIL, I don't have access to request here
The error is thrown after calling identity_api.get_connections() as it tries to get the token using oauth2_token_getter and fails
def xero_tenants(request):
identity_api = IdentityApi(api_client)
accounting_api = AccountingApi(api_client)
available_tenants = []
for connection in identity_api.get_connections():
...
Any help appreciated, Thanks
Are you able to store the token in the django cache? (django.core.cache import cache) Then instead of specifying oauth2_token_getter=oauth2_token_getter in the ApiClient constructor you can do:
#api_client.oauth2_token_getter
def obtain_xero_oauth2_token():
return cache.get('token')
and something similar for the setter/saver.
this is a two-part question: I have seen individual pieces discussed, but can't seem to get the recommended suggestions to work together. I want to create a web service to store images and their metadata passed from a caller and run a test call from Postman to make sure it is working. So to pass an image (Drew16.jpg) to the web service via Postman, it appears I need something like this:
For the web service, I have some python/flask code to read the request (one of many variations I have tried):
from flask import Flask, jsonify, request, render_template
from flask_restful import Resource, Api, reqparse
...
def post(self, name):
request_data = request.get_json()
userId = request_data['UserId']
type = request_data['ImageType']
image = request.files['Image']
Had no problem with the data portion and straight JSON but adding the image has been a bugger. Where am I going wrong on my Postman config? What is the actual set of Python commands for reading the metadata and the file from the post? TIA
Pardon the almost blog post. I am posting this because while you can find partial answers in various places, I haven't run across a complete post anywhere, which would have saved me a ton of time. The problem is you need both sides to the story in order to verify either.
So I want to send a request using Postman to a Python/Flask web service. It has to have an image along with some metadata.
Here are the settings for Postman (URL, Headers):
And Body:
Now on to the web service. Here is a bare bones service which will take the request, print the metadata and save the file:
from flask import Flask, request
app = Flask(__name__)
# POST - just get the image and metadata
#app.route('/RequestImageWithMetadata', methods=['POST'])
def post():
request_data = request.form['some_text']
print(request_data)
imagefile = request.files.get('imagefile', '')
imagefile.save('D:/temp/test_image.jpg')
return "OK", 200
app.run(port=5000)
Enjoy!
Make sure `request.files['Image'] contains the image you are sending and follow http://flask.pocoo.org/docs/1.0/patterns/fileuploads/ to save the file to your file system. Something like
file = request.files['Image']
file.save('./test_image.jpg')
might do what you want, while you will have to work out the details of how the file should be named and where it should be placed.
I need to read some values from the wsgi request before my flask app is loaded. If I read the url from the wsgi request I can access the file without any issues once the flask app is loaded (after the middleware runs).
But if I attempt to access the params it seems to remove the post data once the flask app is loaded. I even went to the extreme of wrapping the wsgi request with a special Webob Request to prevent this "read once" problem.
Does anyone know how to access values from the wsgi request in middleware without doing any sort of side effect harm to the request so you can get post data / file data in a flask app?
from webob import Request
class SomeMiddleware(object):
def __init__(self, environ):
self.request = Request(environ)
self.orig_environ = environ
def apply_middleware(self):
print self.request.url #will not do any harm
print self.request.params #will cause me to lose data
Here is my flask view
#app.route('/')
def hello_world():
from flask import request
the_file = request.files['file']
print "and the file is", the_file
From what I can tell, this is a limitation of the way that WSGI works. The stream needs only be consumable once (PEP 333 and 3333 only require that the stream support read* calls, tell does not need to be supported). Once the stream is exhausted it cannot be re-streamed to other WSGI applications further "inward". Take a look at these two sections of Werkzeug's documentation for more information:
http://werkzeug.pocoo.org/docs/request_data/
http://werkzeug.pocoo.org/docs/http/#module-werkzeug.formparser
The way to avoid this issue is to wrap the input stream (wsgi.input) in an object that implements the read and readline methods. Then, only when the final application in the chain actually attempts to exhaust the stream will your methods be run. See Flask's documentation on generating a request checksum for an example of this pattern.
That being said, are you sure a middleware is the best solution to your problems? If you need to perform some action (dispatch, logging, authentication) based on the content of the body of the request you may be better off making it a part of your application, rather than a stand-alone application of its own.
I use python Appengine. I'm trying to create a link on a webpage, which a user can click to download a csv file. How can I do this?
I've looked at csv module, but it seems to want to open a file on the server, but appengine doesn't allow that.
I've looked at remote_api, but it seems that its only for uploading or downloading using app config, and from account owner's terminal.
Any help thanks.
Pass a StringIO object as the first parameter to csv.writer; then set the content-type and content-disposition on the response appropriately (probably "text/csv" and "attachment", respectively) and send the StringIO as the content.
I used this code:
self.response.headers['Content-Type'] = 'application/csv'
writer = csv.writer(self.response.out)
writer.writerow(['foo','foo,bar', 'bar'])
Put it in your handler's get method. When user requests it, user's browser will download the list content automatically.
Got from: generating a CSV file online on Google App Engine
I want users on the site to be able to download files whose paths are obscured so they cannot be directly downloaded.
For instance, I'd like the URL to be something like this: http://example.com/download/?f=somefile.txt
And on the server, I know that all downloadable files reside in the folder /home/user/files/.
Is there a way to make Django serve that file for download as opposed to trying to find a URL and View to display it?
For the "best of both worlds" you could combine S.Lott's solution with the xsendfile module: django generates the path to the file (or the file itself), but the actual file serving is handled by Apache/Lighttpd. Once you've set up mod_xsendfile, integrating with your view takes a few lines of code:
from django.utils.encoding import smart_str
response = HttpResponse(mimetype='application/force-download') # mimetype is replaced by content_type for django 1.7
response['Content-Disposition'] = 'attachment; filename=%s' % smart_str(file_name)
response['X-Sendfile'] = smart_str(path_to_file)
# It's usually a good idea to set the 'Content-Length' header too.
# You can also set any other required headers: Cache-Control, etc.
return response
Of course, this will only work if you have control over your server, or your hosting company has mod_xsendfile already set up.
EDIT:
mimetype is replaced by content_type for django 1.7
response = HttpResponse(content_type='application/force-download')
EDIT:
For nginx check this, it uses X-Accel-Redirect instead of apache X-Sendfile header.
A "download" is simply an HTTP header change.
See http://docs.djangoproject.com/en/dev/ref/request-response/#telling-the-browser-to-treat-the-response-as-a-file-attachment for how to respond with a download.
You only need one URL definition for "/download".
The request's GET or POST dictionary will have the "f=somefile.txt" information.
Your view function will simply merge the base path with the "f" value, open the file, create and return a response object. It should be less than 12 lines of code.
For a very simple but not efficient or scalable solution, you can just use the built in django serve view. This is excellent for quick prototypes or one-off work, but as has been mentioned throughout this question, you should use something like apache or nginx in production.
from django.views.static import serve
filepath = '/some/path/to/local/file.txt'
return serve(request, os.path.basename(filepath), os.path.dirname(filepath))
S.Lott has the "good"/simple solution, and elo80ka has the "best"/efficient solution. Here is a "better"/middle solution - no server setup, but more efficient for large files than the naive fix:
http://djangosnippets.org/snippets/365/
Basically, Django still handles serving the file but does not load the whole thing into memory at once. This allows your server to (slowly) serve a big file without ramping up the memory usage.
Again, S.Lott's X-SendFile is still better for larger files. But if you can't or don't want to bother with that, then this middle solution will gain you better efficiency without the hassle.
Just mentioning the FileResponse object available in Django 1.10
Edit: Just ran into my own answer while searching for an easy way to stream files via Django, so here is a more complete example (to future me). It assumes that the FileField name is imported_file
views.py
from django.views.generic.detail import DetailView
from django.http import FileResponse
class BaseFileDownloadView(DetailView):
def get(self, request, *args, **kwargs):
filename=self.kwargs.get('filename', None)
if filename is None:
raise ValueError("Found empty filename")
some_file = self.model.objects.get(imported_file=filename)
response = FileResponse(some_file.imported_file, content_type="text/csv")
# https://docs.djangoproject.com/en/1.11/howto/outputting-csv/#streaming-large-csv-files
response['Content-Disposition'] = 'attachment; filename="%s"'%filename
return response
class SomeFileDownloadView(BaseFileDownloadView):
model = SomeModel
urls.py
...
url(r'^somefile/(?P<filename>[-\w_\\-\\.]+)$', views.SomeFileDownloadView.as_view(), name='somefile-download'),
...
Tried #Rocketmonkeys solution but downloaded files were being stored as *.bin and given random names. That's not fine of course. Adding another line from #elo80ka solved the problem.
Here is the code I'm using now:
from wsgiref.util import FileWrapper
from django.http import HttpResponse
filename = "/home/stackoverflow-addict/private-folder(not-porn)/image.jpg"
wrapper = FileWrapper(file(filename))
response = HttpResponse(wrapper, content_type='text/plain')
response['Content-Disposition'] = 'attachment; filename=%s' % os.path.basename(filename)
response['Content-Length'] = os.path.getsize(filename)
return response
You can now store files in a private directory (not inside /media nor /public_html) and expose them via django to certain users or under certain circumstances.
Hope it helps.
Thanks to #elo80ka, #S.Lott and #Rocketmonkeys for the answers, got the perfect solution combining all of them =)
It was mentioned above that the mod_xsendfile method does not allow for non-ASCII characters in filenames.
For this reason, I have a patch available for mod_xsendfile that will allow any file to be sent, as long as the name is url encoded, and the additional header:
X-SendFile-Encoding: url
Is sent as well.
http://ben.timby.com/?p=149
Try: https://pypi.python.org/pypi/django-sendfile/
"Abstraction to offload file uploads to web-server (e.g. Apache with mod_xsendfile) once Django has checked permissions etc."
You should use sendfile apis given by popular servers like apache or nginx in production. For many years I was using the sendfile api of these servers for protecting files. Then created a simple middleware based django app for this purpose suitable for both development & production purposes. You can access the source code here.
UPDATE: in new version python provider uses django FileResponse if available and also adds support for many server implementations from lighthttp, caddy to hiawatha
Usage
pip install django-fileprovider
add fileprovider app to INSTALLED_APPS settings,
add fileprovider.middleware.FileProviderMiddleware to MIDDLEWARE_CLASSES settings
set FILEPROVIDER_NAME settings to nginx or apache in production, by default it is python for development purpose.
in your class-based or function views, set the response header X-File value to the absolute path of the file. For example:
def hello(request):
# code to check or protect the file from unauthorized access
response = HttpResponse()
response['X-File'] = '/absolute/path/to/file'
return response
django-fileprovider implemented in a way that your code will need only minimum modification.
Nginx configuration
To protect file from direct access you can set the configuration as
location /files/ {
internal;
root /home/sideffect0/secret_files/;
}
Here nginx sets a location url /files/ only access internaly, if you are using above configuration you can set X-File as:
response['X-File'] = '/files/filename.extension'
By doing this with nginx configuration, the file will be protected & also you can control the file from django views
def qrcodesave(request):
import urllib2;
url ="http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl=s&chld=H|0";
opener = urllib2.urlopen(url);
content_type = "application/octet-stream"
response = HttpResponse(opener.read(), content_type=content_type)
response["Content-Disposition"]= "attachment; filename=aktel.png"
return response
Django recommend that you use another server to serve static media (another server running on the same machine is fine.) They recommend the use of such servers as lighttp.
This is very simple to set up. However. if 'somefile.txt' is generated on request (content is dynamic) then you may want django to serve it.
Django Docs - Static Files
Another project to have a look at: http://readthedocs.org/docs/django-private-files/en/latest/usage.html
Looks promissing, haven't tested it myself yet tho.
Basically the project abstracts the mod_xsendfile configuration and allows you to do things like:
from django.db import models
from django.contrib.auth.models import User
from private_files import PrivateFileField
def is_owner(request, instance):
return (not request.user.is_anonymous()) and request.user.is_authenticated and
instance.owner.pk = request.user.pk
class FileSubmission(models.Model):
description = models.CharField("description", max_length = 200)
owner = models.ForeignKey(User)
uploaded_file = PrivateFileField("file", upload_to = 'uploads', condition = is_owner)
I have faced the same problem more then once and so implemented using xsendfile module and auth view decorators the django-filelibrary. Feel free to use it as inspiration for your own solution.
https://github.com/danielsokolowski/django-filelibrary
Providing protected access to static html folder using https://github.com/johnsensible/django-sendfile: https://gist.github.com/iutinvg/9907731
I did a project on this. You can look at my github repo:
https://github.com/nishant-boro/django-rest-framework-download-expert
This module provides a simple way to serve files for download in django rest framework using Apache module Xsendfile. It also has an additional feature of serving downloads only to users belonging to a particular group