I've a Python/Flask app that is working ok locally. I have deployed it to the cloud (pythonanywhere) and it is all working on there as well except for a file that is being downloaded to the user that is coming as html, so the empty lines of the file are being excluded. That file is txt. When the user click on that, it opens on notepad. If opening that file in notepad++ the empty lines are there in the way it should be.
Following the Flask code to send that file:
response = make_response(result)
response.headers["Content-Disposition"] = "attachment; filename=file_to_user.txt"
If I use "inline instead of attachment", the empty lines are showed OK directly on the browser.
I've tried to add "Content type text/plain" before "Content-Disposition", but I believe that it is the default, so, no effect.
Anyone knows how could the user see that as txt file, instead of html when opening directly using notepad for example?
If you're just trying to send an existing file on the server, use send_from_directory.
If you're trying to make a response (for example, if you're generating data in memory, make_response defaults to text/html (it's just a shortcut which isn't applicable in your case). Create a response even more directly in order to override that using app.response_class.
This is a small example demonstrating both techniques.
from flask import Flask, send_from_directory
app = Flask(__name__)
#app.route('/file')
def download_file():
# change app.root_path to whatever the directory actually is
# this just serves this python file (named example.py) as plain text
return send_from_directory(
app.root_path, 'example.py',
as_attachment=True, mimetype='text/plain'
)
#app.route('/mem')
def download_mem():
# instantiate the response class directly
# pass the mimetype
r = app.response_class('test data\n\ntest data', mimetype='text/plain')
# add the attachment header
r.headers.set('Content-Disposition', 'attachment', filename='test_data.txt')
return r
app.run('localhost', debug=True)
I am wondering what is the best and possibly easiest way to serve files from GridFS using Pyramid. I use nginx as a proxy server (for ssl) and waitress as my application server.
The file types I need to be able to serve are the following: mp3, pdf, jpg, png
The files should be accessible through the following url "/files/{userid}/{filename}"
Right now the files are opened by the right application on client-side because I explicitly set the content-type in my code like so:
if filename[-3:] == "pdf":
response = Response(content_type='application/pdf')
elif filename[-3:] in ["jpg", "png"]:
response = Response(content_type='image/*')
elif filename[-3:] in ["mp3"]:
response = Response(content_type='audio/mp3')
else:
response = Response(content_type="application/*")
response.app_iter = file #file is a GridFS file object
return response
The only thing is that I can't stream the mp3s properly. I use audio.js to play them. They open up and play but no track length is shown and I can't seek them. I know it has something to do with the "accept-ranges" property but I can't seem to set it right. Does it have to do with nginx or waitress? Or am I just not setting the header correctly?
I would like to use something as easy as return FileResponse(file) like specified here but my file does not come from the filesystem directly... Is there a plug and play way to make this work?
Any advice would be really appreciated!
Thank you very much for your help!
I found a solution on this blog.
The idea is to use a patched DataApp from paste.fileapp. All the details are in the post, and now my app behaves just like I want!
I just solved the problem without the paste dependency in Pyramid 1.4, Python 3.
It seems that the attribute "conditional_response=True" and the "content_length" is important:
f = request.db.fs.files.find_one( { 'filename':filename, 'metadata.bucket': bucket } )
fs = gridfs.GridFS( request.db )
with fs.get( f.get( '_id') ) as gridout:
response = Response(content_type=gridout.content_type,body_file=gridout,conditional_response=True)
response.content_length = f.get('length')
return response
Another way (using Pyramid 1.5.7 on Python 2.7):
fs = GridFS(request.db, 'MyFileCollection')
grid_out = fs.get(file_id)
response = request.response
response.app_iter = FileIter(grid_out)
response.content_disposition = 'attachment; filename="%s"' % grid_out.name
return response
I have a few different applications that require me to POST a file FROM Google App Engine to a remote site. I've tried a few approaches with urllib2, but I've run into problems with each approach as I have moved the code into GAE.
What is the simplest way to post a file (csv, zip, etc.) from Google App Engine to a remote website? Once I can post an existing file, I can move on to posting files from the datastore.
Have you looked at urlfetch. Example from docs.
import urllib
from google.appengine.api import urlfetch
with open('/file', 'r') as f:
data = f.read()
result = urlfetch.fetch(url=url,
payload=data,
method=urlfetch.POST,
headers={'Content-Type': 'application/x-www-form-urlencoded'})
From the reference,
payload: Body content for a POST or PUT request.
So just load the contents of the file and set the payload a la
with open(filename, 'r') as fh:
payload = fh.read()
response = urlfetch.fetch(url, payload=payload, method='POST')
and do what you would with response.
This would work in the exact same fashion with a string from a datastore object.
EDIT: filename will likely be a path relative to your project. So if your project lives in /home/dinosaurs/sinclair on your local machine and you have /home/dinosaurs/sinclair/stuff/contents.xml in your project, then your relative path that will work in production on App Engine is stuff/contents.xml.
I'm trying to upload files to the blobstore in my Google App without using a form. But I'm stuck at how to get the app to read my local file. I'm pretty new to python and Google Apps but after some cut and pasting I've ended up with this:
import webapp2
import urllib
import os
from google.appengine.api import files
from poster.encode import multipart_encode
class Upload(webapp2.RequestHandler):
def get(self):
# Create the file in blobstore
file_name = files.blobstore.create(mime_type='application/octet-stream')
# Get the local file path from an url param
file_path = self.request.get('file')
# Read the file
file = open(file_path, "rb")
datagen, headers = multipart_encode({"file": file})
data = str().join(datagen) # this is supposedly memory intense and slow
# Open the blobstore file and write to it
with files.open(file_name, 'a') as f:
f.write(data)
# Finalize the file. Do this before attempting to read it.
files.finalize(file_name)
# Get the file's blob key
blob_key = files.blobstore.get_blob_key(file_name)
The problem now is I don't really know how to get hold of the local file
You can't read from the local file system from within the application itself, you will need to use http POST to send the file to the app.
You can certainly do this from within another application - you just need to create the mime multipart message with the file content and POST it to your app, the sending application will just have to create the http request that you will post to the app manually. You should have a read on how to create a mime mulitpart message using c#.
I want users on the site to be able to download files whose paths are obscured so they cannot be directly downloaded.
For instance, I'd like the URL to be something like this: http://example.com/download/?f=somefile.txt
And on the server, I know that all downloadable files reside in the folder /home/user/files/.
Is there a way to make Django serve that file for download as opposed to trying to find a URL and View to display it?
For the "best of both worlds" you could combine S.Lott's solution with the xsendfile module: django generates the path to the file (or the file itself), but the actual file serving is handled by Apache/Lighttpd. Once you've set up mod_xsendfile, integrating with your view takes a few lines of code:
from django.utils.encoding import smart_str
response = HttpResponse(mimetype='application/force-download') # mimetype is replaced by content_type for django 1.7
response['Content-Disposition'] = 'attachment; filename=%s' % smart_str(file_name)
response['X-Sendfile'] = smart_str(path_to_file)
# It's usually a good idea to set the 'Content-Length' header too.
# You can also set any other required headers: Cache-Control, etc.
return response
Of course, this will only work if you have control over your server, or your hosting company has mod_xsendfile already set up.
EDIT:
mimetype is replaced by content_type for django 1.7
response = HttpResponse(content_type='application/force-download')
EDIT:
For nginx check this, it uses X-Accel-Redirect instead of apache X-Sendfile header.
A "download" is simply an HTTP header change.
See http://docs.djangoproject.com/en/dev/ref/request-response/#telling-the-browser-to-treat-the-response-as-a-file-attachment for how to respond with a download.
You only need one URL definition for "/download".
The request's GET or POST dictionary will have the "f=somefile.txt" information.
Your view function will simply merge the base path with the "f" value, open the file, create and return a response object. It should be less than 12 lines of code.
For a very simple but not efficient or scalable solution, you can just use the built in django serve view. This is excellent for quick prototypes or one-off work, but as has been mentioned throughout this question, you should use something like apache or nginx in production.
from django.views.static import serve
filepath = '/some/path/to/local/file.txt'
return serve(request, os.path.basename(filepath), os.path.dirname(filepath))
S.Lott has the "good"/simple solution, and elo80ka has the "best"/efficient solution. Here is a "better"/middle solution - no server setup, but more efficient for large files than the naive fix:
http://djangosnippets.org/snippets/365/
Basically, Django still handles serving the file but does not load the whole thing into memory at once. This allows your server to (slowly) serve a big file without ramping up the memory usage.
Again, S.Lott's X-SendFile is still better for larger files. But if you can't or don't want to bother with that, then this middle solution will gain you better efficiency without the hassle.
Just mentioning the FileResponse object available in Django 1.10
Edit: Just ran into my own answer while searching for an easy way to stream files via Django, so here is a more complete example (to future me). It assumes that the FileField name is imported_file
views.py
from django.views.generic.detail import DetailView
from django.http import FileResponse
class BaseFileDownloadView(DetailView):
def get(self, request, *args, **kwargs):
filename=self.kwargs.get('filename', None)
if filename is None:
raise ValueError("Found empty filename")
some_file = self.model.objects.get(imported_file=filename)
response = FileResponse(some_file.imported_file, content_type="text/csv")
# https://docs.djangoproject.com/en/1.11/howto/outputting-csv/#streaming-large-csv-files
response['Content-Disposition'] = 'attachment; filename="%s"'%filename
return response
class SomeFileDownloadView(BaseFileDownloadView):
model = SomeModel
urls.py
...
url(r'^somefile/(?P<filename>[-\w_\\-\\.]+)$', views.SomeFileDownloadView.as_view(), name='somefile-download'),
...
Tried #Rocketmonkeys solution but downloaded files were being stored as *.bin and given random names. That's not fine of course. Adding another line from #elo80ka solved the problem.
Here is the code I'm using now:
from wsgiref.util import FileWrapper
from django.http import HttpResponse
filename = "/home/stackoverflow-addict/private-folder(not-porn)/image.jpg"
wrapper = FileWrapper(file(filename))
response = HttpResponse(wrapper, content_type='text/plain')
response['Content-Disposition'] = 'attachment; filename=%s' % os.path.basename(filename)
response['Content-Length'] = os.path.getsize(filename)
return response
You can now store files in a private directory (not inside /media nor /public_html) and expose them via django to certain users or under certain circumstances.
Hope it helps.
Thanks to #elo80ka, #S.Lott and #Rocketmonkeys for the answers, got the perfect solution combining all of them =)
It was mentioned above that the mod_xsendfile method does not allow for non-ASCII characters in filenames.
For this reason, I have a patch available for mod_xsendfile that will allow any file to be sent, as long as the name is url encoded, and the additional header:
X-SendFile-Encoding: url
Is sent as well.
http://ben.timby.com/?p=149
Try: https://pypi.python.org/pypi/django-sendfile/
"Abstraction to offload file uploads to web-server (e.g. Apache with mod_xsendfile) once Django has checked permissions etc."
You should use sendfile apis given by popular servers like apache or nginx in production. For many years I was using the sendfile api of these servers for protecting files. Then created a simple middleware based django app for this purpose suitable for both development & production purposes. You can access the source code here.
UPDATE: in new version python provider uses django FileResponse if available and also adds support for many server implementations from lighthttp, caddy to hiawatha
Usage
pip install django-fileprovider
add fileprovider app to INSTALLED_APPS settings,
add fileprovider.middleware.FileProviderMiddleware to MIDDLEWARE_CLASSES settings
set FILEPROVIDER_NAME settings to nginx or apache in production, by default it is python for development purpose.
in your class-based or function views, set the response header X-File value to the absolute path of the file. For example:
def hello(request):
# code to check or protect the file from unauthorized access
response = HttpResponse()
response['X-File'] = '/absolute/path/to/file'
return response
django-fileprovider implemented in a way that your code will need only minimum modification.
Nginx configuration
To protect file from direct access you can set the configuration as
location /files/ {
internal;
root /home/sideffect0/secret_files/;
}
Here nginx sets a location url /files/ only access internaly, if you are using above configuration you can set X-File as:
response['X-File'] = '/files/filename.extension'
By doing this with nginx configuration, the file will be protected & also you can control the file from django views
def qrcodesave(request):
import urllib2;
url ="http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl=s&chld=H|0";
opener = urllib2.urlopen(url);
content_type = "application/octet-stream"
response = HttpResponse(opener.read(), content_type=content_type)
response["Content-Disposition"]= "attachment; filename=aktel.png"
return response
Django recommend that you use another server to serve static media (another server running on the same machine is fine.) They recommend the use of such servers as lighttp.
This is very simple to set up. However. if 'somefile.txt' is generated on request (content is dynamic) then you may want django to serve it.
Django Docs - Static Files
Another project to have a look at: http://readthedocs.org/docs/django-private-files/en/latest/usage.html
Looks promissing, haven't tested it myself yet tho.
Basically the project abstracts the mod_xsendfile configuration and allows you to do things like:
from django.db import models
from django.contrib.auth.models import User
from private_files import PrivateFileField
def is_owner(request, instance):
return (not request.user.is_anonymous()) and request.user.is_authenticated and
instance.owner.pk = request.user.pk
class FileSubmission(models.Model):
description = models.CharField("description", max_length = 200)
owner = models.ForeignKey(User)
uploaded_file = PrivateFileField("file", upload_to = 'uploads', condition = is_owner)
I have faced the same problem more then once and so implemented using xsendfile module and auth view decorators the django-filelibrary. Feel free to use it as inspiration for your own solution.
https://github.com/danielsokolowski/django-filelibrary
Providing protected access to static html folder using https://github.com/johnsensible/django-sendfile: https://gist.github.com/iutinvg/9907731
I did a project on this. You can look at my github repo:
https://github.com/nishant-boro/django-rest-framework-download-expert
This module provides a simple way to serve files for download in django rest framework using Apache module Xsendfile. It also has an additional feature of serving downloads only to users belonging to a particular group