Develop Reference

Django download a BinaryField as a CSV file

I am working with a legacy project and we need to implement a Django Admin that helps download a csv report that was stored as a BinaryField.
The model is something like this:
class MyModel(models.Model):
csv_report = models.BinaryField(blank=True,null=True)
Everything seems to being stored as expected but I have no clue how to decode the field back to a csv file for later use.
I am using something like these (as an admin action on MyModelAdmin class)
class MyModelAdmin(admin.ModelAdmin):
...
...
actions = ["download_file",]
def download_file(self, request,queryset):
# just getting one for testing
contents = queryset[0].csv_report
encoded_data = base64.b64encode(contents).decode()
with open("report.csv", "wb") as binary_file:
# Write bytes to file
decoded_image_data = base64.decodebytes(encoded_data)
binary_file.write(decoded_image_data)
response = HttpResponse(encoded_data)
response['Content-Disposition'] = 'attachment; filename=report.csv'
return response
download_file.short_description = "file"
But all I download is a scrambled csv file. I don't seem to understand if it is a problem of the format I am using to decode (.decode('utf-8') does nothing either )
PD:
I know it is a bad practice to use BinaryField for this. But requirements are requirements. Nothing to do about it.
EDIT:
As #TimRoberts pointed out, encoding and then decoding is REALLY silly :$. I've changed the method like so:
def download_file(self, request,queryset):
# print(self,request)
contents = queryset[0].csv_report
# print(type(contents))
encoded_data = base64.b64decode(contents)
with open("my_file.csv", "wb") as binary_file:
binary_file.write(encoded_data)
response = HttpResponse(encoded_data)
response['Content-Disposition'] = 'attachment; filename=blob.csv'
return response
download_file.short_description = "file"
Still I am getting a csv file with something like this:

A big fat case of the old RTFM: I was getting carried away by the all base64.. Obviously I didn't have any idea of what I was doing.
After tampering with the shell and reading the docs, I just changed my method to:
def download_file(self, request,queryset):
**contents = bytes(queryset[0].csv_report)**
response = HttpResponse(contents)
response['ContentDisposition']='attachment;filename=report.csv'
return response
Note that I was scrambling the data on purpose by doing the encoded_data = base64.b64decode(contents) stuff. I just needed to apply bytes on my BinaryField and voilá

How to generate a file without saving it to disk in python?

I'm using Python 2.7 and Django 1.7.
I have a method in my admin interface that generates some kind of a csv file.
def generate_csv(args):
...
#some code that generates a dictionary to be written as csv
....
# this creates a directory and returns its filepath
dirname = create_csv_dir('stock')
csvpath = os.path.join(dirname, 'mycsv_file.csv')
fieldnames = [#some field names]
# this function creates the csv file in the directory shown by the csvpath
newcsv(data, csvheader, csvpath, fieldnames)
# this automatically starts a download from that directory
return HttpResponseRedirect('/media/csv/stock/%s' % csvfile)
All in all I create a csv file, save it somewhere on the disk, and then pass its URL to the user for download.
I was thinking if all this can be done without writing to disc. I googled around a bit and maybe content disposition attachment might help me, but I got lost in documentation a bit.
Anyway if there's an easier way of doing this I'd love to know.

Thanks to #Ragora, you pointed me towards the right direction.
I rewrote the newcsv method:
from io import StringIO
import csv
def newcsv(data, csvheader, fieldnames):
"""
Create a new csv file that represents generated data.
"""
new_csvfile = StringIO.StringIO()
wr = csv.writer(new_csvfile, quoting=csv.QUOTE_ALL)
wr.writerow(csvheader)
wr = csv.DictWriter(new_csvfile, fieldnames = fieldnames)
for key in data.keys():
wr.writerow(data[key])
return new_csvfile
and in the admin:
csvfile = newcsv(data, csvheader, fieldnames)
response = HttpResponse(csvfile.getvalue(), content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename=stock.csv'
return response

If it annoys you that you are saving a file to disk, just add the application/octet-stream content-type to the Content-Disposition header then delete the file from disk.
If this header (Content-Disposition) is used in a response with the application/octet- stream content-type, the implied suggestion is that the user agent should not display the response, but directly enter a `save response as...' dialog.

Serving Excel(xlsx) file to the user for download in Django(Python)

I'm trying create and serve excel files using Django. I have a jar file which gets parameters and produces an excel file according to parameters and it works with no problem. But when i'm trying to get the produced file and serve it to the user for download the file comes out broken. It has 0kb size. This is the code piece I'm using for excel generation and serving.
def generateExcel(request,id):
if os.path.exists('./%s_Report.xlsx' % id):
excel = open("%s_Report.xlsx" % id, "r")
output = StringIO.StringIO(excel.read())
out_content = output.getvalue()
output.close()
response = HttpResponse(out_content,content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = 'attachment; filename=%s_Report.xlsx' % id
return response
else:
args = ['ServerExcel.jar', id]
result = jarWrapper(*args) # this creates the excel file with no problem
if result:
excel = open("%s_Report.xlsx" % id, "r")
output = StringIO.StringIO(excel.read())
out_content = output.getvalue()
output.close()
response = HttpResponse(out_content,content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = 'attachment; filename=%s_Report.xlsx' % id
return response
else:
return HttpResponse(json.dumps({"no":"excel","no one": "cries"}))
I have searched for possible solutions and tried to use File Wrapper also but the result did not changed. I assume i have problem with reading the xlsx file into StringIO object. But dont have any idea about how to fix it

Why on earth are you passing your file's content to a StringIO just to assign StringIO.get_value() to a local variable ? What's wrong with assigning file.read() to your variable directly ?
def generateExcel(request,id):
path = './%s_Report.xlsx' % id # this should live elsewhere, definitely
if os.path.exists(path):
with open(path, "r") as excel:
data = excel.read()
response = HttpResponse(data,content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = 'attachment; filename=%s_Report.xlsx' % id
return response
else:
# quite some duplication to fix down there
Now you may want to check weither you actually had any content in your file - the fact that the file exists doesn't mean it has anything in it. Remember that you're in a concurrent context, you can have one thread or process trying to read the file while another (=>another request) is trying to write it.

In addition to what Bruno says, you probably need to open the file in binary mode:
excel = open("%s_Report.xlsx" % id, "rb")

You can use this library to create excel sheets on the fly.
http://xlsxwriter.readthedocs.io/
For more information see this page. Thanks to #alexcxe
XlsxWriter object save as http response to create download in Django

my answer is:
def generateExcel(request,id):
if os.path.exists('./%s_Report.xlsx' % id):
with open('./%s_Report.xlsx' % id, "rb") as file:
response = HttpResponse(file.read(),content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = 'attachment; filename=%s_Report.xlsx' % id
return response
else:
# quite some duplication to fix down there
why using "rb"? because HttpResponse class init parameters is (self, content=b'', *args, **kwargs), so we should using "rb" and using .read() to get the bytes.

Django httpresponse stripping CR

When I open my text file attachment that is generated using the below code, the HTTP response always seems to strip out the CR from each line, the users of this file will be using Notepad so I need CR/LF on each line.
the_file = tempfile.TemporaryFile(mode='w+b')
<procedure call to generate lines of text in "the_file">
the_file.seek(0)
filestring = the_file.read()
response = HttpResponse(filestring,
mimetype="text/plain")
response['Content-Length'] = the_file.tell()
response['Content-Disposition'] = 'attachment; filename="4cos_example.txt"'
return response
If I use this method, I get CR/LF in my files but I'd like to avoid having to write the file to disk at all, so it doesn't seem to be a good solution:
the_file = open('myfile.txt','w+')
<procedure call to generate lines of text in "the_file">
the_file.close
the_file = open('myfile.txt','rb')
filestring = the_file.read()
response = HttpResponse(filestring,
mimetype="text/plain")
response['Content-Length'] = the_file.tell()
response['Content-Disposition'] = 'attachment; filename="4cos_example.txt"'
return response
I feel like the solution should be obvious. but I can't close a tempfile and re-open it in binary mode (preserving the CR/LR). Heck I'm not even sure i'm in the right ballpark regarding how to correctly do this :) None the less, I'd like to pass this data as an attachment to the user after the configuration is assembled and have it display correctly in notepad. Is tempfile the wrong solution here or is there a mechanic of tempfile that will solve this issue for me without having to use file IO on disk.

Instead of using TemporaryFile, just use HttpResponse:
response = HttpResponse('', content_type='text/plain')
response['Content-Disposition'] = 'attachment; filename="4cos_example.txt"'
response.write('first line\r\n')
response.write('second line\r\n')
return response
FYI, if this is a very large response, you can also use StreamingHttpResponse. But only do that if required, since headers like Content-Length will not be able to be added automatically.

How can I serve temporary files from Python Pyramid

Currently, I'm just serving files like this:
# view callable
def export(request):
response = Response(content_type='application/csv')
# use datetime in filename to avoid collisions
f = open('/temp/XML_Export_%s.xml' % datetime.now(), 'r')
# this is where I usually put stuff in the file
response.app_iter = f
response.headers['Content-Disposition'] = ("attachment; filename=Export.xml")
return response
The problem with this is that I can't close or, even better, delete the file after the response has been returned. The file gets orphaned. I can think of some hacky ways around this, but I'm hoping there's a standard way out there somewhere. Any help would be awesome.

You do not want to set a file pointer as the app_iter. This will cause the WSGI server to read the file line by line (same as for line in file), which is typically not the most efficient way to control a file upload (imagine one character per line). Pyramid's supported way of serving files is via pyramid.response.FileResponse. You can create one of these by passing a file object.
response = FileResponse('/some/path/to/a/file.txt')
response.headers['Content-Disposition'] = ...
Another option is to pass a file pointer to app_iter but wrap it in the pyramid.response.FileIter object, which will use a sane block size to avoid just reading the file line by line.
The WSGI specification has strict requirements that response iterators which contain a close method will be invoked at the end of the response. Thus setting response.app_iter = open(...) should not cause any memory leaks. Both FileResponse and FileIter also support a close method and will thus be cleaned up as expected.
As a minor update to this answer I thought I'd explain why FileResponse takes a file path and not a file pointer. The WSGI protocol provides servers an optional ability to provide an optimized mechanism for serving static files via environ['wsgi.file_wrapper']. FileResponse will automatically handle this if your WSGI server has provided that support. With this in mind, you find it to be a win to save your data to a tmpfile on a ramdisk and providing the FileResponse with the full path, instead of trying to pass a file pointer to FileIter.
http://docs.pylonsproject.org/projects/pyramid/en/1.4-branch/api/response.html#pyramid.response.FileResponse

Update:
Please see Michael Merickel's answer for a better solution and explanation.
If you want to have the file deleted once response is returned, you can try the following:
import os
from datetime import datetime
from tempfile import NamedTemporaryFile
# view callable
def export(request):
response = Response(content_type='application/csv')
with NamedTemporaryFile(prefix='XML_Export_%s' % datetime.now(),
suffix='.xml', delete=True) as f:
# this is where I usually put stuff in the file
response = FileResponse(os.path.abspath(f.name))
response.headers['Content-Disposition'] = ("attachment; filename=Export.xml")
return response
You can consider using NamedTemporaryFile:
NamedTemporaryFile(prefix='XML_Export_%s' % datetime.now(), suffix='.xml', delete=True)
Setting delete=True so that the file is deleted as soon as it is closed.
Now, with the help of with you can always have the guarantee that the file will be closed, and hence deleted:
from tempfile import NamedTemporaryFile
from datetime import datetime
# view callable
def export(request):
response = Response(content_type='application/csv')
with NamedTemporaryFile(prefix='XML_Export_%s' % datetime.now(),
suffix='.xml', delete=True) as f:
# this is where I usually put stuff in the file
response.app_iter = f
response.headers['Content-Disposition'] = ("attachment; filename=Export.xml")
return response

The combination of Michael and Kay's response works great under Linux/Mac but won't work under Windows (for auto-deletion). Windows doesn't like the fact that FileResponse tries to open the already open file (see description of NamedTemporaryFile).
I worked around this by creating a FileDecriptorResponse class which is essentially a copy of FileResponse, but takes the file descriptor of the open NamedTemporaryFile. Just replace the open with a seek(0) and all the path based calls (last_modified, content_length) with their fstat equivalents.
class FileDescriptorResponse(Response):
"""
A Response object that can be used to serve a static file from an open
file descriptor. This is essentially identical to Pyramid's FileResponse
but takes a file descriptor instead of a path as a workaround for auto-delete
not working with NamedTemporaryFile under Windows.
``file`` is a file descriptor for an open file.
``content_type``, if passed, is the content_type of the response.
``content_encoding``, if passed is the content_encoding of the response.
It's generally safe to leave this set to ``None`` if you're serving a
binary file. This argument will be ignored if you don't also pass
``content-type``.
"""
def __init__(self, file, content_type=None, content_encoding=None):
super(FileDescriptorResponse, self).__init__(conditional_response=True)
self.last_modified = fstat(file.fileno()).st_mtime
if content_type is None:
content_type, content_encoding = mimetypes.guess_type(path,
strict=False)
if content_type is None:
content_type = 'application/octet-stream'
self.content_type = content_type
self.content_encoding = content_encoding
content_length = fstat(file.fileno()).st_size
file.seek(0)
app_iter = FileIter(file, _BLOCK_SIZE)
self.app_iter = app_iter
# assignment of content_length must come after assignment of app_iter
self.content_length = content_length
Hope that's helpful.

There is also repoze.filesafe which will take care of generating a temporary file for you, and delete it at the end. I use it for saving files uploaded to my server. Perhaps it can be useful to you too.

Because your Object response is holding a file handle for the file '/temp/XML_Export_%s.xml'. Use del statement to delete handle 'response.app_iter'.
del response.app_iter

both Michael Merickel and Kay Zhu are fine.
I found out that I also need to reset file position at the begninnign of the NamedTemporaryFile before passing it to response, as it seems that the response starts from the actual position in the file and not from the beginning (It's fine, you just need to now it).
With NamedTemporaryFile with deletion set, you can not close and reopen it, because it would delete it (and you can't reopen it anyway), so you need to use something like this:
f = tempfile.NamedTemporaryFile()
#fill your file here
f.seek(0, 0)
response = FileResponse(
f,
request=request,
content_type='application/csv'
)
hope it helps ;)