Develop Reference

Flask api request, read csv and save as json

SITE_ROOT = os.path.realpath(os.path.dirname(__file__))
json_url = os.path.join(SITE_ROOT, "data", "results.json")
json_data = json.loads(open(json_url).read()).
What I want to hopefully do is, that it reads my csv file, column - question, and loads it as json format.

you can use pandas for this issue i think.
import pandas as pd
read_csv = pd.read_csv('path/csv_name.csv',delimiter= ',') # or delimiter = ';'
read_csv.head() # display your data and check it
csv_to_json = read_csv.to_json(orient = 'columns')

Django / Pandas - Create Excel file and serve as download

I am trying to create an Excel file using pandas and serving it to the user as a downloadable file via Django. I put together some different answers on the topic that I found on here and ended up with this code:
collection = [{"title": "something", "price": 34, "quantity": 23}, {..}]
output = BytesIO()
df = pd.DataFrame(collection, columns=['title', 'price', 'quantity'])
writer = pd.ExcelWriter(output, engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1')
writer.save()
output.seek(0)
workbook = output.getvalue()
response = StreamingHttpResponse(workbook, content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = f'attachment; filename={output_name}.xlsx'
return response
It all works well until I try to open the resulting file - I can an error saying that the file is damaged or that there is something wrong with the data-format. I suspect that it could have something to do with the data being binary? How can I resolve this issue?
SOLUTION
Turns out I had to remove some stuff so the code looks like this now and works fine:
collection = [{"title": "something", "price": 34, "quantity": 23}, {..}]
output = BytesIO()
df = pd.DataFrame(collection, columns=['title', 'price', 'quantity'])
writer = pd.ExcelWriter(output, engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1')
writer.save()
output.seek(0)
# workbook = output.getvalue()
response = StreamingHttpResponse(output, content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = f'attachment; filename={output_name}.xlsx'
return response

I think you might be making that a lot more complicated than it needs to be.
Below works fine for me:
import pandas as pd
from django.http import HttpResponse
df = pd.DataFrame(data)
response = HttpResponse(content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = 'attachment; filename="filename.xlsx"'
df.to_excel(response)
return response

SOLUTION
Turns out I had to remove some stuff so the code looks like this now and works fine:
collection = [{"title": "something", "price": 34, "quantity": 23}, {..}]
output = BytesIO()
df = pd.DataFrame(collection, columns=['title', 'price', 'quantity'])
writer = pd.ExcelWriter(output, engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1')
writer.save()
output.seek(0)
# workbook = output.getvalue()
response = StreamingHttpResponse(output, content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = f'attachment; filename={output_name}.xlsx'
return response

Probably a datatype issue when opening in Excel, try converting the data into strings and then create excel and try.
Another thought is to create file with a sample set of records, rather than whole frame to validate if its a data issue. There might be an issue with Nan's in the dataset as well. Check if you need to ignore/convert/replace that.

Merging two PDF generating views in Django

To the new readers, I'd really appreciate it if you could suggest how I could improve my question. Thanks!
I have two working PDF generating views. I would like to create a view that combines these two views & merges the output PDFs produced by the two views to generate one PDF containing the output PDFs of the two views.
I would also like to specify that these two views use different approaches to generate PDFs. The first one renders an HTML template to create PDF & the second creates a PDF from scratch using Reportlab.
I am new to Django, How can I do this?
View 1
class GenerateAllDocs(View):
def allDocGen(request):
if request.method == 'POST':
all_doc_gen_form = GenerateAllForms(request.POST)
if all_doc_gen_form.is_valid():
some_Field_1= all_doc_gen_form.cleaned_data['some_Field_1']
some_Field_2= all_doc_gen_form.cleaned_data['some_Field_2']
template = get_template('PDF_templates/att_pg_pdf_template.html')
data = {
'some_Field_1': some_Field_1,
'some_Field_2': some_Field_2,
}
html = template.render(data)
pdf = render_to_pdf('PDF_templates/att_pg_pdf_template.html', data)
if pdf:
response = HttpResponse(pdf, content_type = 'application/pdf')
filename = "something - %s.pdf" %(data.get('zzzzz'))
content = "inline; filename=%s" %(filename)
download = request.GET.get('download')
if download:
content = "attachment; filename%s" %(filename)
response['Content-Disposition'] = content
return response
return HttpResponse('Not Found')
all_doc_gen_form = GenerateAllForms()
return render(request, 'form_UI_templates/pg_att_form_UI_template.html', {'all_doc_gen_form':all_doc_gen_form})
View 2
def template_PDF_view(request):
# Create the HttpResponse object with the appropriate PDF headers.
response = HttpResponse(content_type='application/pdf')
response['Content-Disposition'] = 'attachment; inline; filename="somefilename.pdf"'
buffer = BytesIO()
# Create the PDF object, using the BytesIO object as its "file."
p = canvas.Canvas(buffer, pagesize=letter)
# Draw things on the PDF. Here's where the PDF generation happens.
p.drawImage('D:/worrk/PyDjango/mysite - PROD VERSION/main/static/images/cerfaImg.jpg',0,0, width=8.27 * inch, height= 11.69 * inch)
p.drawString(40, 724, " ".join('XX-XXX-XX'.replace('-','').upper())) # car_licence_plate
p.drawString(193, 149, 'hello 123') # address_city
# Close the PDF object cleanly.
p.showPage()
p.save()
# Get the value of the BytesIO buffer and write it to the response.
pdf = buffer.getvalue()
buffer.close()
response.write(pdf)
return response
EDIT #1
Following suggestion by ktowen, here is my updated code. I created a function based view for my view 2 and call that function inside of view 1 & then attempt to merge the two PDFs created by the views. This still doesn't work - error message is written after code.
def create_cerfa(request):
response = HttpResponse(content_type='application/pdf')
response['Content-Disposition'] = 'attachment; filename="somefilename.pdf"'
buffer_1 = BytesIO()
p = canvas.Canvas(buffer_1)
p.drawImage('D:/worrk/PyDjango/mysite - PROD VERSION/main/static/images/cerfaImg.jpg',0,0, width=8.27 * inch, height= 11.69 * inch)
p.drawString(40, 724, " ".join('AZ-343-BT'.replace('-','').upper()))
p.drawString(178, 724, " ".join('VF77JNFUC9J177958').upper())
p.save()
pdf = buffer_1.getvalue()
buffer_1.close()
response.write(pdf)
return response
def createAttestation(request):
if request.method == 'POST':
all_doc_gen_form = GenerateAllForms(request.POST)
if all_doc_gen_form.is_valid():
data1 = all_doc_gen_form.cleaned_data['data1']
data12 = all_doc_gen_form.cleaned_data['data12']
template = get_template('PDF_templates/att_pg_pdf_template.html')
data = {
'data1': data1,
'data12': data12,
}
html = template.render(data)
pdf = render_to_pdf('PDF_templates/att_pg_pdf_template.html', data)
cerfa = create_cerfa(request)
pdf1resp = HttpResponse(pdf, content_type = 'application/pdf')
pdf2resp = HttpResponse(cerfa, content_type = 'application/pdf')
pdfs = [pdf1resp, pdf2resp]
merger = PdfFileMerger()
for item in pdfs:
merger.append(item)
merger.write()
if pdf:
response = HttpResponse(merger, content_type = 'application/pdf')
return response
all_doc_gen_form = GenerateAllForms()
return render(request, 'form_UI_templates/pg_att_form_UI_template.html', {'all_doc_gen_form':all_doc_gen_form})
Error Message
Traceback (most recent call last):
File "C:\Users\Work\AppData\Local\Programs\Python\Python38-32\lib\site-packages\django\core\handlers\exception.py", line 34, in inner
response = get_response(request)
File "C:\Users\Work\AppData\Local\Programs\Python\Python38-32\lib\site-packages\django\core\handlers\base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "C:\Users\Work\AppData\Local\Programs\Python\Python38-32\lib\site-packages\django\core\handlers\base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "D:\worrk\PyDjango\mysite - PROD VERSION\main\views.py", line 460, in createAttestation
merger.append(item)
File "C:\Users\Work\AppData\Local\Programs\Python\Python38-32\lib\site-packages\PyPDF2\merger.py", line 203, in append
self.merge(len(self.pages), fileobj, bookmark, pages, import_bookmarks)
File "C:\Users\Work\AppData\Local\Programs\Python\Python38-32\lib\site-packages\PyPDF2\merger.py", line 133, in merge
pdfr = PdfFileReader(fileobj, strict=self.strict)
File "C:\Users\Work\AppData\Local\Programs\Python\Python38-32\lib\site-packages\PyPDF2\pdf.py", line 1084, in __init__
self.read(stream)
File "C:\Users\Work\AppData\Local\Programs\Python\Python38-32\lib\site-packages\PyPDF2\pdf.py", line 1689, in read
stream.seek(-1, 2)
Exception Type: AttributeError at /tst2/
Exception Value: 'HttpResponse' object has no attribute 'seek'
EDIT #2
So I've narrowed down the problem to the actual code of merging the PDFs. I tested the approach given in EDIT 2 by ktowen.
I commented the merging part and then returned the PDFs as follows
return FileResponse(pdf1, as_attachment=True, content_type='application/pdf')
I checked it with PDF1 & PDF2 - Both are getting generated, but I don't understand what is wrong with the merging part? & how can I fix it?
Minor Edit - #2.1
Here's my Render to PDF function
from __future__ import print_function
from io import BytesIO
from django.http import HttpResponse
from django.template.loader import get_template
from xhtml2pdf import pisa
def render_to_pdf(template_src, context_dict={}):
template = get_template(template_src)
html = template.render(context_dict)
result = BytesIO()
pdf = pisa.pisaDocument(BytesIO(html.encode("ISO-8859-1")), result)
if not pdf.err:
return HttpResponse(result.getvalue(), content_type='application/pdf')
return None
EDIT #3
Here I modified the PDF definition inside the render_to_pdf function as follows.
pdf = pisa.pisaDocument(BytesIO(html.encode("ISO-8859-1")), result)
pdf = result.getvalue()
Now error is 'utf-8' codec can't decode byte 0x93 in position 10: invalid start byte
So to resolve this error, I tried changing the encoding inside the render_to_pdf function. Since I use French a lot in this project, I tested recommended encodings like latin1, UTF-8, & the default ISO-8859-1 but still get the same error.
Following is the full error message
Traceback (most recent call last):
File "C:\Users\Work\AppData\Roaming\Python\Python38\site-packages\django\core\handlers\exception.py", line 47, in inner
response = get_response(request)
File "C:\Users\Work\AppData\Roaming\Python\Python38\site-packages\django\core\handlers\base.py", line 179, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "D:\worrk\PyDjango\mysite - PROD VERSION\main\views.py", line 596, in createAttestation
pdf1 = PdfFileReader(open(p1,'rb'))
Exception Type: UnicodeDecodeError at /tst2/
Exception Value: 'utf-8' codec can't decode byte 0x93 in position 10: invalid start byte

You can extract the pdf generation logic to two functions and use PyPDF2 PdfFileMerger to merge the two pdfs.
This is the general idea:
from PyPDF2 import PdfFileMerger
def merged_PDF_view(request):
# Create the HttpResponse object with the appropriate PDF headers.
response = HttpResponse(content_type='application/pdf')
response['Content-Disposition'] = 'attachment; inline; filename="somefilename.pdf"'
pdf1_buffer = get_pdf1()
pdf2_buffer = get_pdf2()
pdf_merger = PdfFileMerger()
pdf_merger.append(pdf1_buffer)
pdf_merger.append(pdf2_buffer)
# This can probably be improved
pdf_merger.write(buffer)
pdf_merger.close()
buffer.seek(0)
response.write(buffer.getvalue())
return response
EDIT 1
Try with this
class WillThisWork(View): # I hope so :|
def merged_PDF(request):
buffer = BytesIO()
response = HttpResponse(content_type='application/pdf')
response['Content-Disposition'] = 'attachment; inline; filename="somefilename.pdf"'
pdf1_buffer = createAttestation(request)
pdf2_buffer = create_cerfa(request)
pdf_merger = PdfFileMerger()
pdf_merger.append(pdf1_buffer)
pdf_merger.append(pdf2_buffer)
pdf_merger.write(buffer)
pdf_merger.close()
buffer.seek(0)
return FileResponse(buffer, as_attachment=True, filename='hello.pdf')
EDIT 2
Based in your edit try this
def create_cerfa_pdf():
filestream = BytesIO()
p = canvas.Canvas(filestream)
p.drawImage('D:/worrk/PyDjango/mysite - PROD VERSION/main/static/images/cerfaImg.jpg',0,0, width=8.27 * inch, height= 11.69 * inch)
p.drawString(40, 724, " ".join('AZ-343-BT'.replace('-','').upper()))
p.drawString(178, 724, " ".join('VF77JNFUC9J177958').upper())
p.save()
filestream.seek(0)
return filestream
def create_cerfa(request):
pdf = create_cerfa_pdf()
return FileResponse(pdf, as_attachment=True, filename="somefilename.pdf")
def createAttestation(request):
if request.method == 'POST':
all_doc_gen_form = GenerateAllForms(request.POST)
if all_doc_gen_form.is_valid():
data1 = all_doc_gen_form.cleaned_data['data1']
data12 = all_doc_gen_form.cleaned_data['data12']
template = get_template('PDF_templates/att_pg_pdf_template.html')
data = {
'data1': data1,
'data12': data12,
}
html = template.render(data)
pdf1 = render_to_pdf('PDF_templates/att_pg_pdf_template.html', data)
pdf2 = create_cerfa_pdf()
merger = PdfFileMerger()
for item in [pdf1, pdf2]:
merger.append(item)
filestream = BytesIO()
merger.write(filestream)
merger.close()
filestream.seek(0)
return FileResponse(filestream, as_attachment=True)
all_doc_gen_form = GenerateAllForms()
return render(request, 'form_UI_templates/pg_att_form_UI_template.html', {'all_doc_gen_form':all_doc_gen_form})

So I solved it quite a while ago...
In approach 1, I was using an HTML template, & filling data from the form in it and converting it to PDF, and in approach 2 I was using reportlab. So I just re-wrote the approach 1 using reportlab - No merging needeed. the .showpage method creates page breaks.

How to export excel file in django

I need help with exporting data using a template. I installed django-import-export and added it to admin panel, now I can only export data from the admin panel. I want to know how can i export excel file using template.

This should get you started:
import StringIO
import xlsxwriter
from django.http import HttpResponse
def export_page(request):
# create our spreadsheet. I will create it in memory with a StringIO
output = StringIO.StringIO()
workbook = xlsxwriter.Workbook(output)
worksheet = workbook.add_worksheet()
worksheet.write('A1', 'Some Data')
workbook.close()
# create a response
response = HttpResponse(content_type='application/vnd.ms-excel')
# tell the browser what the file is named
response['Content-Disposition'] = 'attachment;filename="some_file_name.xlsx"'
# put the spreadsheet data into the response
response.write(output.getvalue())
# return the response
return response

I tried the same with newer version of Django and after trial and error found this worked.
import io
import xlsxwriter
def excelreport(request):
buffer = io.BytesIO()
workbook = xlsxwriter.Workbook(buffer)
worksheet = workbook.add_worksheet()
worksheet.write('A1', 'Some Data')
workbook.close()
buffer.seek(0)
return FileResponse(buffer, as_attachment=True, filename='report.xlsx')

You can alos use xlwt if you really need to export to a .xls file. You will be able to add formating as bold font, font size, define column size, etc.
$ pip install xlwt
import xlwt
from django.http import HttpResponse
from django.contrib.auth.models import User
def export_users_xls(request):
response = HttpResponse(content_type='application/ms-excel')
response['Content-Disposition'] = 'attachment; filename="users.xls"'
wb = xlwt.Workbook(encoding='utf-8')
ws = wb.add_sheet('Users')
# Sheet header, first row
row_num = 0
font_style = xlwt.XFStyle()
font_style.font.bold = True
columns = ['Username', 'First name', 'Last name', 'Email address', ]
for col_num in range(len(columns)):
ws.write(row_num, col_num, columns[col_num], font_style)
# Sheet body, remaining rows
font_style = xlwt.XFStyle()
rows = User.objects.all().values_list('username', 'first_name', 'last_name', 'email')
for row in rows:
row_num += 1
for col_num in range(len(row)):
ws.write(row_num, col_num, row[col_num], font_style)
wb.save(response)
return response

If you are using pandas, this is probably the easiest and most concise way:
import pandas as pd
from django.http import HttpResponse
def export_excel_file(request):
df = pd.read_excel("excel_filename.xlsx")
response = HttpResponse(content_type='application/vnd.ms-excel')
response['Content-Disposition'] = f'attachment; filename=excel_filename.xlsx'
df.to_excel(response, index=False)
return response

Django return file as string, not as a file

I try to put statistics in excel spreadsheets in dynamic way, so when
excel.js
$('.js-excel').on('click', function () {
$.get(
'/ajax/stat_excel/',
{
'excel': 'loan',
'date_from': $('#date_from').val(),
'date_to': $('#date_to').val()
}
)
})
then
view.py
output = StringIO.StringIO()
workbook = xlsxwriter.Workbook(output)
if request.GET.get('excel') == 'loan':
workbook = loanChart.excel(workbook)
if request.GET.get('excel') == 'debet':
workbook = debetChart.excel(workbook)
workbook.close()
xlsx_data = output.getvalue()
response = HttpResponse(xlsx_data, mimetype='application/vnd.ms-excel')
response['Content-Type'] = 'application/vnd.ms-excel'
response['Content-Disposition'] = 'attachment; filename=report.xlsx'
return response
And I'm not sure what I'm doing wrong, because response be like
PK�������F��AS]$��w������xl/worksheets/sheet1.xml��[oɑ���W|W+#�>�dx(�{}�%j$�$
$������ʞ��8�]C�.��QU���������//�q�����ۛ:�����?|���77��y�n^<=�}�p�������������y����O���,���
Excel file generates excellent. I can see it, if I don't use StringIO
I'm not sure, what I need to use, Mimetype or Content-Type. Can't see any difference. Works exactly the same, no matter which type I write in response.
Where can be my problem?

Found answer here https://stackoverflow.com/a/4518775/4498908.
I can't use ajax for file download. But I can:
function download(path,val) {
window.location.href = path+"download.php?val="+val;
};