I have a script that I'm building an interface for so people can execute after uploading a CSV file. I'm able to execute everything and have the script run properly, but how do I display a continuous output? What changes should I make to my code?
Below are my files:
scripts.html - Scripts are executed from here and executed on the server via an AJAX call. Output is placed into div#output once the script is done executing.
<div class="table_container">
<form method="post" enctype="multipart/form-data" data-ajax="false">{% csrf_token %}
<h4>Rebilling</h4>
<div class="clear"></div>
<img class="loading-gif" src="{{ STATIC_URL }}img/loading-clear.gif" alt="" />
<table>
<tbody>
<tr>
<td style="width: 180px;"><label>Upload the CSV</label></td>
<td>
<input type="hidden" name="script_type" value="renew_subscriptions">
<input type="file" name="csv_file" />
</td>
</tr>
<tr>
<td style="width: 180px;"></td>
<td>
<input type="submit" name="Execute" />
</td>
</tr>
</tbody>
</table>
</form>
</div>
<h2>Script Output</h2>
<div id="output">
{% autoescape off %}
{% endautoescape %}
</div>
<script type="text/javascript">
// Variable to store your files
var files;
// Add events
$('input[type=file]').on('change', prepareUpload);
// Grab the files and set them to our variable
function prepareUpload(event)
{
files = event.target.files;
}
$('form').on('submit', submitForm);
// Catch the form submit and upload the files
function submitForm(event)
{
event.stopPropagation(); // Stop stuff happening
event.preventDefault(); // Totally stop stuff happening
$("#output").html("");
var form = $(this);
form.find(".loading-gif").css("display", "block");
form.find("input[type='submit']").prop('disabled', true);
// Create a formdata object and add the files
var data = new FormData(form.get(0));
$.ajax({
url: '/crm/scripts',
type: 'POST',
data: data,
cache: false,
dataType: 'html',
processData: false,
contentType: false,
success: function(data)
{
// console.dir(data);
$("#output").html(data);
},
error: function(jqXHR, textStatus, errorThrown)
{
// Handle errors here
console.log('ERRORS: ' + textStatus);
},
complete: function()
{
form.find(".loading-gif").css("display", "none");
form.find("input[type='submit']").prop('disabled', false);
}
});
return false;
}
</script>
views.py - The AJAX is sent here and the command is executed through Django Management
def all_scripts(request): # Accounts page
# c = {}
script_type = None
csv_file = None
out = StringIO()
if request.is_ajax and request.method == 'POST':
csv_file = request.FILES.get('csv_file')
if csv_file:
# print "over here"
### write the csv_file to a temp file
tup = tempfile.mkstemp() # make a tmp file
f = os.fdopen(tup[0], 'w') # open the tmp file for writing
f.write(csv_file.read()) # write the tmp file
f.close()
### return the path of the file
filepath = tup[1] # get the filepath
# print filepath
if 'script_type' in request.POST:
script_type = request.POST['script_type']
if script_type == "change_credit":
credit_amount = None
if 'credit_amount' in request.POST:
credit_amount = request.POST['credit_amount']
if 'function' in request.POST:
function = request.POST['function']
if function == "remove":
management.call_command(script_type, filepath, credit_amount, remove=[True], stdout=out)
else:
management.call_command(script_type, filepath, credit_amount, stdout=out)
elif script_type == "renew_subscriptions":
management.call_command(script_type, filepath, verbosity=1, interactive=False, stdout=out)
print out.getvalue()
return HttpResponse(out.getvalue())
return render_to_response('crm/scripts.html', context_instance=RequestContext(request))
Just need the output to display continuously line by line. Any help is much appreciated.
Cheers,
Zee
“The web request is a scary place, you want to get in and out as quick
as you can” - Rick Branson
What you have done here is created an architectural issue. Basically you are creating additional disk IO when writing your CSV file. You are doing this in the web request. 'Not a good idea'.
However its also the crux of the issue you are describing.
Quick n dirty:
you could get a return value from a django management command like so. Pass that back to the success method of jquery's ajax call as your data.
However: please don't to that!
You need an async task system to hand off the write of that csv file. Additionally you want to write the data somewhere ( dbms / nosql ) that your webpage can listen for via ( polling, streaming or websockets ) This is not a trivial undertaking but the end result is well worth the effort. Here are some proven django-stack choices to solve this type of issue.
Building a Asynchronous Tasking/Queuing System
celery
Rabbit the broker used by celery
memcached / mongoDB / reddis
Polling for that data
what are longpolling and websockets
websockets vs long polling
This pycon talk covers these technologies.
Messaging at Scale at Instagram
Related
Please I want to handle the front-end in python as much as possible and make it dynamic by ajax:
this is the client side:
<div id="gene">
<!-- <select id="gene"> -->
<!-- add an empty value -->
<!--<option value="*">-- Select --</option> -->
<!--</select>-->
</div>
And the Ajax function in the same file:
var updateGeneInput = function(){
$.ajax({
url: 'get.py',
type: 'get',
data: {'table':'gene'},
success: function(data){
$("#gene").html(data);
}
});
};
And I want to retrieve the options to populate the drop list using the cgi (get.py):
if (table == "gene"):
query = """
SELECT category
FROM Categories
"""
cursor.execute(query)
rows=cursor.fetchall()
print("""<select >""")
print("""<option value='*'>- Select -</option>""")
for row in rows:
print("""<option value="s%">s%</option>"""%row)
print("""</select>""")
It does show the results but not in drop list.
Thank you very much
Looking for recommendation of a library in Python(first preference) or NodeJS that can generate a pdf file preferably from dynamic html template to be run in AWS. Requirement is to generate invoice pdf to be sent to customers.
Have come across below 2 Node libraries:
PDFKit
jsPDF
Here we might have to deal with numbers for X and Y.
Better approach would be something where we can simply use html/css to generate template with placeholders which can be replaced with dynamic data(coming from database query).
Any suggestions would be appreciated.
Thanks!
This approach worked for me in Python using below libraries:
Jinja2 - for generating HTML with custom data
xhtml2pdf - for generating PDF from HTML
Consider within your PROJECT DIR, there is a template file(invoice.html) and python file(pdf_generator.py)
pdf_generator.py
from xhtml2pdf import pisa
import jinja2
templateLoader = jinja2.FileSystemLoader(searchpath="./")
templateEnv = jinja2.Environment(loader=templateLoader)
TEMPLATE_FILE = "invoice.html"
template = templateEnv.get_template(TEMPLATE_FILE)
# This data can come from database query
body = {
"data":{
"order_id": 123,
"order_creation_date": "2020-01-01 14:14:52",
"company_name": "Test Company",
"city": "Mumbai",
"state": "MH",
}
}
# This renders template with dynamic data
sourceHtml = template.render(json_data=body["data"])
outputFilename = "invoice.pdf"
# Utility function
def convertHtmlToPdf(sourceHtml, outputFilename):
# open output file for writing (truncated binary)
resultFile = open(outputFilename, "w+b")
# convert HTML to PDF
pisaStatus = pisa.CreatePDF(
src=sourceHtml, # the HTML to convert
dest=resultFile) # file handle to receive result
# close output file
resultFile.close()
# return True on success and False on errors
print(pisaStatus.err, type(pisaStatus.err))
return pisaStatus.err
if __name__ == "__main__":
pisa.showLogging()
convertHtmlToPdf(sourceHtml, outputFilename)
invoice.html
<!DOCTYPE html>
<html lang="en">
<body>
Name: {{ json_data.company_name }} <br>
City/State: {{ json_data.city }}, {{ json_data.state }} <br>
Date: {{ json_data.order_creation_date }} <br>
Order ID: {{ json_data.order_id }} <br>
</body>
</html>
https://www.npmjs.com/package/pdfmake
The above library allows flexibility when it comes to dynamic invoices in node.js
With https://getpdfapi.com you can design PDF templates using a web-based editor and once you finish, a REST API endpoint will be created just for you.
This endpoint can be used to send your data in JSON format to the PDF template you've just created.
The only code you need to write is the code that integrates the API.
You can check a quick demo here: https://www.youtube.com/watch?v=cv4ZYd_aJO8
I'm running a script that shows up some data in python bottle, one of it is signal strength that I'd like to show in real time. When I refresh the page the value doesn't change so I have to rerun the server in order to update that signal strength. I've tried ajax but it seems that it doesn't work.
What should I use to make this to work?
EDIT: The variables are lists that come from another file.
iplist = [192.168.1.1, 192.168.1.2]
hostlist = [android234567, android677896]
maclist = [a1:b2:c3:d4:e5:f6, a1:b2:c3:d4:e5:f6]
signallist = [-56, 23]
.
#app.route('/')
def index():
info={'iplist': iplist, 'maclist': maclist, 'signallist': signallist, 'hostlist': hostlist}
tpl = '''
<!DOCTYPE html>
<html>
<head>
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
<script type="text/javascript">
function ajax(){
var req = new XMLHttpRequest();
req.onreadystatechange = function(){
if (req.readyState == 4 && req.status == 200) {
document.getElementById('signallist').innerHTML = req.responseText;
}
}
req.open('REQUEST', '../connectedDevices.py', true);
req.send();
}
(function(){ajax();}, 1000);
</script>
</head>
<body onload="ajax();">
<table>
<tr>
<td>IP address</td>
<td>Hostname</td>
<td>MAC address</td>
<td>Signal</td>
</tr>
%for i in range(len(maclist)):
<tr>
<td>{{iplist[i]}}</td>
<td>{{hostlist[i]}}</td>
<td>{{maclist[i]}}</td>
<td id="signallist">{{signallist[i]}}</td>
</tr>
%end
</table>
</body>
</html>
'''
return template(tpl, info)
This prints a chart where it shows Ip, host, mac and signal which the one that I want to get updated in real time.
Bottle caches the templates. So if you are feeding values in during the template creation, it will cache those values.
from bottle import TEMPLATES
TEMPLATES.clear()
Will reset those.
However, you really should think about where your data is coming and leverage websockets. gevent has a great websocket library and works great with bottle to make your code async. With a little work and javascript you can query your api and feed data real time into your code.
I have exactly the same problem as this post
Python submitting webform using requests
but your answers do not solve it. When I execute this HTML file called api.htm in the browser, then for a second or so I see its page.
Then the browser shows the data I want with the URL https://api.someserver.com/api/ as as per the action below. But I want the data written to a file so I try the Python 2.7 script below.
But all I get is the source code of api.htm Please put me on the right track!
<html>
<body>
<form id="ID" method="post" action="https://api.someserver.com/api/ ">
<input type="hidden" name="key" value="passkey">
<input type="text" name="start" value ="2015-05-01">
<input type="text" name="end" value ="2015-05-31">
<input type="submit" value ="Submit">
</form>
<script type="text/javascript">
document.getElementById("ID").submit();
</script>
</body>
</html>
The code:
import urllib
import requests
def main():
try:
values = {'start' : '2015-05-01',
'end' : '2015-05-31'}
req=requests.post("http://my-api-page.com/api.htm",
data=urllib.urlencode(values))
filename = "datafile.csv"
output = open(filename,'wb')
output.write(req.text)
output.close()
return
main()
I can see several problems:
Your post target URL is incorrect. The form action attribute tells you where to post to:
<form id="ID" method="post" action="https://api.someserver.com/api/ ">
You are not including all the fields; type=hidden fields need to be posted too, but you are ignoring this one:
<input type="hidden" name="key" value="passkey">
Do not URL-encode your POST variables yourself; leave this to requests to do for you. By encoding yourself requests won't recognise that you are using an application/x-www-form-urlencoded content type as the body. Just pass in the dictionary as the data parameters and it'll be encoded for you and the header will be set.
You can also stream the response straight to a file object; this is helpful when the response is large. Switch on response streaming, make sure the underlying raw urllib3 file-like object decodes from transfer encoding and use shutil.copyfileobj to write to disk:
import requests
import shutil
def main():
values = {
'start': '2015-05-01',
'end': '2015-05-31',
'key': 'passkey',
}
req = requests.post("http://my-api-page.com/api.htm",
data=values, stream=True)
if req.status_code == 200:
with open("datafile.csv", 'wb') as output:
req.raw.decode_content = True
shutil.copyfileobj(req.raw, output)
There may still be issues with that key value however; perhaps the server sets a new value for each session, coupled with a cookie, for example. In that case you'd have to use a Session() object to preserve cookies, first do a GET request to the api.htm page, parse out the key hidden field value and only then post. If that is the case then using a tool like robobrowser might just be easier.
I'm trying to save a chart by converting SVG to PNG with a Python script.
So I start storing the svg data in a variable with :
var svgdata = Ext.draw.engine.SvgExporter.generate(chart.surface);
When I do alert(svgdata), I can see that this output is correct.
But when I send it to the server like this :
Ext.draw.engine.ImageExporter.defaultUrl = "data/svg_to_png.py?svgdata="+svgdata;
The svgdata that has been sent looks like this :
<?xml version=
I'm new to extjs, please help me on this one. What is the right way to send svg data to my python script and render a png image ?
This is my python script :
import cairo
import cgi
import rsvg
print "Content-type: image/png\n\n"
arguments = cgi.FieldStorage()
img = cairo.ImageSurface(cairo.FORMAT_ARGB32, 640,480)
ctx = cairo.Context(img)
handler= rsvg.Handle(None, str(arguments["svgdata"]))
handler.render_cairo(ctx)
img.write_to_png("svg.png")
HELP ME PLEASE!
<div style="display:none;">
<iframe id="file_download_iframe" src="blank.html"></iframe>
</div>
You will need a blank html page on your server for this to work properly in all browsers. Basically the blank.html page is an empty page to satisfy that the ifram always has a page in it.
Then you need a basic form hidden somewhere too:
<div style="display:none;">
<form
id = "file_download_iframe_form"
name = "file_download_iframe_form"
target = "file_download_iframe"
method = "post"
action = "data/svg_to_png.py"
>
<input type="hidden" id="svgdata" name="svgdata"/>
</form>
</div>
Then have a javascript function like this:
function getImage(svgdata){
var form = document.getElementById("file_download_iframe_form");
document.getElementById("svgdata").value = svgdata;
form.submit();
};