Develop Reference

How to save a file to a custom repository?

I am using python-docx to save a word file and I have set a specific path but since path aren't the same for every user I would like to have a dialogue when the user clicks on download so he can choose exactly in which local repository to save the file.
What I have now:
#app.route('/download_file')
def save_doc():
filename = 'test.docx'
filepath = os.path.join(r'C:\Users\Joe\Desktop', filename)
document.save(filepath)
return 'meh'

Implementing the logic you described requires work on the front-end. Let's simplify the problem by assuming that the user manually types in the download target directory. (In practice, it would make more sense for there to be a pop-up window allowing the user to browse the directory on a file explorer.)
<form action="/download" method="post">
<input type="text" value="" name="dir">
<input type="submit" value="Download">
</form>
Then, in Flask, you might specify the following:
from flask import Flask, request
#app.route('/download', methods=['GET', 'POST'])
def save_doc():
if request.method=='POST':
download_dir = request.form['dir'] # Access user input from backend
filename = 'test.docx'
filepath = os.path.join(download_dir, filename)
document.save(filepath)
return 'Download complete'
else:
pass # Whatever implementation you want for 'GET' method
This is incomplete code that I wrote without knowledge of the structure of your project, and so may not work if it is simply copied and pasted into your source code. The implementation is also quite basic, so consider it a baseline model to help you get started on baking the user interactive dialogue system.
I'd be happy to answer any questions you might have.

Image Thumbnail in Web2py: Unable to display the thumbnail

I have used the suggestion here http://www.web2pyslices.com/slice/show/1387/upload-image-and-make-a-thumbnail
to make a thumbnail of an image.
I have got the thumbnail but I am unable to display it.
The following are my functions:
db.py :
db.define_table('uploads', Field('dataset', 'reference dataset'),
Field('filename', represent = lambda x, row: "None" if x == None else [:45]),
Field('image', 'upload', uploadseparate=True, requires=IS_NOT_EMPTY() and IS_IMAGE(extensions=('jpeg', 'png','jpg','tif')) ),
Field('thumb', 'upload', uploadseparate=True, requires=IS_NOT_EMPTY() and IS_IMAGE(extensions=('jpeg', 'png', 'jpg', 'tif'))))
default.py :
def makeThumbnail(dbtable,ImageID,size=(150,150)):
try:
thisImage=db(dbtable.id==ImageID).select()[0]
import os, uuid
from PIL import Image
except: return
im=Image.open(request.folder + 'uploads/' + thisImage.image)
im.thumbnail(size,Image.ANTIALIAS)
thumbName='uploads.thumb.%s.jpg' % (uuid.uuid4())
im.save(request.folder + 'uploads/' + thumbName,'jpeg')
thisImage.update_record(thumb=thumbName)
return
def insertImage():
response.menu = [
(T('Home'),False,URL('default','experimenter')),
(T('Manage Data Set'),False,URL('default','MDS')),
(T('Manage Experiment'),False,URL('default','ME')),
(T('Manage Workflow Element'),False,URL('default','MWE'))]
dbtable = db.uploads
record = None
record = db(db.dataset.id == request.args[0],ignore_common_filters=True).select().first()
form = FORM(dbtable, INPUT(_name='up_files', _type='file',
_multiple=True, requires=IS_NOT_EMPTY()),INPUT(_type='submit'))
# The multiple param lets us choose multiple files.
if form.process().accepted:
#onvalidation checks the uploaded files to make sure they are only txt, config, or log.
makeThumbnail(dbtable,form.vars.id,(300,300))
response.flash = 'files uploaded'
files = request.vars['up_files']
if not isinstance(files, list):
#convert files to a list if they are not one already.
files = [files]
for file in files:
db.uploads.insert(dataset=record.id, filename=file.filename, image=db.uploads.image.store(file, file.filename))
#store is a FIELD method that let's you save a file to disk. you can choose the directory if you want using the 'path' param.
else:
response.flash = 'Choose the Files you would like to upload'
return dict(form=form, record=record)
And then the view:
{{extend 'layout.html'}}
<h4>Manage Image of dataset: {{=record.name}}</h4>
{{if images:}}
<div style="overflow: auto;" width="80%">
<table>
<tr> <th> Image </th> </tr>
{{
for image in images:
=TR(TD(image.filename), IMG(_src=URL('default', 'download', args=image.thumb)), A(str(T('View')),_href=URL("show", args=[image.id,rowId])), A(str(T('Delete')),_href=URL('deleteImage',args=image.id)))}}
{{pass}}
</table>
</div>
{{pass}}
Note: I am trying to display the the thumbnails for a each image in a list of images.(see the View).
I am not getting the thumbnail but rather small question marks in its place.
PS: i am unable to upload the image.
I want images in place of question mark. I am doing something wrong in insertImage() function and also in the view.
Thanks in Advance for the help!

First, you appear to be conflating FORM and SQLFORM. The former is for creating custom forms (not connected with any database tables), and the latter is for building a form based on a database table (and therefore automatically handling inserts). You cannot pass a DAL Table object to FORM as in your code -- that will simply serialize the Table object to its string name, which will be included in the HTML form DOM to no effect. Further, in this case, form.vars.id will simply be None (FORM does not generate record IDs, as it does not do any database inserts).
Also, rather than directly saving the file in makeThumbnail, a better option would be to save the image to a StringIO object and then pass that object to db.uploads.thumbnail.store() (as you do for storing the original image). In that case, the .store() method of the thumbnail field will handle the file naming and saving automatically.
from cStringIO import StringIO
tmp = StringIO()
im.save(tmp, 'jpeg')
tmp.seek(0)
thisImage.update_record(thumb=db.uploads.thumb.store(tmp, filename='thumbnail.jpg'))
For more details, see http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer.

File responses in Python Pyramid 'cancelling' each other out?

Here are three Pyramid view_callables which setup a page with data and two image files. Trouble is, only one of the images (file responses) is returned. It seems I can only return one of the images at a time. If I take one of the file response vc's away, the other images is returned. However, if I have both file respone vc's there, only the second image is returned. Is there some object in the first vc I'm overwriting with the second vc?
Is there a better way to return both images (files), even within the first # title vc? As it is now, even if it worked, I have to retrieve the same document from the database 3 times for the one template. Any advice or clues would be greatly appreciated.
# title
#view_config(
route_name='title',
renderer='templates/titles/title.jinja2')
def title(request):
title = Title().find_one({'_id':ObjectId(request.matchdict['_id'])})
result = dict(
user = request.user,
title = title)
return result
# view title image
#view_config(route_name="view_title_image")
def jpg(request):
fd = Title().find_one({'_id':ObjectId(request.matchdict['title_id'])}).TitleImage
response = Response(content_type='application/jpg')
response.app_iter = fd.File
print fd
return response
# view trailer thumbnail
#view_config(route_name="view_trailer_thumbnail")
def jpg(request):
fd = Title().find_one({'_id':ObjectId(request.matchdict['title_id'])}).TrailerThumbnail
response = Response(content_type='application/jpg')
response.app_iter = fd.File
print fd
return response
Here are the route configs from __init__:
# title
config.add_route('title', '/title/{_id}')
# view title image
config.add_route('view_title_image', '/view/title_image/{title_id}')
# view title image
config.add_route('view_trailer_thumbnail', '/view/trailer_thumbnail/{title_id}')
This is how its used in the Jinja2 template:
<img src="/view/title_image/{{ title._id }}">
<img src="/view/trailer_thumbnail/{{ title._id }}">

I think your problem is that both views have the function named jpg.
Although it's not a great idea to overwrite functions like that, I would have thought that this would be no problem at all for the view_config decorator. The only thing I can think of is that rather recording a reference to the function, view_config works out what the dotted path would be and records that.
Anyway, give the view functions different names and you should be fine.

django static files versioning

I'm working on some universal solution for problem with static files and updates in it.
Example: let's say there was site with /static/styles.css file - and site was used for a long time - so a lot of visitors cached this file in browser
Now we doing changes in this css file, and update on server, but some users still have old version (despite modification date returned by server)
The obvious solution is to add some version to file /static/styles.css?v=1.1 but in this case developer must track changes in this file and manually increase version
A second solution is to count the md5 hash of the file and add it to the url /static/styels.css/?v={mdp5hashvalue} which looks much better, but md5 should be calculated automatically somehow.
they possible way I see it - create some template tag like this
{% static_file "style.css" %}
which will render
<link src="/static/style.css?v=md5hash">
BUT, I do not want this tag to calculate md5 on every page load, and I do not want to store hash in django-cache, because then we will have to clear after updating file...
any thoughts ?

Django 1.4 now includes CachedStaticFilesStorage which does exactly what you need (well... almost).
Since Django 2.2 ManifestStaticFilesStorage should be used instead of CachedStaticFilesStorage.
You use it with the manage.py collectstatic task. All static files are collected from your applications, as usual, but this storage manager also creates a copy of each file with the MD5 hash appended to the name. So for example, say you have a css/styles.css file, it will also create something like css/styles.55e7cbb9ba48.css.
Of course, as you mentioned, the problem is that you don't want your views and templates calculating the MD5 hash all the time to find out the appropriate URLs to generate. The solution is caching. Ok, you asked for a solution without caching, I'm sorry, that's why I said almost. But there's no reason to reject caching, really. CachedStaticFilesStorage uses a specific cache named staticfiles. By default, it will use your existing cache system, and voilà! But if you don't want it to use your regular cache, perhaps because it's a distributed memcache and you want to avoid the overhead of network queries just to get static file names, then you can setup a specific RAM cache just for staticfiles. It's easier than it sounds: check out this excellent blog post. Here's what it would look like:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
'LOCATION': '127.0.0.1:11211',
},
'staticfiles': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'staticfiles-filehashes'
}
}

I would suggest using something like django-compressor. In addition to automatically handling this type of stuff for you, it will also automatically combine and minify your files for fast page load.
Even if you don't end up using it in entirety, you can inspect their code for guidance in setting up something similar. It's been better vetted than anything you'll ever get from a simple StackOverflow answer.

I use my own templatetag which add file modification date to url: https://bitbucket.org/ad3w/django-sstatic

Is reinventing the wheel and creating own implementation that bad? Furthermore I would like low level code (nginx for example) to serve my staticfiles in production instead of python application, even with backend. And one more thing: I'd like links stay the same after recalculation, so browser fetches only new files. So here's mine point of view:
template.html:
{% load md5url %}
<script src="{% md5url "example.js" %}"/>
out html:
static/example.js?v=5e52bfd3
settings.py:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(PROJECT_DIR, 'static')
appname/templatetags/md5url.py:
import hashlib
import threading
from os import path
from django import template
from django.conf import settings
register = template.Library()
class UrlCache(object):
_md5_sum = {}
_lock = threading.Lock()
#classmethod
def get_md5(cls, file):
try:
return cls._md5_sum[file]
except KeyError:
with cls._lock:
try:
md5 = cls.calc_md5(path.join(settings.STATIC_ROOT, file))[:8]
value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
except IsADirectoryError:
value = settings.STATIC_URL + file
cls._md5_sum[file] = value
return value
#classmethod
def calc_md5(cls, file_path):
with open(file_path, 'rb') as fh:
m = hashlib.md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
#register.simple_tag
def md5url(model_object):
return UrlCache.get_md5(model_object)
Note, to apply changes an uwsgi application (to be specific a process) should be restarted.

Django 1.7 added ManifestStaticFilesStorage, a better alternative to CachedStaticFilesStorage that doesn't use the cache system and solves the problem of the hash being computed at runtime.
Here is an excerpt from the documentation:
CachedStaticFilesStorage isn’t recommended – in almost all cases ManifestStaticFilesStorage is a better choice. There are several performance penalties when using CachedStaticFilesStorage since a cache miss requires hashing files at runtime. Remote file storage require several round-trips to hash a file on a cache miss, as several file accesses are required to ensure that the file hash is correct in the case of nested file paths.
To use it, simply add the following line to settings.py:
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
And then, run python manage.py collectstatic; it will append the MD5 to the name of each static file.

How about you always have a URL Parameter in your URL with a version and whenever you have a major release you change the version in your URL Parameter. Even in the DNS. So if www.yourwebsite.com loads up www.yourwebsite.com/index.html?version=1.0 then after the major release the browser should load www.yourwebsite.com/index.html?version=2.0
I guess this is similar to your solution 1. Instead of tracking files can you track whole directories? For example ratehr than /static/style/css?v=2.0 can you do /static-2/style/css or to make it even granular /static/style/cssv2/.

There is an update for #deathangel908 code. Now it works well with S3 storage also (and with any other storage I think). The difference is using of static file storage for getting file content. Original doesn't work on S3.
appname/templatetags/md5url.py:
import hashlib
import threading
from django import template
from django.conf import settings
from django.contrib.staticfiles.storage import staticfiles_storage
register = template.Library()
class UrlCache(object):
_md5_sum = {}
_lock = threading.Lock()
#classmethod
def get_md5(cls, file):
try:
return cls._md5_sum[file]
except KeyError:
with cls._lock:
try:
md5 = cls.calc_md5(file)[:8]
value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
except OSError:
value = settings.STATIC_URL + file
cls._md5_sum[file] = value
return value
#classmethod
def calc_md5(cls, file_path):
with staticfiles_storage.open(file_path, 'rb') as fh:
m = hashlib.md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
#register.simple_tag
def md5url(model_object):
return UrlCache.get_md5(model_object)

The major advantage of this solution: you dont have to modify anything in the templates.
This will add the build version into the STATIC_URL, and then the webserver will remove it with a Rewrite rule.
settings.py
# build version, it's increased with each build
VERSION_STAMP = __versionstr__.replace(".", "")
# rewrite static url to contain the number
STATIC_URL = '%sversion%s/' % (STATIC_URL, VERSION_STAMP)
So the final url would be for example this:
/static/version010/style.css
And then Nginx has a rule to rewrite it back to /static/style.css
location /static {
alias /var/www/website/static/;
rewrite ^(.*)/version([\.0-9]+)/(.*)$ $1/$3;
}

Simple templatetag vstatic that creates versioned static files urls that extends Django's behaviour:
from django.conf import settings
from django.contrib.staticfiles.templatetags.staticfiles import static
#register.simple_tag
def vstatic(path):
url = static(path)
static_version = getattr(settings, 'STATIC_VERSION', '')
if static_version:
url += '?v=' + static_version
return url
If you want to automatically set STATIC_VERSION to the current git commit hash, you can use the following snippet (Python3 code adjust if necessary):
import subprocess
def get_current_commit_hash():
try:
return subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).strip().decode('utf-8')
except:
return ''
At settings.py call get_current_commit_hash(), so this will be calculated only once:
STATIC_VERSION = get_current_commit_hash()

I use a global base context in all my views, where I set the static version to be the millisecond time (that way, it will be a new version every time I restart my application):
# global base context
base_context = {
"title": settings.SITE_TITLE,
"static_version": int(round(time.time() * 1000)),
}
# function to merge context with base context
def context(items: Dict) -> Dict:
return {**base_context, **items}
# view
def view(request):
cxt = context({<...>})
return render(request, "page.html", cxt)
my page.html extends my base.html template, where I use it like this:
<link rel="stylesheet" type="text/css" href="{% static 'style.css' %}?v={{ static_version }}">
fairly simple and does the job

Troubleshooting dynamically generated KML

I use dynamically generated KML that fails intermittently seemingly because of "timing issues" with Google's server's. Last time I checked it worked but that seems like a coincident. Should I make a cron job that accesses the page every 5 minutes to make data stay fresh in memory? The serverside KML generator is:
class KMLHandler(webapp2.RequestHandler):
def get(self):
self.response.headers['Cache-Control'] = 'public,max-age=%s' \
% 86400
start = datetime.datetime.now() - timedelta(days=60)
from google.appengine.api import memcache
memcache_key = 'ads'
data = memcache.get(memcache_key)
if data is None:
a = Ad.all().filter('modified >',
start).filter('published =',
True).order('-modified').fetch(1000)
memcache.set('ads', a)
else:
a = data
dispatch = 'templates/kml.html'
template_values = {'a': a, 'request': self.request,
'host': os.environ.get('HTTP_HOST',
os.environ['SERVER_NAME'])}
path = os.path.join(os.path.dirname(__file__), dispatch)
output = template.render(path, template_values)
self.response.headers['Content-Type'] = \
'application/vnd.google-earth.kml+xml'
self.response.headers['Content-Length'] = len(output)
self.response.out.write(output)
--
The template file is:
<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom">
<Document>
{% for ad in a %}
{% if ad.geopt %}
<Placemark><name></name><description>
<![CDATA[{% if ad.uri2view %}<img src="{{ ad.uri2view }}">{% endif %} {{ ad.title }} <br/>{{ad.text|truncatewords:20}}]]></description><Point><coordinates>{{ad.geopt.lon|floatformat:2}},{{ad.geopt.lat|floatformat:2}}</coordinates></Point></Placemark>{% endif %}{% endfor %}</Document></kml>
--
I've also run the output through a KML validator to ensure that it is correct. It is dynamically generated and I try to cache it. It just fails intermittently for no apperant reason. I have Python that generates the KML and the code is checked and should be working. I don't see where the program is wrong? I can take the output of my script, save it as a static kml file, and that works and the KML is valid so it seems there are "timing issues" in serving it to google when it is being dynamically generated. But how can I troubleshoot this further? I could change the whole strategy to using JSON instead but that will be a lot of recoding and I think I'm close to making it work.
The 3 strategies I can think of is
Change to Jinja2 templates
Change to JSON instead of KML
Write a cron job that accesses the file every 5 minutes to keep the data in memcache
Thanks for any help

I suspect it's taking too long to build your KML so Google is giving up. I tested the link a few times and each time it took several seconds to load, sometimes >5 seconds. The servers that download and render these KML files will time out / give up if the file can't be loaded quickly. This sort of behavior is consistent with intermittent failures when the file can't be loaded sometimes or the cache (when a good load has occured) expires.
If the process that generates the KML can't be improved, you should cache the actual file data you generate so you can return it quicker in the future. Using the Task Queue API / Cron you could keep that cache up to date by running a rebuild task every N minutes.