Develop Reference

Django decorator #transaction.non_atomic_requests not working in a ViewSet method

I recently ran into the need to disable transaction requests in one of my views, in order to be able to call db.connection.close() and connect() during requests in an effort to improve performance.
I have a DRF ViewSet, and used the following very simple view to verify that the non_atomic_requests decoractor seems to have no effect. ATOMIC_REQUESTS=True is enabled in settings.py, and DEBUG=False.
from django.db import transaction
#transaction.non_atomic_requests
def create(self, *args, **kwargs):
m = MyModel(stuff="hello")
m.save()
raise Exception('exception! row should still be saved though')
return Response()
After calling the view, I open Django shell, and verify that the amount of rows in the db has not grown, even though it should have. Also opening a debugger during the request to halt execution after the line m.save(), I can observe in Django shell that a new row is not visible yet.
If I set ATOMIC_REQUESTS=False in settings.py, the code works as expected, and the number of rows in the db is grown by one, even if an error is raised before returning from the view.
When ATOMIC_REQUESTS=False, using #transaction.atomic decorator does work as expected though. So as a workaround, I could use it to set every other view as atomic instead...
I am currently thinking this is a bug in the framework. Can anybody verify my findings, or point out if I am misunderstanding how this decorator is meant to function?
I am using Python 3.6, Django 2.0 and DRF 3.7.7.

As documented, non_atomic_requests only works if it's applied to the view itself.
In your case, create is a viewset method, it is not the view itself. With a regular class based view in Django, you'd need to wrap the dispatch method using method_decorator.
#method_decorator(transaction.non_atomic_requests, name='dispatch')
class MyViewSet(ViewSet):
...
def create(self, *args, **kwargs):
...
I'm not familiar enough with the rest framework internals to say whether this will work or not. Note that it will disable atomic requests for all views handled by the viewset, not just the create method.
The non_atomic_requests method has this limitation because the Django request handler has to inspect the view before it runs so that it knows whether to run it in a transaction. The transaction.atomic decorator does not have the same requirement - Django can simply start the transaction as soon as it enters an atomic function or block.

In case you are using db other than 'default':
you need to explicitly mention the 'using' property.
otherwise, it would default to 'default'
transaction.non_atomic_requests(using='db_name')
in case of class based views-
either apply it on dispatch in views:
#method_decorator(transaction.non_atomic_requests(using='db_name'), name='dispatch')
class MyViewSet(ViewSet):
...
or apply it on as_view method in urls
path(r'endpoint/', transaction.non_atomic_requests(using='db_name')(MyViewSet.as_view()), name='myview')

How should I create a Flask extension which depends on another extension?

I want to create a Flask extension which depends on another Flask extension. For the sake of argument, say that it's Flask-Foo, and that it needs Flask-Redis to store some specific data in a Redis database.
I know that I can add an install dependency to Flask-Redis. However I don't understand how I should instantiate and initialize Flask-Redis.
The setup for Flask-Foo sets up the Flask-Redis object. The drawback of this is that it assumes that the app isn't also using Flask-Redis for some other reason, configured explicitly outside of Flask-Foo. If it is, we get two objects which exist side-by-side, which seems wrong.
The user has to themselves instantiate and configure Flask-Redis. Flask-Foo checks that it has been initialized for that app, and complains otherwise. The problem with this is that it seems to impose boilerplate on the user - why should they have to set up Flask-Redis to use Flask-Foo, when they have no other knowledge or interest in the configuration of Flask-Redis? Furthermore, aren't we asking for trouble if this means that Flask-Foo.init_app() always has to be called after Flask-Redis.init_app()?
Don't use Flask-Redis. Use the Redis package directly, and manage the connection in Flask-Foo code. This would probably avoid the above problems. But it seems unelegant - we will basically have to resolve problems solved by Flask-Redis. If Flask-Foo goes on to support an alternative database, it will become complicated as we have to maintain code to manage the different types of connection.
Just to be clear, this is not a question specifically about Flask-Redis or how it works! I just want to understand what is generally the right way to build an extension on top of an extension.

You can pass depend extension to init_app. http://flask.pocoo.org/docs/1.0/extensiondev/
flask_foo/init.py
class FooManager:
def __init__(self, app=None, db=None, **kwargs):
self.app = app
if app is not None:
self.init_app(app, db, **kwargs)
def init_app(self, app, db, **kwargs):
self.db = db
app.config.setdefault('xxx', xxx)
# Bind Flask-Foo to app
app.foo_manager = self
Now, you can get foo_manager object from current_app like this:
models.py
from flask import current_app
db = current_app.foo_manager.db
class XXX(db.Model):
pass
Last, maybe you must register foo by app_context():
run.py
with app.app_context():
FooManager(app, db) # or xx = FooManager(); xx.init_app(app, db)
wonderful, depend extension works good for us.
Other tip: https://stackoverflow.com/a/51739367/5204664

Flask extension has the same structure as a python module. You should specify all requirements in setup.py file.
For example flask-babel
install_requires=[
'Flask',
'Babel>=2.3',
'Jinja2>=2.5'
],

Tools to help developers reading class hierarchy faster

I mostly spend time on Python/Django and Objective-C/CocoaTouch and js/jQuery in the course of my daily work.
My editor of choice is vim for Python/Django and js/jQuery and xcode for Objective-C/CocoaTouch.
One of the bottlenecks on my development speed is the pace at which I read existing code, particularly open source libraries which I use.
In Python/Django for example, when I encounter some new features introduced by django developers, I get curious and begin exploring the code base manually. For example, when class-based views were introduced from django 1.3 onwards, reference - https://docs.djangoproject.com/en/dev/topics/class-based-views/ - I will check out the example code shown:
from django.views.generic import TemplateView
class AboutView(TemplateView):
template_name = "about.html"
And try it out on one of my projects. More importantly, I am curious about what goes on behind the scenes, so I will dig into the source code -
# django/views/generic/__init__.py file
from django.views.generic.base import View, TemplateView, RedirectView
from django.views.generic.dates import (ArchiveIndexView, YearArchiveView, MonthArchiveView,
WeekArchiveView, DayArchiveView, TodayArchiveView,
DateDetailView)
from django.views.generic.detail import DetailView
from django.views.generic.edit import FormView, CreateView, UpdateView, DeleteView
from django.views.generic.list import ListView
class GenericViewError(Exception):
"""A problem in a generic view."""
pass
From here, I will trace it backwards to the django/views/generic/base.py file and find out exactly what TemplateView class does:-
class TemplateView(TemplateResponseMixin, View):
"""
A view that renders a template.
"""
def get_context_data(self, **kwargs):
return {
'params': kwargs
}
def get(self, request, *args, **kwargs):
context = self.get_context_data(**kwargs)
return self.render_to_response(context)
And here's it shows that TemplateView class inherits from TemplateResponseMixin and View classes... and I continue digging further... and so on...
The problem is, this is an extremely inefficient and slow process (to "follow" class hierachies manually and opening up each file along the way).
So the question is - is there an easy way/UI tool (or other visual solution) that parses Python code in a particular project and visualize class hierarchies which I can then inspect easily by "clicking" on a specific class I am interested to read about?
Note that I am aware of IPython shell but that doesn't seem as user-friendly as a visual display tool.
For example, there's F-Script in the world of Objective-C/iOS/Mac programming, which not only provides a shell (much like python or IPython shell), but provides a visual way for developers to introspect class hierachies.
Reference screenshot:-
So is there a class-hierarchy visualization tool (for Python specifically, but even better if it's generic and can be used for different languages)??? What are your methods of getting up to speed efficiently when reading open source source code???
UPDATED
Per advice below, I tried out ctags and vim plugin taglist and I was able to use :TlistOpen to open up a side buffer in vim like this:-
This looks really cool as :TlistOpen now essentially shows me all the classes and functions that are available on my currently open buffer.
My problem now is that when I attempt to do Ctrl] while my cursor is on TemplateView, I get the following error:-
What am I doing wrong? Is it because my django source code is in a virtualenv? Or is there something specific I have to do to make ctags/taglist "aware" of the django source code?

Tags are a very good start indeed. (There's too much stuff all over the place on it, so I'll just provide you with one extra keyword to search with: ctags.)
In Vim, it ends up (in the basic case) with Ctrl+] to go to a class/function definition and Ctrl+T to return.

The IDLE editor included with Python has an effective class browser that efficiently navigates everything in a given module. I believe in would not be difficult to modify that tool to navigate a full class-hierarchy with some assistance from the inspect module and the pyclbr module.

I've been using exuberant ctags with taglist for vim. Use ctrl] to jump to class definition in the current window, ctrlw] to jump to the definition in a split window.
You can install exuberant ctags via homebrew:
brew install ctags
Be sure to use the one installed at /usr/local/bin by homebrew, not the old ctags in /usr/bin.
It is also helpful to put --python-kinds=-vi in ~/.ctags to skip indexing variables and imports for Python files.
Another alternative would be to use a variant of cscope or pyscope though you must have your vim compiled with cscope option enabled.

Preprocess SHPAML in Django's template loader?

Is there any way to make Django's template loader run all templates it loads (i.e. directly or via extend/include) through SHPAML if it figures the HTML is out of date?
I know how to invoke SHPAML recursively over an entire directory, but I would prefer to be able to run it on demand so I don't have to remember to sync the HTML every time I change the SHPAML source.
I guess invoking SHPAML from manage.py would work too (at least for test servers), but being able to hack into Django's template engine and make it run every file it loads through a preprocessor would be nicer.

I suspect you can achieve what you want by inheriting from django.template.loaders.app_directories.Loader (or whatever loader you use) and overwriting the load_template_source method, e.g.:
from django.template.loaders.app_directories import Loader
from shpaml import convert_text
class SHPAMLLoader(Loader):
def load_template_source(self, *args, **kwargs):
shpaml_source = super(SHPAMLLoader, self).load_template_source(*args, **kwargs)
html = convert_text(shpaml_source)
return html
Then put your loader at the beginning of the TEMPLATE_LOADERS tuple in your settings.py. Of course, you will be doing the SHPAML to HTML dance every time a template is loaded, so you may see some overhead. The upcoming Django 1.2 features template caching, which could help mitigating that overhead...
Disclaimer: this code is completely untested, sorry.

Just created a project based on the snippet in piquadrat's answer. It's a little more feature complete and supports django 1.1 and 1.2 (probably 1.0 as well)
django-shpaml-template-loader on bitbucket
Thought it might come in handy for the future :)

How to externally populate a Django model?

What is the best idea to fill up data into a Django model from an external source?
E.g. I have a model Run, and runs data in an XML file, which changes weekly.
Should I create a view and call that view URL from a curl cronjob (with the advantage that that data can be read anytime, not only when the cronjob runs), or create a python script and install that script as a cron (with DJANGO _SETTINGS _MODULE variable setup before executing the script)?

There is excellent way to do some maintenance-like jobs in project environment- write a custom manage.py command. It takes all environment configuration and other stuff allows you to concentrate on concrete task.
And of course call it directly by cron.

You don't need to create a view, you should just trigger a python script with the appropriate Django environment settings configured. Then call your models directly the way you would if you were using a view, process your data, add it to your model, then .save() the model to the database.

I've used cron to update my DB using both a script and a view. From cron's point of view it doesn't really matter which one you choose. As you've noted, though, it's hard to beat the simplicity of firing up a browser and hitting a URL if you ever want to update at a non-scheduled interval.
If you go the view route, it might be worth considering a view that accepts the XML file itself via an HTTP POST. If that makes sense for your data (you don't give much information about that XML file), it would still work from cron, but could also accept an upload from a browser -- potentially letting the person who produces the XML file update the DB by themselves. That's a big win if you're not the one making the XML file, which is usually the case in my experience.

"create a python script and install that script as a cron (with DJANGO _SETTINGS _MODULE variable setup before executing the script)?"
First, be sure to declare your Forms in a separate module (e.g. forms.py)
Then, you can write batch loaders that look like this. (We have a LOT of these.)
from myapp.forms import MyObjectLoadForm
from myapp.models import MyObject
import xml.etree.ElementTree as ET
def xmlToDict( element ):
return dict(
field1= element.findtext('tag1'),
field2= element.findtext('tag2'),
)
def loadRow( aDict ):
f= MyObjectLoadForm( aDict )
if f.is_valid():
f.save()
def parseAndLoad( someFile ):
doc= ET.parse( someFile ).getroot()
for tag in doc.getiterator( "someTag" )
loadRow( xmlToDict(tag) )
Note that there is very little unique processing here -- it just uses the same Form and Model as your view functions.
We put these batch scripts in with our Django application, since it depends on the application's models.py and forms.py.
The only "interesting" part is transforming your XML row into a dictionary so that it works seamlessly with Django's forms. Other than that, this command-line program uses all the same Django components as your view.
You'll probably want to add options parsing and logging to make a complete command-line app out of this. You'll also notice that much of the logic is generic -- only the xmlToDict function is truly unique. We call these "Builders" and have a class hierarchy so that our Builders are all polymorphic mappings from our source documents to Python dictionaries.