Long story short, I need to call a python script from a Celery worker using subprocess. This script interacts with a REST API. I would like to avoid hard-coding the URLs and django reverse seems like a nice way to do that.
Is there a way to use reverse outside of Django while avoiding the following error?
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
I would prefer something with low-overhead.
As documented
If you’re using components of Django “standalone” – for example,
writing a Python script which loads some Django templates and renders
them, or uses the ORM to fetch some data – there’s one more step
you’ll need in addition to configuring settings.
After you’ve either set DJANGO_SETTINGS_MODULE or called configure(),
you’ll need to call django.setup() to load your settings and populate
Django’s application registry.
I am using a custom manager command to boot my django app from external scripts. It is like smashing a screw with a hammer, but the setup is fast and it takes care for pretty much everything.
Related
I have an existing Django app. Part of it is specific to a Django web app of ours, but another part does a lot of complicated computation, and it may be useful to more of our products. I am busy splitting off this more generic part into a library that does not depend on Django.
The Django app has a few settings -- for instance, it needs to know the location of certain data files. In Django I find that location as the MYAPP_DATA_DIR setting in settings.py, and I use django-appconf to set defaults for such settings.
What's the most common way to handle settings for generic Python libraries? Should I have a configure() function that can be called to set up the library, look in environment variables, something else?
Update: I've noticed that entities are saved (and available at the Datastore Viewer) when I save them using views (and the create_object function). But when I use shell (manage.py shell) to create and save new entity it isn't commited to the storage (but still can be seen in Tes.objects.all()).
I started playing with the django-nonrel with google appengine and I am getting frustrated by thing as simple as saving Entities.
I've set up my environment as described in instruction. I managed to run sample application and it runs ok. I'd like to extend it so it saves my entity to the storage. To do so:
I added new django module with models.py:
from django.db import models
class Tes(models.Model):
name = models.CharField(max_length=150)
I created a script to save some data:
import os
import sys
sys.path.append("d:\\workspace\\project\\")
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from testmodule.models import Tes
t = Tes(name="test")
t.save()
tes = Tes.objects.all()
for t in tes:
print t.name
The script runs without errrors. When I run it few times one after another, it prints more and more "test" strings. But when I try running it after a minute break the Tes.objects.all() returns nothing. During that time the datastore file changes it size (but maybe that's just some kind of logs). When I look at the http://localhost:8000/_ah/admin/datastore I can select only AhAdminXrsfToken from select field.
Anyway, what am I missing? Where I can find some kind of logs which would tell me what's wrong?
This is a gotcha that causes a lot of confusion. From the djangoappengine docs:
Also, never run manage.py runserver together with other management
commands at the same time. The changes won't take effect. That's an
App Engine SDK limitation which might get fixed in a later release.
So you can't do manage.py runserver and manage.py shell at the same time. If you do, changes to the datastore in one will not be visible in the other. There's a lock on the local datastore enforced by the App Engine SDK. Make sure you have stopped the server before you start the shell.
Shoudn't it be t.put() if you are creating an entity rather than saving it? I use put() to create an entity and it works for me. And if you import django you may want to know that there are alternatives to django such as my choice GAE + Jinja2 + WTForms especially now that google.db.djangoforms is deprecated selecting a form framework for forms, a templating engine and perhaps a db framework and you do't have to import django which often results in forcing you to import much more than you need.
So my recommendation is to avoid import django... and instead use Jinja2 + WTForms. If you really want django on app engine then you might want to check in the project www.allbuttonspressed.com that enables all django for google app engine but be certain that you need this much django when I suspect all we need is a template engine and a form framework and we can do without django.
I am a seasoned .Net developer who's trying to write some Python code. On one of the projects I am contributing to, we have a services layer which is a set of classes which abstract away functionality and a django web app which consumes these in process services (which are just classes).
I had created a repository layer and ensured that all interaction with the database happens through the services layer through this repository. We have a document oriented database and thus we do not have the usual object-relational muck.
During a recent code review, one developer who is supposedly seasoned with python shunned at this and commented that this was not the python way of doing things. He remarked that python developers are used to having a save and delete method on the object instance itself (and do not use the repository pattern as much) and this would confuse python devs looking to contribute to our OSS project. Python devs, your views? Would you be confused?
Edit: This is not django code, but will be code called by the django app (It an in process service layer)
Maybe that is a Django pattern, but not a Python one by all means.
That said, if the target audience of your module are Django developers, I would advise you to follow as much as possible the Django philosophy and its associated patterns.
Django's ORM provides save() and delete() methods on the object. SQLAlchemy on the other hand has a so called session to which you add or delete objects.
Both are very popular so I'd say that both methods are about equal in terms of popularity. However in the context of a Django application going with the Django convention is probably preferable unless you have a good reason not to.
Best of my recollection Django's models include save() and delete() methods so you can deal exclusively with objects, rather than interacting with a database connection object. I don't know that it's instantly a Python way of doing things, but I'm pretty sure it's a pervasive Django pattern.
If I was told "this is Django code" but the code diverged from how Django does things, that might be confusing.
Don't Repeat Yourself. If all the data stored in the database is meant to be accessible through django (e.g., they are defined in the django models.py); there's a django-ORM that is already designed to safely (no SQL injection) and easily access the database for you via save() and delete(). There's also helpful wrapper functions to create transactions (e.g., #transaction.commit_on_success to group together actions. You can use the ORM in python scripts outside of the running django web-app. E.g., create a django management command or run a script from the django shell (./manage shell)
I definitely agree having another repository layer creates confusion and potentially leads to major issues for people using your . E.g., sometimes you have model validation that goes beyond the database validation and if you save it outside of django that validation never runs. Or maybe every time a specific model is saved, extra behavior should occur (like creating a complimentary object; or generate a task) that would be skipped if save() is not called so the pre_save post_save signals are never generated.
Granted you said this is a document-oriented-database (e.g., mongdb/couchdb), and AFAIK django does not support these sorts of NoSQL dbs, so then ignore what I said.
I want to be able to instrument Python applications so that I know:
Page generation time.
Percentage of time spent in external requests (mysql, api calls).
Number of mysql queries, what the MySQL queries were.
I want this data from production (not offline profiling) - because the time spent in various places will be different under load.
In PHP I can do this with XHProf or instrumentation-for-php. In Ruby on Rails/.NET/Java, I can do this with New Relic.
Is there such a package recommended for Python or django?
Yes, it's perfectly possible. E.g. use some magic switch in URL, like "?profile-me" which triggers profiling in Django middleware.
There are a number of snippets on the Internet, like this one: http://djangosnippets.org/snippets/70/ or modules like this one: http://code.google.com/p/django-profiling/ - but I haven't used any of them so I cannot recommend anything.
Anyway, the approach they take is similar to what I do - i.e. use Python Hotshot profiler module in a middleware that wraps your view. For the MySQL part, you can just use connection.queries form Django.
The nice thing about Hotshot is that its output can be browsed using Kcachegrind like here: http://www.rkblog.rk.edu.pl/w/p/django-profiling-hotshot-and-kcachegrind/
New Relic now had a package for Python, including Django through mod_wsgi.
https://support.newrelic.com/help/kb/python
django-prometheus is a good choice for handling production workloads, especially in a container environment like Kubernetes. Out of the box, it has middleware for tracking request latencies and counts (by view method), as well as Database and cache access times. It wouldn't be a good solution for tracking which queries are actually executing, but that's where a logging solution like ELK would come into play. If it helps, I've written a post which walks through how to add custom metrics to a Django application.
I'm building a website that doesn't require a database because a REST API "is the database". (Except you don't want to be putting site-specific things in there, since the API is used by mostly mobile clients)
However there's a few things that normally would be put in a database, for example the "jobs" page. You have master list view, and the detail views for each job, and it should be easy to add new job entries. (not necessarily via a CMS, but that would be awesome)
e.g. example.com/careers/ and example.com/careers/77/
I could just hardcode this stuff in templates, but that's no DRY- you have to update the master template and the detail template every time.
What do you guys think? Maybe a YAML file? Or any better ideas?
Thx
Why not still keep it in a database? Your remote REST store is all well and funky, but if you've got local data, there's nothing (unless there's spec saying so) to stop you storing some stuff in a local db. Doesn't have to be anything v glamorous - could be sqlite, or you could have some fun with redis, etc.
You could use the Memcachedb via the Django cache interface.
For example:
Set the cache backend as memcached in your django settings, but install/use memcachedb instead.
Django can't tell the difference between the two because the provide the same interface (at least last time I checked).
Memcachedb is persistent, safe for multithreaded django servers, and won't lose data during server restarts, but it's just a key value store. not a complete database.
Some alternatives built into the Python library are listed in the Data Persistence chapter of the documentation. Still another is JSON.