I was wondering if it was possible (and, if so, how) to chain together multiple managers to produce a query set that is affected by both of the individual managers. I'll explain the specific example that I'm working on:
I have multiple abstract model classes that I use to provide small, specific functionality to other models. Two of these models are a DeleteMixin and a GlobalMixin.
The DeleteMixin is defined as such:
class DeleteMixin(models.Model):
deleted = models.BooleanField(default=False)
objects = DeleteManager()
class Meta:
abstract = True
def delete(self):
self.deleted = True
self.save()
Basically it provides a pseudo-delete (the deleted flag) instead of actually deleting the object.
The GlobalMixin is defined as such:
class GlobalMixin(models.Model):
is_global = models.BooleanField(default=True)
objects = GlobalManager()
class Meta:
abstract = True
It allows any object to be defined as either a global object or a private object (such as a public/private blog post).
Both of these have their own managers that affect the queryset that is returned. My DeleteManager filters the queryset to only return results that have the deleted flag set to False, while the GlobalManager filters the queryset to only return results that are marked as global. Here is the declaration for both:
class DeleteManager(models.Manager):
def get_query_set(self):
return super(DeleteManager, self).get_query_set().filter(deleted=False)
class GlobalManager(models.Manager):
def globals(self):
return self.get_query_set().filter(is_global=1)
The desired functionality would be to have a model extend both of these abstract models and grant the ability to only return the results that are both non-deleted and global. I ran a test case on a model with 4 instances: one was global and non-deleted, one was global and deleted, one was non-global and non-deleted, and one was non-global and deleted. If I try to get result sets as such: SomeModel.objects.all(), I get instance 1 and 3 (the two non-deleted ones - great!). If I try SomeModel.objects.globals(), I get an error that DeleteManager doesn't have a globals (this is assuming my model declaration is as such: SomeModel(DeleteMixin, GlobalMixin). If I reverse the order, I don't get the error, but it doesn't filter out the deleted ones). If I change GlobalMixin to attach GlobalManager to globals instead of objects (so the new command would be SomeModel.globals.globals()), I get instances 1 and 2 (the two globals), while my intended result would be to only get instance 1 (the global, non-deleted one).
I wasn't sure if anyone had run into any situation similar to this and had come to a result. Either a way to make it work in my current thinking or a re-work that provides the functionality I'm after would be very much appreciated. I know this post has been a little long-winded. If any more explanation is needed, I would be glad to provide it.
Edit:
I have posted the eventual solution I used to this specific problem below. It is based on the link to Simon's custom QuerySetManager.
See this snippet on Djangosnippets: http://djangosnippets.org/snippets/734/
Instead of putting your custom methods in a manager, you subclass the queryset itself. It's very easy and works perfectly. The only issue I've had is with model inheritance, you always have to define the manager in model subclasses (just: "objects = QuerySetManager()" in the subclass), even though they will inherit the queryset. This will make more sense once you are using QuerySetManager.
Here is the specific solution to my problem using the custom QuerySetManager by Simon that Scott linked to.
from django.db import models
from django.contrib import admin
from django.db.models.query import QuerySet
from django.core.exceptions import FieldError
class MixinManager(models.Manager):
def get_query_set(self):
try:
return self.model.MixinQuerySet(self.model).filter(deleted=False)
except FieldError:
return self.model.MixinQuerySet(self.model)
class BaseMixin(models.Model):
admin = models.Manager()
objects = MixinManager()
class MixinQuerySet(QuerySet):
def globals(self):
try:
return self.filter(is_global=True)
except FieldError:
return self.all()
class Meta:
abstract = True
class DeleteMixin(BaseMixin):
deleted = models.BooleanField(default=False)
class Meta:
abstract = True
def delete(self):
self.deleted = True
self.save()
class GlobalMixin(BaseMixin):
is_global = models.BooleanField(default=True)
class Meta:
abstract = True
Any mixin in the future that wants to add extra functionality to the query set simply needs to extend BaseMixin (or have it somewhere in its heirarchy). Any time I try to filter the query set down, I wrapped it in a try-catch in case that field doesn't actually exist (ie, it doesn't extend that mixin). The global filter is invoked using globals(), while the delete filter is automatically invoked (if something is deleted, I never want it to show). Using this system allows for the following types of commands:
TemporaryModel.objects.all() # If extending DeleteMixin, no deleted instances are returned
TemporaryModel.objects.all().globals() # Filter out the private instances (non-global)
TemporaryModel.objects.filter(...) # Ditto about excluding deleteds
One thing to note is that the delete filter won't affect admin interfaces, because the default Manager is declared first (making it the default). I don't remember when they changed the admin to use Model._default_manager instead of Model.objects, but any deleted instances will still appear in the admin (in case you need to un-delete them).
I spent a while trying to come up with a way to build a nice factory to do this, but I'm running into a lot of problems with that.
The best I can suggest to you is to chain your inheritance. It's not very generic, so I'm not sure how useful it is, but all you would have to do is:
class GlobalMixin(DeleteMixin):
is_global = models.BooleanField(default=True)
objects = GlobalManager()
class Meta:
abstract = True
class GlobalManager(DeleteManager):
def globals(self):
return self.get_query_set().filter(is_global=1)
If you want something more generic, the best I can come up with is to define a base Mixin and Manager that redefines get_query_set() (I'm assuming you only want to do this once; things get pretty complicated otherwise) and then pass a list of fields you'd want added via Mixins.
It would look something like this (not tested at all):
class DeleteMixin(models.Model):
deleted = models.BooleanField(default=False)
class Meta:
abstract = True
def create_mixin(base_mixin, **kwargs):
class wrapper(base_mixin):
class Meta:
abstract = True
for k in kwargs.keys():
setattr(wrapper, k, kwargs[k])
return wrapper
class DeleteManager(models.Manager):
def get_query_set(self):
return super(DeleteManager, self).get_query_set().filter(deleted=False)
def create_manager(base_manager, **kwargs):
class wrapper(base_manager):
pass
for k in kwargs.keys():
setattr(wrapper, k, kwargs[k])
return wrapper
Ok, so this is ugly, but what does it get you? Essentially, it's the same solution, but much more dynamic, and a little more DRY, though more complex to read.
First you create your manager dynamically:
def globals(inst):
return inst.get_query_set().filter(is_global=1)
GlobalDeleteManager = create_manager(DeleteManager, globals=globals)
This creates a new manager which is a subclass of DeleteManager and has a method called globals.
Next, you create your mixin model:
GlobalDeleteMixin = create_mixin(DeleteMixin,
is_global=models.BooleanField(default=False),
objects = GlobalDeleteManager())
Like I said, it's ugly. But it means you don't have to redefine globals(). If you want a different type of manager to have globals(), you just call create_manager again with a different base. And you can add as many new methods as you like. Same for the manager, you just keep adding new functions that will return different querysets.
So, is this really practical? Maybe not. This answer is more an exercise in (ab)using Python's flexibility. I haven't tried using this, though I do use some of the underlying principals of dynamically extending classes to make things easier to access.
Let me know if anything is unclear and I'll update the answer.
Another option worth considering is the PassThroughManager:
https://django-model-utils.readthedocs.org/en/latest/managers.html#passthroughmanager
You should use QuerySet instead of Manager.
See Documentation here.
Related
I have an Events table which contains various types of events. I care only about one of those types. As a result, every query I write begins with
Events.objects.filter(event_type="the_type").\
etc(...).etc(...)`.
Obviously this is repetitive and easy to forget. Is there a way to use a custom Manager so that the objects attribute always returns a specific subset of the rows, without explicitly asking for it? Or any other way to restrict the model to specific subset of the rows??
Yes, we can make a manager like:
from django.db import models
class EventManager(models.Manager):
def get_queryset(self):
return super(EventManager, self).get_queryset().filter(event_type="the_type")
and then add the manager to the Event class:
class Event(models.Model):
# ...
objects = EventManager()
Note however that some parts of Django will not use .objects, but ._base_manager, and thus still return the entire set. Furthermore my own experience with overriding the .objects manager is that it turns out to cause a lot of harm, for example if you want to set an attribute of all events, then writing Event.objects.all().update(foo='bar') will only update the events with the_type as type, whereas the code suggests otherwise.
Personally I think it is better to construct a manager with a different name, that at least hints that something is filtered, for example:
class Event(models.Model):
# ...
all_events = models.Manager()
type_events = EventManager()
here Event.objects no longer exist, but you write Event.all_events, or Event.type_events, and thus the code clearly hints what subset you take.
I have a Django model where a lot of fields are choices. So I had to write a lot of "is_something" properties of the class to check whether the instance value is equal to some choice value. Something along the lines of:
class MyModel(models.Model):
some_choicefield = models.IntegerField(choices=SOME_CHOICES)
#property
def is_some_value(self):
return self.some_choicefield == SOME_CHOICES.SOME_CHOICE_VALUE
# a lot of these...
In order to automate this and spare me a lot of redundant code, I thought about patching the instance at creation, with a function that adds a bunch of methods that do the checks.
The code became as follows (I'm assuming there's a "normalize" function that makes the label of the choice a usable function name):
def dynamic_add_checks(instance, field):
if hasattr(field, 'choices'):
choices = getattr(field, 'choices')
for (value,label) in choices:
def fun(instance):
return getattr(instance, field.name) == value
normalized_func_name = "is_%s_%s" % (field.name, normalize(label))
setattr(instance, normalized_func_name, fun(instance))
class MyModel(models.Model):
def __init__(self, *args, **kwargs):
super(MyModel).__init__(*args, **kwargs)
dynamic_add_checks(self, self._meta.get_field('some_choicefield')
some_choicefield = models.IntegerField(choices=SOME_CHOICES)
Now, this works but I have the feeling there is a better way to do it. Perhaps at class creation time (with metaclasses or in the new method)? Do you have any thoughts/suggestions about that?
Well I am not sure how to do this in your way, but in such cases I think the way to go is to simply create a new model, where you keep your choices, and change the field to ForeignKey. This is simpler to code and manage.
You can find a lot of information at a basic level in Django docs: Models: Relationships. In there, there are many links to follow expanding on various topics. Beyong that, I believe it just needs a bit of imagination, and maybe trial and error in the beginning.
I came across a similar problem where I needed to write large number of properties at runtime to provide backward compatibility while changing model fields. There are 2 standard ways to handle this -
First is to use a custom metaclass in your models, which inherits from models default metaclass.
Second, is to use class decorators. Class decorators sometimes provides an easy alternative to metaclasses, unless you have to do something before the creation of class, in which case you have to go with metaclasses.
I bet you know Django fields with choices provided will automatically have a display function.
Say you have a field defined like this:
category = models.SmallIntegerField(choices=CHOICES)
You can simply call a function called get_category_display() to access the display value. Here is the Django source code of this feature:
https://github.com/django/django/blob/baff4dd37dabfef1ff939513fa45124382b57bf8/django/db/models/base.py#L962
https://github.com/django/django/blob/baff4dd37dabfef1ff939513fa45124382b57bf8/django/db/models/fields/init.py#L704
So we can follow this approach to achieve our dynamically set property goal.
Here is my scenario, a little bit different from yours but down to the end it's the same:
I have two classes, Course and Lesson, class Lesson has a ForeignKey field of Course, and I want to add a property name cached_course to class Lesson which will try to get Course from cache first, and fallback to database if cache misses:
Here is a typical solution:
from django.db import models
class Course(models.Model):
# some fields
class Lesson(models.Model):
course = models.ForeignKey(Course)
#property
def cached_course(self):
key = key_func()
course = cache.get(key)
if not course:
course = get_model_from_db()
cache.set(key, course)
return course
Turns out I have so many ForeignKey fields to cache, so here is the code following the similar approach of Django get_FIELD_display feature:
from django.db import models
from django.utils.functional import curry
class CachedForeignKeyField(models.ForeignKey):
def contribute_to_class(self, cls, name, **kwargs):
super(models.ForeignKey, self).contribute_to_class(cls, name, **kwargs)
setattr(cls, "cached_%s" % self.name,
property(curry(cls._cached_FIELD, field=self)))
class BaseModel(models.Model):
def _cached_FIELD(self, field):
value = getattr(self, field.attname)
Model = field.related_model
return cache.get_model(Model, pk=value)
class Meta:
abstract = True
class Course(BaseModel):
# some fields
class Lesson(BaseModel):
course = CachedForeignKeyField(Course)
By customizing CachedForeignKeyField, and overwrite the contribute_to_class method, along with BaseModel class with a _cached_FIELD method, every CachedForeignKeyField will automatically have a cached_FIELD property accordingly.
Too good to be true, bravo!
I have been trying to figure out the best way to automate creating multiple SQL tables based on separate but identical models, all based on the same base class. I'm basically creating pseudo message boards or walls with different Groups, and I wanted each Group to have its own db_table of Posts, each Post containing the user id, timestamp, etc.
My first thought was to have one base class of Posts and just include a field for Group name, but I thought this would be bad practice. My rationale was that one table containing every Post for all Groups would get really big (in theory anyway) and slow down filtering, and also that the extra field for group name would in the long run be a waste of memory when I could have separate tables per group and skip this field.
I've also considered using a ForeignKey with a Many-to-One relationship, but as far as I can tell this has the same drawbacks. Am I wrong to think that? Or are these size concerns not really an issue?
So my next idea was to make Posts an abstract class, and then create subclasses based on each Group. This is ultimately what I did. However, I found myself having to copy and paste the code over and over and change the class name each time. This felt very unPythonic to me. It was something like:
class Posts(models.Model):
timestamp = models.DateTimeField(auto_now_add=True, unique=False)
user_id = ...
#etc.
#
class Meta:
abstract = True
class GroupA(Posts):
class Meta(Posts.Meta):
db_table = 'groupa_board'
class GroupB(Posts):
class Meta(Posts.Meta):
db_table = 'groupb_board'
class GroupC...etc.
What I really was looking for was a factory function to do this for me. I tried this sort of thing:
def makeBoard(group):
class Board(Posts):
class Meta(Posts.Meta):
db_table = group
return board #note I tried with and without this line
And then I ran a simple for loop using a list of groups.
for group in groups:
makeBoard(group)
I found myself hitting a RuntimeError: conflicting models in application, and I probably deserved it. So then I figured what I need is something like:
def makeBoard(group):
class group(Posts): #***group here being a variable, not the class name
class Meta(Posts.Meta):
db_table = '%s' % group #maybe issues here too, but the table
return group #name is not that important if the class
#name works
But I couldn't figure out how to make this work! Is there a way to pass a variable from a list to a class name?
Anyway if you're still with me I appreciate it. I've been on stackoverflow all day and while I've found guides for creating abstract base classes and subclasses to solve similar issues, I didn't see a way to create a function to do this for me. I ultimately punted here and just make a subclass for each group by hand. If there is a way to automate this process, I'd love to hear it.
Also, if I'm being stupid for not just going with one db table containing every post, I'd like to know that too, and why! Or if there's a better way to implement this kind of system altogether. I apologize if this has been answered before, I really couldn't find it.
Thank you!
Using a single table would not be bad practice. The extra memory is minimal, on modern systems that shouldn't be a problem. You shouldn't worry about performance either, premature optimization (not including the actual system design) is considered bad practice, but if you run into performance problems you can always specify an index on the group column:
group = models.CharField(max_length=100, db_index=True)
That's not to say that it is the best option, or that your method isn't good. Also, it is entirely possible to dynamically create models, using the type() built-in function. The only difference with dynamically creating models and creating other classes is that you must specifically pass the __module__ attribute. You can create subclasses for Posts in the following way:
def fabric(names, baseclass=Posts):
for name in names:
class Meta:
db_table = '%s_table' % name.lower()
attrs = {'__module__': baseclass.__module__, 'Meta': Meta}
# specify any other class attributes here. E.g. you can specify extra fields:
attrs.update({'my_field': models.CharField(max_length=100)})
newclass = type(str(name), (baseclass,), attrs)
globals()[name] = newclass
fabric(['GroupA', 'GroupB', 'GroupC', etc...])
Put that code in your models.py after your Posts class, and all classes will be created for you. They can be used in any way normal classes can be used: Django doesn't even know you dynamically created this class. Though your Meta class doesn't inherit from Posts.Meta, your meta settings should still be preserved.
Tested with Django 1.4.
Try smth like this
import app.models as group_models
from django.db.models.base import ModelBase
def fabric(group):
for item in dir(group_models):
c = getattr(group_models, item)
if type(c) is ModelBase:
if c._meta.db_table == '%s_table' % group:
return c
return None
This is a very simple question: Is there any good way to disable calling a bulk-delete (through querysets of course) on all models in an entire Django project?
The reasoning for this is under the premise that completely deleting data is almost always a poor choice, and an accidental bulk-delete can be detrimental.
Like comments says on your first post, you have to create a subclass for each of these elements:
The model manager
Queryset class
BaseModel
After some search, a great example can be found here, all credits to Akshay Shah, the blog author. Before looking to the code, be aware of that:
However, it inevitably leads to data corruption. The problem is simple: using a Boolean to store deletion status makes it impossible to enforce uniqueness constraints in your database.
from django.db import models
from django.db.models.query import QuerySet
class SoftDeletionQuerySet(QuerySet):
def delete(self):
# Bulk delete bypasses individual objects' delete methods.
return super(SoftDeletionQuerySet, self).update(alive=False)
def hard_delete(self):
return super(SoftDeletionQuerySet, self).delete()
def alive(self):
return self.filter(alive=True)
def dead(self):
return self.exclude(alive=True)
class SoftDeletionManager(models.Manager):
def __init__(self, *args, **kwargs):
self.alive_only = kwargs.pop('alive_only', True)
super(SoftDeletionManager, self).__init__(*args, **kwargs)
def get_queryset(self):
if self.alive_only:
return SoftDeletionQuerySet(self.model).filter(alive=True)
return SoftDeletionQuerySet(self.model)
def hard_delete(self):
return self.get_queryset().hard_delete()
class SoftDeletionModel(models.Model):
alive = models.BooleanField(default=True)
objects = SoftDeletionManager()
all_objects = SoftDeletionManager(alive_only=False)
class Meta:
abstract = True
def delete(self):
self.alive = False
self.save()
def hard_delete(self):
super(SoftDeletionModel, self).delete()
Basically, it adds an alive field to check if the row has been deleted or not, and update it when the delete() method is called.
Of course, this method works only on project where you can manipulate the code base.
There are nice off-the-shelf applications that allow for restoring deleted models (if that is what you are interested in), here are ones I used:
Django softdelete: https://github.com/scoursen/django-softdelete I used it more
Django reversion: https://github.com/etianen/django-reversion this one is updated more often, and allows you to revert to any version of your model (not only after delete, but as well after update).
If you really want to forbid bulk deletes, I'd discourage you from this approach as it will:
Break expectations about applicaiton behaviour. If I call MyModel.objects.all().delete() I want table to be empty afterwards.
Break existing applications.
If you want do do it please follow advice from comment:
I'm guessing this would involve subclassing QuerySet and changing the delete method to your liking, subclassing the default manager and have it use your custom query set, subclassing model - create an abstract model and have it use your custom manager and then finally have all your models subclass your custom abstract model.
In Django, when you have a parent class and multiple child classes that inherit from it you would normally access a child through parentclass.childclass1_set or parentclass.childclass2_set, but what if I don't know the name of the specific child class I want?
Is there a way to get the related objects in the parent->child direction without knowing the child class name?
(Update: For Django 1.2 and newer, which can follow select_related queries across reverse OneToOneField relations (and thus down inheritance hierarchies), there's a better technique available which doesn't require the added real_type field on the parent model. It's available as InheritanceManager in the django-model-utils project.)
The usual way to do this is to add a ForeignKey to ContentType on the Parent model which stores the content type of the proper "leaf" class. Without this, you may have to do quite a number of queries on child tables to find the instance, depending how large your inheritance tree is. Here's how I did it in one project:
from django.contrib.contenttypes.models import ContentType
from django.db import models
class InheritanceCastModel(models.Model):
"""
An abstract base class that provides a ``real_type`` FK to ContentType.
For use in trees of inherited models, to be able to downcast
parent instances to their child types.
"""
real_type = models.ForeignKey(ContentType, editable=False)
def save(self, *args, **kwargs):
if self._state.adding:
self.real_type = self._get_real_type()
super(InheritanceCastModel, self).save(*args, **kwargs)
def _get_real_type(self):
return ContentType.objects.get_for_model(type(self))
def cast(self):
return self.real_type.get_object_for_this_type(pk=self.pk)
class Meta:
abstract = True
This is implemented as an abstract base class to make it reusable; you could also put these methods and the FK directly onto the parent class in your particular inheritance hierarchy.
This solution won't work if you aren't able to modify the parent model. In that case you're pretty much stuck checking all the subclasses manually.
In Python, given a ("new-style") class X, you can get its (direct) subclasses with X.__subclasses__(), which returns a list of class objects. (If you want "further descendants", you'll also have to call __subclasses__ on each of the direct subclasses, etc etc -- if you need help on how to do that effectively in Python, just ask!).
Once you have somehow identified a child class of interest (maybe all of them, if you want instances of all child subclasses, etc), getattr(parentclass,'%s_set' % childclass.__name__) should help (if the child class's name is 'foo', this is just like accessing parentclass.foo_set -- no more, no less). Again, if you need clarification or examples, please ask!
Carl's solution is a good one, here's one way to do it manually if there are multiple related child classes:
def get_children(self):
rel_objs = self._meta.get_all_related_objects()
return [getattr(self, x.get_accessor_name()) for x in rel_objs if x.model != type(self)]
It uses a function out of _meta, which is not guaranteed to be stable as django evolves, but it does the trick and can be used on-the-fly if need be.
It turns out that what I really needed was this:
Model inheritance with content type and inheritance-aware manager
That has worked perfectly for me. Thanks to everyone else, though. I learned a lot just reading your answers!
You can use django-polymorphic for that.
It allows to automatically cast derived classes back to their actual type. It also provides Django admin support, more efficient SQL query handling, and proxy model, inlines and formset support.
The basic principle seems to be reinvented many times (including Wagtail's .specific, or the examples outlined in this post). It takes more effort however, to make sure it doesn't result in an N-query issue, or integrate nicely with the admin, formsets/inlines or third party apps.
Here's my solution, again it uses _meta so isn't guaranteed to be stable.
class Animal(models.model):
name = models.CharField()
number_legs = models.IntegerField()
...
def get_child_animal(self):
child_animal = None
for r in self._meta.get_all_related_objects():
if r.field.name == 'animal_ptr':
child_animal = getattr(self, r.get_accessor_name())
if not child_animal:
raise Exception("No subclass, you shouldn't create Animals directly")
return child_animal
class Dog(Animal):
...
for a in Animal.objects.all():
a.get_child_animal() # returns the dog (or whatever) instance
You can achieve this looking for all the fields in the parent that are an instance of django.db.models.fields.related.RelatedManager. From your example it seems that the child classes you are talking about are not subclasses. Right?
An alternative approach using proxies can be found in this blog post. Like the other solutions, it has its benefits and liabilities, which are very well put in the end of the post.