I have a foreign key relationship in my Django (v3) models:
class Example(models.Model):
title = models.CharField(max_length=200) # this is irrelevant for the question here
not_before = models.DateTimeField(auto_now_add=True)
...
class ExampleItem(models.Model):
myParent = models.ForeignKey(Example, on_delete=models.CASCADE)
execution_date = models.DateTimeField(auto_now_add=True)
....
Can I have code running/triggered whenever an ExampleItem is "added to the list of items in an Example instance"? What I would like to do is run some checks and, depending on the concrete Example instance possibly alter the ExampleItem before saving it.
To illustrate
Let's say the Example's class not_before date dictates that the ExampleItem's execution_date must not be before not_before I would like to check if the "to be saved" ExampleItem's execution_date violates this condition. If so, I would want to either change the execution_date to make it "valid" or throw an exception (whichever is easier). The same is true for a duplicate execution_date (i.e. if the respective Example already has an ExampleItem with the same execution_date).
So, in a view, I have code like the following:
def doit(request, example_id):
# get the relevant `Example` object
example = get_object_or_404(Example, pk=example_id)
# create a new `ExampleItem`
itm = ExampleItem()
# set the item's parent
itm.myParent = example # <- this should trigger my validation code!
itm.save() # <- (or this???)
The thing is, this view is not the only way to create new ExampleItems; I also have an API for example that can do the same (let alone that a user could potentially "add ExampleItems manually via REPL). Preferably the validation code must not be duplicated in all the places where new ExampleItems can be created.
I was looking into Signals (Django docu), specifically pre_save and post_save (of ExampleItem) but I think pre_save is too early while post_save is too late... Also m2m_changed looks interesting, but I do not have a many-to-many relationship.
What would be the best/correct way to handle these requirements? They seem to be rather common, I imagine. Do I have to restructure my model?
The obvious solution here is to put this code in the ExampleItem.save() method - just beware that Model.save() is not invoked by some queryset bulk operations.
Using signals handlers on your own app's models is actually an antipattern - the goal of signal is to allow for your app to hook into other app's lifecycle without having to change those other apps code.
Also (unrelated but), you can populate your newly created models instances directly via their initializers ie:
itm = ExampleItem(myParent=example)
itm.save()
and you can even save them directly:
# creates a new instance, populate it AND save it
itm = ExampleItem.objects.create(myParent=example)
This will still invoke your model's save method so it's safe for your use case.
As a project grows, so do dependencies and event chains, especially in overridden save() methods and post_save and pre_save signals.
Example:
An overridden A.save creates two related objects to A - B and C. When C is saved, the post_save signal is invoked that does something else, etc...
How can these event chins be made more clear? Is there a way to visualize (generate automatically) such chains/flows? I'm not looking for ERD nor a Class diagram. I need to be sure that doing one thing one place won't affect something on the other side of the project, so simple visualization would be the best.
EDIT
To be clear, I know that it would be almost impossible to check dynamically generated signals. I just want to check all (not dynamically generated) post_save, pre_save, and overridden save methods and visualize them so I can see immediately what is happening and where when I save something.
This is not the full solution, but I hope it can be a good starting point. Consider this code:
from django.db import models
from django.db.models.signals import pre_save
from django.dispatch import receiver
class A(models.Model):
def save(self, *args, **kwargs):
if not self.pk:
C.objects.create()
class B(models.Model):
pass
class C(models.Model):
b = models.ForeignKey(B, on_delete=models.CASCADE, blank=True)
#receiver(pre_save, sender=C)
def pre_save_c(sender, instance, **kwargs):
if not instance.pk:
b = B.objects.create()
instance.b = b
We can get the dependencies for the app name list using inspect, django get_models(), and signals in this manner:
import inspect
import re
from collections import defaultdict
from django.apps import apps
from django.db.models import signals
RECEIVER_MODELS = re.compile('sender=(\w+)\W')
SAVE_MODELS = re.compile('(\w+).objects.')
project_signals = defaultdict(list)
for signal in vars(signals).values():
if not isinstance(signal, signals.ModelSignal):
continue
for _, receiver in signal.receivers:
rcode = inspect.getsource(receiver())
rmodel = RECEIVER_MODELS.findall(rcode)
if not rmodel:
continue
auto_by_signals = [
'{} auto create -> {}'.format(rmodel[0], cmodel)
for cmodel in SAVE_MODELS.findall(rcode)
]
project_signals[rmodel[0]].extend(auto_by_signals)
for model in apps.get_models():
is_self_save = 'save' in model().__class__.__dict__.keys()
if is_self_save:
scode = inspect.getsource(model.save)
model_name = model.__name__
for cmodel in SAVE_MODELS.findall(scode):
print('{} auto create -> {}'.format(model_name, cmodel))
for smodels in project_signals.get(cmodel, []):
print(smodels)
This gives:
A auto create -> C
C auto create -> B
Updated: change method to found overridden save by the instance class dict.
is_self_save = 'save' in model().__class__.__dict__.keys()
(Too long to fit into a comment, lacking code to be a complete answer)
I can't mock up a ton of code right now, but another interesting solution, inspired by Mario Orlandi's comment above, would be some sort of script that scans the whole project and searches for any overridden save methods and pre and post save signals, tracking the class/object that creates them. It could be as simple as a series of regex expressions that look for class definitions followed by any overridden save methods inside.
Once you have scanned everything, you could use this collection of references to create a dependency tree (or set of trees) based on the class name and then topologically sort each one. Any connected components would illustrate the dependencies, and you could visualize or search these trees to see the dependencies in a very easy, natural way. I am relatively naive in django, but it seems you could statically track dependencies this way, unless it is common for these methods to be overridden in multiple places at different times.
If you only want to track models saves, and not interested in other things happening inside overridden save methods and signals, you can use a mechanism like angio. You can register a global post_save receiver, without a sender argument, one that will be called for all model saves, and print the saved model name in that function. Then, write a script to just call save for all existing models. Something like the following could work:
#receiver(models.signals.post_save)
def global_post_save(sender, instance, created, *args, **kwargs):
print(' --> ' + str(sender.__name__))
from django.apps import apps
for model in apps.get_models():
instance = model.objects.first()
if instance:
print('Saving ' + str(model.__name__))
instance.save()
print('\n\n')
With the following model structure;
class A(models.Model):
...
def save(self, *args, **kwargs):
B.objects.create()
#receiver(post_save, sender=B)
def post_save_b(sender, instance, **kwargs):
C.objects.create()
The script would print:
Saving A
--> A
--> B
--> C
Saving B
--> B
--> C
Saving C
--> C
This is just a basic sketch of what could be done, and can be improved according to the structure of your application. This assumes you already have an entry in the db for each model. Though not changing anything, this approach also saves things in the database, so would be better run on a test db.
Assuming your ultimate goal is to track changes in database when an instance of some model is saved, one potential solution can be scanning database for changes instead of source code. The upside of this approach is that it can also cover dynamic code. And downside is, obviously, it will ONLY cover database changes.
This can be achieved using simple testing techniques. Assuming following models..
from django.db import models
from django.db.models.signals import pre_save, post_save
from django.dispatch import receiver
class B(models.Model):
def save(self, *args, **kwargs):
X.objects.create()
super().save(*args, **kwargs)
class C(models.Model):
y = models.OneToOneField('Y', on_delete=models.CASCADE)
class D(models.Model):
pass
class X(models.Model):
pass
class Y(models.Model):
related = models.ForeignKey('Z', on_delete=models.CASCADE)
class Z(models.Model):
pass
#receiver(pre_save, sender=D)
def pre_save_d(*args, instance, **kwargs):
Z.objects.create()
#receiver(post_save, sender=C)
def pre_save_c(*args, instance, **kwargs):
Y.objects.create(related=Z.objects.create())
I can write up a test case which takes count all database instances, creates an instance of a model, takes count again and calculates the difference. Database instances can be created using factories like mommy. Here is a simple but working example of this technique.
class TestModelDependency(TestCase):
def test_dependency(self):
models = apps.get_models()
models = [model for model in models if model._meta.app_label == 'model_effects']
for model in models:
kwargs = self.get_related_attributes(model)
initial_count = self.take_count(models)
mommy.make(model, **kwargs)
final_count = self.take_count(models)
diff = self.diff(initial_count, final_count)
print(f'Creating {model._meta.model_name}')
print(f'Created {" | ".join(f"{v} instance of {k}" for k, v in diff.items())}')
call_command('flush', interactive=False)
#staticmethod
def take_count(models):
return {model._meta.model_name: model.objects.count() for model in models}
#staticmethod
def diff(initial, final):
result = dict()
for k, v in final.items():
i = initial[k]
d = v - i
if d != 0:
result[k] = d
return result
#staticmethod
def get_related_attributes(model):
kwargs = dict()
for field in model._meta.fields:
if any(isinstance(field, r) for r in [ForeignKey, OneToOneField]):
kwargs[field.name] = mommy.make(field.related_model)
return kwargs
And my output is
Creating b
Created 1 instance of b | 1 instance of x
Creating c
Created 1 instance of c | 1 instance of y | 1 instance of z
Creating d
Created 1 instance of d | 1 instance of z
Creating x
Created 1 instance of x
Creating y
Created 1 instance of y
Creating z
Created 1 instance of z
For large applications it can be slow, but I use in memory sqlite database for testing and it runs pretty fast.
I'working in a Django app that does something similar, but while I get it done, I will comment about the use case you've presented here:
I need to be sure that doing one thing one place won't affect something on the other side of the project ...
You surely could write tests with some dummy signal handlers in order to know if the execution of certain code, triggers unwanted behavior, for instance:
# I use pytest, put this example is suitable also for
# django's TestCase and others
class TestSome:
# For Django TestCase this would be setUp
def setup_method(self, test_method):
self.singals_info = []
def dummy_handler(*args, **kwargs):
# collect_info is a function you must implement, it would
# gather info about signal, sender, instance, etc ... and
# save that info in (for example) self.signals_info.
# You can then use that info for test assertions.
self.collect_info(*args, **kwargs)
# connect your handler to every signal you want to control
post_save.connect(dummy_handler)
def test_foo():
# Your normal test here ...
some_value = some_tested_function()
# Check your signals behave
assert self.signals_behave(self.signals_info)
Why this is better than having a script that show's the events chain?
Well, as you say, when the need for things like this emerges, is because the size of the project is very big, and if you use a tool like you are asking for, you can end with a result like this:
Save A -> Creates B -> Creates C
Save B -> Creates D
Save B -> Creates C
.
.
.
# Imagine here 3 or 4 more lines.
You will end up solving a puzzle every time you want to add some code, that saves/modify something.
However ...
It would be better, to write your code, and then, some test fails (resolving the puzzle for you) and showing to you exactly where will your code miss-behave.
Conclusion:
Implements those tests and your life will be easier.
Best scenario using tests: Write your code, and if no test fail, you're ready to tackle your next programming task.
Worst scenario using tests: Write your code, some test fail, as you know where exactly your code broke, just fix it.
Best scenario using the tool: Analyze the tool output, write your code, everything is ok.
Worst scenario using the tool: Analyze the tool output, write your code, something fails, repeat until all ok.
So, a tool like that would be helpful? Of course, but is not the right tool to ensure things are well, use tests for that.
I have a model like this
class Authority(models.Model):
name=models.CharField(max_length=100)
country=models.ForeignKey(Country)
category=models.ForeignKey(Category)
competitors=models.ManyToManyField("self",related_name="competitors")
I want authorities having the same country and category and itself to be automatically give an M2M relationship,so i did this
def save(self,*args,**kwargs):
z=Authority.objects.filter(country=self.country).filter(category=self.category)
this_authority=Authority.objects.get(id=self.id)
for a in z:
this_authority.competitors.add(a)
super(Authority,self).save(*args,**kwargs)
It wasn't working and not bringing any error,I also tries this below
def save(self,*args,**kwargs):
z=Authority.objects.filter(country=self.country).filter(category=self.category)
this_authority=Authority.objects.get(id=self.id)
self.competitors=z
super(Authority,self).save(*args,**kwargs)
What might be wrong with my code?Thanks in advance.
The reason this isn't working the way you expect is because of how Django handles creating m2m relationships in the database. Long story very short, when you save something like a new Authority to the database, Django first writes the new object then goes back in and writes the m2m relationships for that new object. As a result, it's tough to do anything useful to m2m relationships in a custom save method.
A post-save signal may do the trick here. kwargs['created'] = True if we're creating a new object and kwargs['instance'] is the instance whose save fired off the signal receiver.
#receiver(post_save, sender = Authority)
def update_m2m_relationships(sender, **kwargs):
if kwargs['created']: #only fire when creating new objects
competitors_to_add = Authority.objects.filter(
country = kwargs['instance'].country,
category = kwargs['instance'].category
)
for c in competitors_to_add:
c.competitors.add(kwargs['instance'])
c.save() #not creating a new object; this receiver does not fire here
kwargs['instance'].competitors.add(c)
#all competitors have been added to the instance's m2m field
kwargs['instance'].save()
It's important to only fire this when creating new objects. If you don't include that restriction, then the receiver will trigger itself as you update other objects in your for loop.
I haven't tested this out but I think it'll work. Let me know if it doesn't and I'll do my best to help.
i'm writing an app with django and i need to change a specific model when
ever it been saved. i.e lets say i have a model A and a client want to save
changes to that model - i need to also save a change (only if the client
changed a certain field) to the same model (not instance).
my code:
#receiver(pre_save, sender=A)
def my_callable(sender, instance, **kwargs):
a = A.objects.filter(b=True).all()
for my_a in a:
my_a.b= False
my_a.save()
i have 2 problems with that code:
it has an infinite recursion
i don't know how to check which field had changed
and ideas?
Use .update(b=False) on the queryset:
#receiver(pre_save, sender=A)
def my_callable(sender, instance, **kwargs):
A.objects.filter(b=True).update(b=False)
The update is done in SQL, doesn't call model's save() method or trigger any signals
.update() docs
Assuming you're able to use Django1.8, this exact use case is covered in the docs actually: https://docs.djangoproject.com/en/1.8/ref/models/instances/#customizing-model-loading
Cliff notes: use the from_db method to customize loading of the model and save a copy of the instance's attributes as it is loaded, and then compare them before it is saved.
If you want to compare form data to a model instance to see if a user is changing it, do that in the view, not with a signal.
I have a situation where I need to notify some users when something in DB changes. My idea is to catch pre_save and post_save signal and make some kind of diff and mail that. Generally it works good, but I don't know how to get diff for m2m fields.
At the moment I have something like this:
def pre_save(sender, **kwargs):
pk = kwargs['instance'].pk
instance = copy.deepcopy(sender.objects.get(pk=pk))
tracking[sender] = instance
def post_save(sender, **kwargs):
instance = copy.deepcopy(kwargs['instance'])
print diff(instance, (tracking[sender])) # TODO: don't print, save diff somewhere
Diff function should work for every model (at the mommet I have four model classes). With deepcopy I can save old model, but I don't know how to save m2m fields because they are in separate table (yes, I know I can get this data, but at the momment of execution I don't know what fields are m2m and I wouldn't like to create different slot for every model). What I would like is generic solution, so I can just add models later without thinking about notification part.
My plan is to call get_data() and clear_data() functions after save() in view to clean diff that slots have generated.
Is this good way of doing this? Is there a better way? Is there django application that can do this job for me?
Excuse my English, it's not my native language.
First of all, you don't need to use deepcopy for this. Re-querying the sender from the database returns a "fresh" object.
def pre_save(sender, **kwargs):
pk = kwargs['instance'].pk
instance = sender.objects.get(pk=pk)
tracking[sender] = instance
You can get a list of all the many-to-many fields for a class, and check the values related to the current instance:
for field in sender._meta.local_many:
values = field.value_from_object(instance).objects.all()
# Now values is a list of related objects, which you can diff