Develop Reference

Generating marshmallow schema automatically with JSON serializable enums

Long gone are the days of creating marshmallow schemas identical to my models. I found this excellent answer that explained how I could auto generate schemas from my SQA models using a simple decorator, so I implemented it and replaced the deprecated ModelSchema for the newer SQLAlchemyAutoSchema:
def add_schema(cls):
class Schema(SQLAlchemyAutoSchema):
class Meta:
model = cls
cls.Schema = Schema
return cls
This worked great... until I bumped into a model with a bloody Enum.
The error: Object of type MyEnum is not JSON serializable
I searched online and I found this useful answer.
But I'd like to implement it as part of the decorator so that it is generated automatically as well. In other words, I'd like to automatically overwrite all Enums in my model with EnumField(TheEnum, by_value=True) when generating the schema using the add_schema decorator; that way I won't have to overwrite all the fields manually.
What would be the best way to do this?

I have found that the support for enum types that was initially suggested only works if OneOf is the only validation class that exists in field_details. I added in some argument parsing (in a rudimentary way by looking for choices after stringifying the results _repr_args() from OneOf) to check the validation classes to hopefully make this implementation more universally usable:
def add_schema(cls):
class Schema(ma.SQLAlchemyAutoSchema):
class Meta:
model = cls
fields = Schema._declared_fields
# support for enum types
for field_name, field_details in fields.items():
if len(field_details.validate) > 0:
check = str(field_details.validate[0]._repr_args)
if check.__contains__("choices") :
enum_list = field_details.validate[0].choices
enum_dict = {enum_list[i]: enum_list[i] for i in range(0, len(enum_list))}
enum_clone = Enum(field_name.capitalize(), enum_dict)
fields[field_name] = EnumField(enum_clone, by_value=True, validate=validate.OneOf(enum_list))
cls.Schema = Schema
return cls
Thank you jgozal for the initial solution, as I really needed this lead for my current project.

This is my solution:
from marshmallow import validate
from marshmallow_sqlalchemy import SQLAlchemyAutoSchema
from marshmallow_enum import EnumField
from enum import Enum
def add_schema(cls):
class Schema(SQLAlchemyAutoSchema):
class Meta:
model = cls
fields = Schema._declared_fields
# support for enum types
for field_name, field_details in fields.items():
if len(field_details.validate) > 0:
enum_list = field_details.validate[0].choices
enum_dict = {enum_list[i]: enum_list[i] for i in range(0, len(enum_list))}
enum_clone = Enum(field_name.capitalize(), enum_dict)
fields[field_name] = EnumField(enum_clone, by_value=True, validate=validate.OneOf(enum_list))
cls.Schema = Schema
return cls
The idea is to iterate over the fields in the Schema and find those that have validation (usually enums). From there we can extract a list of choices which can then be used to build an enum from scratch. Finally we overwrite the schema field with a new EnumField.
By all means, feel free to improve the answer!

How to hint the type of Django's model field `objects` to a dynamically generated class?

I have a Team model in my Django project. I create its custom model manager with QuerySet.as_manager().
class TeamQuerySet(models.QuerySet):
def active(self) -> "models.QuerySet[Team]":
return self.filter(is_active=True)
class Team(models.Model):
is_active = models.BooleanField()
objects = TeamQuerySet.as_manager()
When I try to execute Team.objects.active(), mypy gives the following error:
error: "Manager[Any]" has no attribute "active"
In [5]: Team.objects
Out[5]: <django.db.models.manager.ManagerFromTeamQuerySet at 0x10eee1f70>
If I was explicitly defining a TeamManager class, there would be not a problem. How can I hint the type of Django model field objects to a dynamically generated class?

Based on the Manager[Any] I assume you are already using django-stubs.
Unfortunately it looks like there's an open issue to make QuerySet.as_manager generic over the model it's attached to that has not been resolved yet.
Even if the PR addressing the issue got merged I'm afraid it wouldn't address your immediate issue because the as_manager needs to be generic over the generic QuerySet subclass used to create the manager in order for both .active to be available and attributes relating to Team be available.
In this regard this other PR, which is unfortunately quite stale, seems to properly address your issue.

I've worked around this with a little switch-a-roo for MyPy's sake:
_Q = TypeVar("_Q", bound="WorkflowQuerySet")
class WorkflowQuerySet(models.QuerySet["WorkflowModel"]):
"""
Queryset for workflow objects.
"""
def count_objects(self) -> int:
raise NotImplementedError
def latest_objects(self: _Q) -> _Q:
raise NotImplementedError
if TYPE_CHECKING:
# Create a type MyPy understands
class WorkflowManager(models.Manager["WorkflowModel"]):
def count_objects(self) -> int:
...
def latest_objects(self) -> _Q:
...
else:
WorkflowManager = WorkflowQuerySet.as_manager
class WorkflowModel(models.Model):
"""
A model that has workflow.
"""
objects = WorkflowManager()

Here is my answer using generics and typevar
from typing import Generic, TypeVar
from django.db import models
class BookQueryset(models.QuerySet['Book']):
...
class Book(models.Model):
objects: BookQueryset = BookQueryset.as_manager()
book = Book.objects.all()[0]
If you inspect book is type Book

Django + Factory Boy: How to use a named parameter to control behavior of factory.Maybe + factory.RelatedFactory

Is there a way to define a named parameter (that is not a model attribute) to control the behavior of factory.Maybe?
For example: I want to create a named parameter that controls the creation of a RelatedFactory through a Maybe condition, I tried to do this:
# factories.py
class Purchase(factory.DjangoModelFactory):
user = factory.SubFactory('project.factories.UserFactory')
total = uniform(10, 100)
class Meta:
model = Purchase
class UserFactory(factory.DjangoModelFactory):
first_name = factory.Faker('first_name')
last_name = factory.Faker('last_name')
has_purchase = False
purchases = factory.Maybe(
'has_purchase',
yes_declaration=factory.RelatedFactory(Purchase, 'purchase'),
no_declaration=None
)
class Meta:
exclude = ['has_purchase', 'purchases']
model = User
# tests.py
user_without_purchase = UserFactory.create()
user_with_purchase = UserFactory.create(has_purchase=True)
It seems that I can't change the value of has_purchase through the named parameter, it's always set as False. I don't know if it's because both fields are declared in exclude Meta attribute. And I have to put them in exclude or else DjangoModelFactory tries to use these fields when saving it to the database, which causes an Integrity error.
Is there a way to do that?

It seems that factory boy have a feature to provided additional parameters without having to add theses fields to the exclude attribute: http://factoryboy.readthedocs.io/en/latest/reference.html#simple-parameters:
class UserFactory(factory.DjangoModelFactory):
class Params:
has_purchase = False
Used in conjunction with the http://factoryboy.readthedocs.io/en/latest/reference.html#lazyattribute maybe ?

Elaborating on Clément Denoix's answer, I've been able to add an optional argument to pass a value directly to a sub-factory (a shortcut, sort of):
class Y_Factory(DjangoModelFactory):
class Meta:
model = Y # has a field named 'z'
class X_Factory(DjangoModelFactory):
class Meta:
model = X # has a foreign key to Y
class Params:
z = None
y = factory.lazy_attribute(lambda o: Y_Factory(z=o.z or 'default_z'))
Now the following syntax X_Factory(y=Y_Factory(z=z)) can be simplified to X_Factory(z=z).
There is one caveat though: using this syntax will mislead newcomers as it wrongly depicts the data structure. Still, I decided that in one particular case, such a shortcut conveniently removed some annoying boilerplate code from my fixtures and was worth the trade-off.

How to override the model.Manager.create() method in Django?

I have plenty of Hardware models which have a HardwareType with various characteristics. Like so:
# models.py
from django.db import models
class HardwareType(model.Models):
name = models.CharField(max_length=32, unique=True)
# some characteristics of this particular piece of hardware
weight = models.DecimalField(max_digits=12, decimal_places=3)
# and more [...]
class Hardware(models.Model):
type = models.ForeignKey(HardwareType)
# some attributes
is_installed = models.BooleanField()
location_installed = models.TextField()
# and more [...]
If I wish to add a new Hardware object, I first have to retrieve the HardwareType every time, which is not very DRY:
tmp_hd_type = HardwareType.objects.get(name='NG35001')
new_hd = Hardware.objects.create(type=tmp_hd_type, is_installed=True, ...)
Therefore, I have tried to override the HardwareManager.create() method to automatically import the type when creating new Hardware like so:
# models.py
from django.db import models
class HardwareType(model.Models):
name = models.CharField(max_length=32, unique=True)
# some characteristics of this particular piece of hardware
weight = models.DecimalField(max_digits=12, decimal_places=3)
# and more [...]
class HardwareManager(models.Manager):
def create(self, *args, **kwargs):
if 'type' in kwargs and kwargs['type'] is str:
kwargs['type'] = HardwareType.objects.get(name=kwargs['type'])
super(HardwareManager, self).create(*args, **kwargs)
class Hardware(models.Model):
objects = HardwareManager()
type = models.ForeignKey(HardwareType)
# some attributes
is_installed = models.BooleanField()
location_installed = models.TextField()
# and more [...]
# so then I should be able to do:
new_hd = Hardware.objects.create(type='ND35001', is_installed=True, ...)
But I keep getting errors and really strange behaviors from the ORM (I don't have them right here, but I can post them if needed). I've searched in the Django documentation and the SO threads, but mostly I end up on solutions where:
the Hardware.save() method is overridden (should I get the HardwareType there ?) or,
the manager defines a new create_something method which calls self.create().
I also started digging into the code and saw that the Manager is some special kind of QuerySet but I don't know how to continue from there. I'd really like to replace the create method in place and I can't seem to manage this. What is preventing me from doing what I want to do?

The insight from Alasdair's answer helped a lot to catch both strings and unicode strings, but what was actually missing was a return statement before the call to super(HardwareManager, self).create(*args, **kwargs) in the HardwareManager.create() method.
The errors I was getting in my tests yesterday evening (being tired when coding is not a good idea :P) were ValueError: Cannot assign None: [...] does not allow null values. because the subsequent usage of new_hd that I had create()d was None because my create() method didn't have a return. What a stupid mistake !
Final corrected code:
class HardwareManager(models.Manager):
def create(self, *args, **kwargs):
if 'type' in kwargs and isinstance(kwargs['type'], basestring):
kwargs['type'] = HardwareType.objects.get(name=kwargs['type'])
return super(HardwareManager, self).create(*args, **kwargs)

Without seeing the traceback, I think the problem is on this line.
if 'type' in kwargs and kwargs['type'] is str:
This is checking whether kwargs['type'] is the same object as str, which will always be false.
In Python 3, to check whether `kwargs['type'] is a string, you should do:
if 'type' in kwargs and isinstance(kwargs['type'], str):
If you are using Python 2, you should use basestring, to catch byte strings and unicode strings.
if 'type' in kwargs and isinstance(kwargs['type'], basestring):

I was researching the same problem as you and decided not to use an override.
In my case making just another method made more sense given my constraints.
class HardwareManager(models.Manager):
def create_hardware(self, type):
_type = HardwareType.objects.get_or_create(name=type)
return self.create(type = _type ....)

Import issues with custom ModelField that refers to a Model in the same app

I've created a series of custom ModelFields that are simply restricted ForeignKeys. Below you'll find CompanyField. When instantiated, you may provide a type (e.g., Client, Vendor). With a type provided, the field ensures that only values that have the appropriate type are allowed.
The crm app, the one that defines the custom fields, compiles and runs just fine. Eventually I added references to the fields to a different app (incidents) using "from crm import fields". Now I'm seeing a whole bunch of errors like this:
incidents.incident: 'group' has a relation with model Company, which has either not been installed or is abstract.
Here are all the gory details. Please let me know if there's any more information I could provide which may be helpful.
## crm/fields.py
import models as crmmods
class CompanyField(models.ForeignKey):
def __init__(self, *args, **kwargs):
# This is a hack to get South working. In either case, we just need to
# make sure the FK refers to Company.
try:
# kwargs['to'] == crmmods.company doesn't work for some reason I
# still haven't figured out
if str(kwargs['to']) != str(crmmods.Company):
raise AttributeError("Only crm.models.Company is accepted " + \
"for keyword argument 'to'")
except:
kwargs['to'] = 'Company'
# See if a CompanyType was provided and, if so, store it as self.type
if len(args) > 0:
company_type = args[0]
# Type is expected to be a string or CompanyType
if isinstance(company_type, str):
company_type = company_type.upper()
if hasattr(crmmods.CompanyType, company_type):
company_type = getattr(crmmods.CompanyType, company_type)
else:
raise AttributeError(
"%s is not a valid CompanyType." % company_type)
elif not isinstance(company_type, crmmods.CompanyType):
raise AttributeError(
"Expected str or CompanyType for first argument.")
self.type = company_type
else:
self.type = None
super(CompanyField, self).__init__(**kwargs)
def formfield(self, **kwargs):
# Restrict the formfield so it only displays Companies with the correct
# type.
if self.type:
kwargs['queryset'] = \
crmmods.Company.objects.filter(companytype__role=self.type)
return super(CompanyField, self).formfield(**kwargs)
def validate(self, value, model_instance):
super(CompanyField, self).validate(value, model_instance)
# No type set, nothing to check.
if not value or not self.type:
return
# Ensure that value is correct type.
if not \
value.companytype_set.filter(role=self.type).exists():
raise ValidationError("Company does not have the " + \
"required roles.")
## crm/models.py
import fields
class CompanyType(models.Model):
name = models.CharField(max_length=25)
class Company(models.Model):
type = models.ForeignKey(CompanyType)
class Person(models.Model):
name = models.CharField(max_length=50)
company = fields.CompanyField("Client")
## incidents/models.py
from crm import fields as crmfields
class Incident(models.Model):
company = crmfields.CompanyField("Client")

You have a circular package dependency. fields imports models which imports fields which imports models which imports fields . . .
Circular package dependecies are A BAD IDEA(tm). Although it may work in some cases, it doesn't work in your case, for a complicated reason involving metaclasses, which I will spare myself.
EDIT
The reason is that the Django ORM module uses metaclasses to turn its class variables (the fields of the Model) into property descriptors on the object. This is done at class definition time by the metaclass. A class is defined when the module is loaded. For this reason its attributes must also be defined at class loading. This is unlike the code of a method, where references to names are resolved the moment a class is executed.
Now, since you refer to a field object in your class definition from models and back again, this will not work.
If you place all three in the same package, your problem will be solved.

The was fixed, after much toil, simply by changing
kwargs['to'] = 'Company'
to
kwargs['to'] = 'crm.Company'
It seems that when the 'to' argument was evaluated outside of the crm app, it was evaluating it in the context of the incidents app. That is, it was looking for 'incidents.Company' which, as the error message suggested, didn't exist.