Related
Problem: I get a ValidationError when trying to perform a .save() when appending a value to an EmbeddedDocumentListField because I am missing required fields that already exist on the document.
Note that at this point the User document has already been created as part of the signup process, so it already has an email and password in the DB.
My classes:
class User(gj.Document):
email = db.EmailField(required=True, unique=True)
password = db.StringField(required=True)
long_list_of_thing_1s = db.EmbeddedDocumentListField("Thing1")
long_list_of_thing_2s = db.EmbeddedDocumentListField("Thing2")
class Thing1(gj.EmbeddedDocument):
some_string = db.StringField()
class Thing2(gj.EmbeddedDocument):
some_string = db.StringField()
Trying to append a new EmbeddedDocument to the EmbeddedDocumentListField in my User class in the Thing2 Resource endpoint:
class Thing2(Resource):
def post(self):
try:
body = request.get_json()
user_id = body["user_id"]
user = UserModel.objects.only("long_list_of_thing_2s").get(id=user_id)
some_string = body["some_string"]
new_thing_2 = Thing2Model()
new_thing_2.some_string = some_string
user.long_list_of_thing_2s.append(new_thing_2)
user.save()
return 201
except Exception as exception:
raise InternalServerError
On hitting this endpoint I get the following error on the user.save()
mongoengine.errors.ValidationError: ValidationError (User:603e39e7097f3e9a6829f422) (Field is required: ['email', 'password'])
I think this is because of the .only("long_list_of_thing_2s")
But I am specifically using UserModel.objects.only("long_list_of_thing_2s") because I don't want to be inefficient in bringing the entire UserModel into memory when I only want to append something the long_list_of_thing_2s
Is there a different way I should be going about this? I am relatively new to Flask and Mongoengine so I am not sure what all the best practices are when going about this process.
You are correct, this is due to the .only and is a known "bug" in MongoEngine.
Unless your Model is really large, using .only() will not make a big difference so I'd recommend to use it only if you observe performance issues.
If you do have to keep the .only() for whatever reason, you should be able to make use of the push atomic operator. An advantage of using the push operator is that in case of race conditions (concurrent requests), it will gracefully deal with the different updates, this is not the case with regular .save() which will overwrite the list.
Django has various numeric fields available for use in models, e.g. DecimalField and PositiveIntegerField. Although the former can be restricted to the number of decimal places stored and the overall number of characters stored, is there any way to restrict it to storing only numbers within a certain range, e.g. 0.0-5.0 ?
Failing that, is there any way to restrict a PositiveIntegerField to only store, for instance, numbers up to 50?
Update: now that Bug 6845 has been closed, this StackOverflow question may be moot. - sampablokuper
You can use Django's built-in validators—
from django.db.models import IntegerField, Model
from django.core.validators import MaxValueValidator, MinValueValidator
class CoolModelBro(Model):
limited_integer_field = IntegerField(
default=1,
validators=[
MaxValueValidator(100),
MinValueValidator(1)
]
)
Edit: When working directly with the model, make sure to call the model full_clean method before saving the model in order to trigger the validators. This is not required when using ModelForm since the forms will do that automatically.
You could also create a custom model field type - see http://docs.djangoproject.com/en/dev/howto/custom-model-fields/#howto-custom-model-fields
In this case, you could 'inherit' from the built-in IntegerField and override its validation logic.
The more I think about this, I realize how useful this would be for many Django apps. Perhaps a IntegerRangeField type could be submitted as a patch for the Django devs to consider adding to trunk.
This is working for me:
from django.db import models
class IntegerRangeField(models.IntegerField):
def __init__(self, verbose_name=None, name=None, min_value=None, max_value=None, **kwargs):
self.min_value, self.max_value = min_value, max_value
models.IntegerField.__init__(self, verbose_name, name, **kwargs)
def formfield(self, **kwargs):
defaults = {'min_value': self.min_value, 'max_value':self.max_value}
defaults.update(kwargs)
return super(IntegerRangeField, self).formfield(**defaults)
Then in your model class, you would use it like this (field being the module where you put the above code):
size = fields.IntegerRangeField(min_value=1, max_value=50)
OR for a range of negative and positive (like an oscillator range):
size = fields.IntegerRangeField(min_value=-100, max_value=100)
What would be really cool is if it could be called with the range operator like this:
size = fields.IntegerRangeField(range(1, 50))
But, that would require a lot more code since since you can specify a 'skip' parameter - range(1, 50, 2) - Interesting idea though...
from django.db import models
from django.core.validators import MinValueValidator, MaxValueValidator
size = models.IntegerField(validators=[MinValueValidator(0),
MaxValueValidator(5)])
I had this very same problem; here was my solution:
SCORE_CHOICES = zip( range(1,n), range(1,n) )
score = models.IntegerField(choices=SCORE_CHOICES, blank=True)
There are two ways to do this. One is to use form validation to never let any number over 50 be entered by a user. Form validation docs.
If there is no user involved in the process, or you're not using a form to enter data, then you'll have to override the model's save method to throw an exception or limit the data going into the field.
Here is the best solution if you want some extra flexibility and don't want to change your model field. Just add this custom validator:
#Imports
from django.core.exceptions import ValidationError
class validate_range_or_null(object):
compare = lambda self, a, b, c: a > c or a < b
clean = lambda self, x: x
message = ('Ensure this value is between %(limit_min)s and %(limit_max)s (it is %(show_value)s).')
code = 'limit_value'
def __init__(self, limit_min, limit_max):
self.limit_min = limit_min
self.limit_max = limit_max
def __call__(self, value):
cleaned = self.clean(value)
params = {'limit_min': self.limit_min, 'limit_max': self.limit_max, 'show_value': cleaned}
if value: # make it optional, remove it to make required, or make required on the model
if self.compare(cleaned, self.limit_min, self.limit_max):
raise ValidationError(self.message, code=self.code, params=params)
And it can be used as such:
class YourModel(models.Model):
....
no_dependents = models.PositiveSmallIntegerField("How many dependants?", blank=True, null=True, default=0, validators=[validate_range_or_null(1,100)])
The two parameters are max and min, and it allows nulls. You can customize the validator if you like by getting rid of the marked if statement or change your field to be blank=False, null=False in the model. That will of course require a migration.
Note: I had to add the validator because Django does not validate the range on PositiveSmallIntegerField, instead it creates a smallint (in postgres) for this field and you get a DB error if the numeric specified is out of range.
Hope this helps :) More on Validators in Django.
PS. I based my answer on BaseValidator in django.core.validators, but everything is different except for the code.
In the forms.py
Class FloatForm(forms.ModelForm):
class Meta:
model = Float
fields = ('name','country', 'city', 'point', 'year')
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields['point'] = forms.FloatField(max_value=100, min_value=1)
It is worth mentioning that sometimes Django validation doesn't work as Django validation is mostly an application-level validation, not validation at the database level. Also, Model validation is not run automatically on the save/create/update of the model. If you want to validate your values instantly in your code then you need to do it manually — using the override save() method:
class UserRating():
SCORE_CHOICES = (
(1, _("Terrible")),
(2, _("Poor")),
(3, _("Average")),
(4, _("Very Good")),
(5, _("Excellent")),
)
score = models.PositiveSmallIntegerField(
choices=SCORE_CHOICES, default=1,
validators=[
MaxValueValidator(5),
MinValueValidator(1)
]
)
def save(self, *args, **kwargs):
if int(self.score) < 1 or int(self.score) > 5:
raise ValidationError('Score must be located between 0 to 5')
super(UserRating, self).save(*args, **kwargs)
...
Add validator like this your model column in models.py
class Planogram(models.Model):
camera = models.ForeignKey(Camera, on_delete=models.CASCADE)
xtl = models.DecimalField(decimal_places=10, max_digits=11,validators=[MaxValueValidator(1),MinValueValidator(0)])
if you are using create function to create objects change it to constructor like below....
and call fullclean() on that object and then save..
everything will work perfectly.
planogram = Planogram(camera_id = camera,xtl=xtl,ytl=ytl,xbr=xbr,ybr=ybr,product_id=product_id)
planogram.full_clean()
planogram.save()
I would like a model to generate automatically a random alphanumeric string as its primary key when I create a new instance of it.
example:
from django.db import models
class MyTemporaryObject(models.Model):
id = AutoGenStringField(lenght=16, primary_key=True)
some_filed = ...
some_other_field = ...
in my mind the key should look something like this "Ay3kJaBdGfcadZdao03293". It's for very temporary use. In case of collision I would like it Django to try a new key.
I was wondering if there was already something out there, or a very simple solution I am not seeing (I am fairly new to python and Django). Otherwise I was thinking to do my own version of models.AutoField, would that be the right approach?
I have already found how to generate the key here, so it's not about the string generation. I would just like to have it work seamlessly with a simple Django service without adding too much complexity to the code.
EDIT:
Possible solution? What do you think?
id = models.CharField(unique=True, primary_key=True, default=StringKeyGenerator(), editable=False)
with
class StringKeyGenerator(object):
def __init__(self, len=16):
self.lenght = len
def __call__(self):
return ''.join(random.choice(string.letters + string.digits) for x in range(self.lenght))
I came up with it after going through the Django documentation one more time.
One of the simplest way to generate unique strings in python is to use uuid module. If you want to get alphanumeric output, you can simply use base64 encoding as well:
import uuid
import base64
uuid = base64.b64encode(uuid.uuid4().bytes).replace('=', '')
# sample value: 1Ctu77qhTaSSh5soJBJifg
You can then put this code in the model's save method or define a custom model field using it.
Here's how I would do it without making the field a primary key:
from django.db import IntegrityError
class MyTemporaryObject(models.Model):
auto_pseudoid = models.CharField(max_length=16, blank=True, editable=False, unique=True)
# add index=True if you plan to look objects up by it
# blank=True is so you can validate objects before saving - the save method will ensure that it gets a value
# other fields as desired
def save(self, *args, **kwargs):
if not self.auto_pseudoid:
self.auto_pseudoid = generate_random_alphanumeric(16)
# using your function as above or anything else
success = False
failures = 0
while not success:
try:
super(MyTemporaryObject, self).save(*args, **kwargs)
except IntegrityError:
failures += 1
if failures > 5: # or some other arbitrary cutoff point at which things are clearly wrong
raise
else:
# looks like a collision, try another random value
self.auto_pseudoid = generate_random_alphanumeric(16)
else:
success = True
Two problems that this avoids, compared to using the field as the primary key are that:
1) Django's built in relationship fields require integer keys
2) Django uses the presence of the primary key in the database as a sign that save should update an existing record rather than insert a new one. This means if you do get a collision in your primary key field, it'll silently overwrite whatever else used to be in the row.
Try this:
The if statement below is to make sure that the model is update able.
Without the if statement you'll update the id field everytime you resave the model, hence creating a new model everytime
from uuid import uuid4
from django.db import IntegrityError
class Book(models.Model):
id = models.CharField(primary_key=True, max_length=32)
def save(self, *args, **kwargs):
if self.id:
super(Book, self).save(*args, **kwargs)
return
unique = False
while not unique:
try:
self.id = uuid4().hex
super(Book, self).save(*args, **kwargs)
except IntegrityError:
self.id = uuid4().hex
else:
unique = True
The code snippet below uses the secrets library that comes with Python, handles id collisions, and continues to pass integrity errors when there isn't an id collision.
example of the ids 0TCKybG1qgAhRuEN , yJariA4QN42E9aLf , AZOMrzlkJ-RKh4dp
import secrets
from django.db import models, IntegrityError
class Test(models.Model):
pk = models.CharField(primary_key=True, max_length=32)
def save(self, *args, **kwargs):
unique = False
while not unique:
try:
self.pk = secrets.token_urlsafe(12)
super(Test, self).save(*args, **kwargs)
except IntegrityError as e :
# keep raising the exception if it's not id collision error
if not 'pk' in e.args[0]:
unique = True
raise e
else:
unique = True
I'm using Flask and flask-SQLAlchemy extension to make a webapp and I've added fields like unique=True in the model declarations.
class Coupon(db.Model):
username = db.Column(db.String(80), primary_key=True)
value = db.Column(db.String(80), unique=True)
is_valid = db.Column(db.Boolean)
def __init__(self, value, username):
self.value = value
self.username = username
self.is_valid = True
What is the best(pythonic) way of recovering with a failed validation. For example -
c1 = Coupon("same_value", "foo")
db.session.add(c1)
c2 = Coupon("same_value", "bar")
db.session.add(c2)
db.session.commit() #gives an IntegrityError
Should I use a try, except block to deal with this? Or is there a better way to deal with transcations that have not succeeded. One additional query regarding transactions - As long as keep adding objects to a session and not do a session.commit is it all a part of one transcation?
Thanks
I am not sure if it makes sense to try-catch the commit() call. Even if you do, how do you proceed further? How do you further locate the problem and fix it? You might have dozens of duplicates already.
The motto here is: CATCH ERRORS EARLY.
Therefore, what I would probably do is: I would add a call to session.flush() just after the session.add(c?). This will indicate the problem early enough for me to handle properly. So it could make more sense to wrap the add/flush in the try-catch block and handle as appropriate:
def _add_coupon(c):
""" #param c: instance of Coupon.
#return: True if success, False on failure.
"""
try:
session.add(c)
session.flush()
return True
except sqlalchemy.exc.IntegrityError as err:
# #todo: handle as appropriate: return existing instance [session.query(Coupon).filter(Coupon.value==c.value).one()] or re-raise
logger.error("Tried to add a duplicate entry for the Coupon value [%s]. Aborting", c.value)
return False
Django has various numeric fields available for use in models, e.g. DecimalField and PositiveIntegerField. Although the former can be restricted to the number of decimal places stored and the overall number of characters stored, is there any way to restrict it to storing only numbers within a certain range, e.g. 0.0-5.0 ?
Failing that, is there any way to restrict a PositiveIntegerField to only store, for instance, numbers up to 50?
Update: now that Bug 6845 has been closed, this StackOverflow question may be moot. - sampablokuper
You can use Django's built-in validators—
from django.db.models import IntegerField, Model
from django.core.validators import MaxValueValidator, MinValueValidator
class CoolModelBro(Model):
limited_integer_field = IntegerField(
default=1,
validators=[
MaxValueValidator(100),
MinValueValidator(1)
]
)
Edit: When working directly with the model, make sure to call the model full_clean method before saving the model in order to trigger the validators. This is not required when using ModelForm since the forms will do that automatically.
You could also create a custom model field type - see http://docs.djangoproject.com/en/dev/howto/custom-model-fields/#howto-custom-model-fields
In this case, you could 'inherit' from the built-in IntegerField and override its validation logic.
The more I think about this, I realize how useful this would be for many Django apps. Perhaps a IntegerRangeField type could be submitted as a patch for the Django devs to consider adding to trunk.
This is working for me:
from django.db import models
class IntegerRangeField(models.IntegerField):
def __init__(self, verbose_name=None, name=None, min_value=None, max_value=None, **kwargs):
self.min_value, self.max_value = min_value, max_value
models.IntegerField.__init__(self, verbose_name, name, **kwargs)
def formfield(self, **kwargs):
defaults = {'min_value': self.min_value, 'max_value':self.max_value}
defaults.update(kwargs)
return super(IntegerRangeField, self).formfield(**defaults)
Then in your model class, you would use it like this (field being the module where you put the above code):
size = fields.IntegerRangeField(min_value=1, max_value=50)
OR for a range of negative and positive (like an oscillator range):
size = fields.IntegerRangeField(min_value=-100, max_value=100)
What would be really cool is if it could be called with the range operator like this:
size = fields.IntegerRangeField(range(1, 50))
But, that would require a lot more code since since you can specify a 'skip' parameter - range(1, 50, 2) - Interesting idea though...
from django.db import models
from django.core.validators import MinValueValidator, MaxValueValidator
size = models.IntegerField(validators=[MinValueValidator(0),
MaxValueValidator(5)])
I had this very same problem; here was my solution:
SCORE_CHOICES = zip( range(1,n), range(1,n) )
score = models.IntegerField(choices=SCORE_CHOICES, blank=True)
There are two ways to do this. One is to use form validation to never let any number over 50 be entered by a user. Form validation docs.
If there is no user involved in the process, or you're not using a form to enter data, then you'll have to override the model's save method to throw an exception or limit the data going into the field.
Here is the best solution if you want some extra flexibility and don't want to change your model field. Just add this custom validator:
#Imports
from django.core.exceptions import ValidationError
class validate_range_or_null(object):
compare = lambda self, a, b, c: a > c or a < b
clean = lambda self, x: x
message = ('Ensure this value is between %(limit_min)s and %(limit_max)s (it is %(show_value)s).')
code = 'limit_value'
def __init__(self, limit_min, limit_max):
self.limit_min = limit_min
self.limit_max = limit_max
def __call__(self, value):
cleaned = self.clean(value)
params = {'limit_min': self.limit_min, 'limit_max': self.limit_max, 'show_value': cleaned}
if value: # make it optional, remove it to make required, or make required on the model
if self.compare(cleaned, self.limit_min, self.limit_max):
raise ValidationError(self.message, code=self.code, params=params)
And it can be used as such:
class YourModel(models.Model):
....
no_dependents = models.PositiveSmallIntegerField("How many dependants?", blank=True, null=True, default=0, validators=[validate_range_or_null(1,100)])
The two parameters are max and min, and it allows nulls. You can customize the validator if you like by getting rid of the marked if statement or change your field to be blank=False, null=False in the model. That will of course require a migration.
Note: I had to add the validator because Django does not validate the range on PositiveSmallIntegerField, instead it creates a smallint (in postgres) for this field and you get a DB error if the numeric specified is out of range.
Hope this helps :) More on Validators in Django.
PS. I based my answer on BaseValidator in django.core.validators, but everything is different except for the code.
In the forms.py
Class FloatForm(forms.ModelForm):
class Meta:
model = Float
fields = ('name','country', 'city', 'point', 'year')
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields['point'] = forms.FloatField(max_value=100, min_value=1)
It is worth mentioning that sometimes Django validation doesn't work as Django validation is mostly an application-level validation, not validation at the database level. Also, Model validation is not run automatically on the save/create/update of the model. If you want to validate your values instantly in your code then you need to do it manually — using the override save() method:
class UserRating():
SCORE_CHOICES = (
(1, _("Terrible")),
(2, _("Poor")),
(3, _("Average")),
(4, _("Very Good")),
(5, _("Excellent")),
)
score = models.PositiveSmallIntegerField(
choices=SCORE_CHOICES, default=1,
validators=[
MaxValueValidator(5),
MinValueValidator(1)
]
)
def save(self, *args, **kwargs):
if int(self.score) < 1 or int(self.score) > 5:
raise ValidationError('Score must be located between 0 to 5')
super(UserRating, self).save(*args, **kwargs)
...
Add validator like this your model column in models.py
class Planogram(models.Model):
camera = models.ForeignKey(Camera, on_delete=models.CASCADE)
xtl = models.DecimalField(decimal_places=10, max_digits=11,validators=[MaxValueValidator(1),MinValueValidator(0)])
if you are using create function to create objects change it to constructor like below....
and call fullclean() on that object and then save..
everything will work perfectly.
planogram = Planogram(camera_id = camera,xtl=xtl,ytl=ytl,xbr=xbr,ybr=ybr,product_id=product_id)
planogram.full_clean()
planogram.save()