I'm using Flask and flask-SQLAlchemy extension to make a webapp and I've added fields like unique=True in the model declarations.
class Coupon(db.Model):
username = db.Column(db.String(80), primary_key=True)
value = db.Column(db.String(80), unique=True)
is_valid = db.Column(db.Boolean)
def __init__(self, value, username):
self.value = value
self.username = username
self.is_valid = True
What is the best(pythonic) way of recovering with a failed validation. For example -
c1 = Coupon("same_value", "foo")
db.session.add(c1)
c2 = Coupon("same_value", "bar")
db.session.add(c2)
db.session.commit() #gives an IntegrityError
Should I use a try, except block to deal with this? Or is there a better way to deal with transcations that have not succeeded. One additional query regarding transactions - As long as keep adding objects to a session and not do a session.commit is it all a part of one transcation?
Thanks
I am not sure if it makes sense to try-catch the commit() call. Even if you do, how do you proceed further? How do you further locate the problem and fix it? You might have dozens of duplicates already.
The motto here is: CATCH ERRORS EARLY.
Therefore, what I would probably do is: I would add a call to session.flush() just after the session.add(c?). This will indicate the problem early enough for me to handle properly. So it could make more sense to wrap the add/flush in the try-catch block and handle as appropriate:
def _add_coupon(c):
""" #param c: instance of Coupon.
#return: True if success, False on failure.
"""
try:
session.add(c)
session.flush()
return True
except sqlalchemy.exc.IntegrityError as err:
# #todo: handle as appropriate: return existing instance [session.query(Coupon).filter(Coupon.value==c.value).one()] or re-raise
logger.error("Tried to add a duplicate entry for the Coupon value [%s]. Aborting", c.value)
return False
Related
Problem: I get a ValidationError when trying to perform a .save() when appending a value to an EmbeddedDocumentListField because I am missing required fields that already exist on the document.
Note that at this point the User document has already been created as part of the signup process, so it already has an email and password in the DB.
My classes:
class User(gj.Document):
email = db.EmailField(required=True, unique=True)
password = db.StringField(required=True)
long_list_of_thing_1s = db.EmbeddedDocumentListField("Thing1")
long_list_of_thing_2s = db.EmbeddedDocumentListField("Thing2")
class Thing1(gj.EmbeddedDocument):
some_string = db.StringField()
class Thing2(gj.EmbeddedDocument):
some_string = db.StringField()
Trying to append a new EmbeddedDocument to the EmbeddedDocumentListField in my User class in the Thing2 Resource endpoint:
class Thing2(Resource):
def post(self):
try:
body = request.get_json()
user_id = body["user_id"]
user = UserModel.objects.only("long_list_of_thing_2s").get(id=user_id)
some_string = body["some_string"]
new_thing_2 = Thing2Model()
new_thing_2.some_string = some_string
user.long_list_of_thing_2s.append(new_thing_2)
user.save()
return 201
except Exception as exception:
raise InternalServerError
On hitting this endpoint I get the following error on the user.save()
mongoengine.errors.ValidationError: ValidationError (User:603e39e7097f3e9a6829f422) (Field is required: ['email', 'password'])
I think this is because of the .only("long_list_of_thing_2s")
But I am specifically using UserModel.objects.only("long_list_of_thing_2s") because I don't want to be inefficient in bringing the entire UserModel into memory when I only want to append something the long_list_of_thing_2s
Is there a different way I should be going about this? I am relatively new to Flask and Mongoengine so I am not sure what all the best practices are when going about this process.
You are correct, this is due to the .only and is a known "bug" in MongoEngine.
Unless your Model is really large, using .only() will not make a big difference so I'd recommend to use it only if you observe performance issues.
If you do have to keep the .only() for whatever reason, you should be able to make use of the push atomic operator. An advantage of using the push operator is that in case of race conditions (concurrent requests), it will gracefully deal with the different updates, this is not the case with regular .save() which will overwrite the list.
How can I write a context manager that, in test/dev, asserts that there are no database queries in its block? The goal here is to force usage of select_related/prefetch related, and make it an error to forget to do so. Eg, given these models:
class SomeSingleThing(models.Model):
somefield = models.IntegerField()
someotherfield = models.CharField(max_length=42)
class SomeManyThing(models.Model):
parent = models.ForeignKey(SomeSingleThing)
This should raise an exception, because select_related is missing, and so a query gets performed inside the forbidden zone:
obj = SomeSingleThing.objects.get(pk=1)
with forbid_queries():
val = obj.parent.somefield
# => exception
And this should not, because we selected everything we needed:
obj = SomeSingleThing.objects.select_related('parent').get(pk=1)
with forbid_queries():
val = obj.parent.somefield
# no exception
I am implementing a field in django model which calls the following function for default value
def get_default_value():
a = MyModel.objects.aggregate(max_id=Max('id'))
return get_unique_value_based_on_num(a['max_id'] or 0)
class MyModel:
default_value_field = CharField(default=get_default_value)
Although I may be wrong on this, I fear this implementation may result in race condition.
Is there a better way to do this ? May be use F object or something else ?
To avoid race conditions, it is best letting the database handle the integrity of your table, which is one of the things databases are made for.
To do so, catch any IntegrityError raised by saving your model instance and try again with a different value when it fails.
from django.db import IntegrityError, models, transaction
def get_default_value():
a = MyModel.objects.aggregate(max_id=Max('id'))
return get_unique_value_based_on_num(a['max_id'] or 0)
class MyModel(models.Model):
# Have unicity enforced at database level with unique=True.
default_value_field = models.CharField(max_length=200, unique=True)
def save(self):
if not self.default_value_field:
max_tries = 100 # Choose a sensible value!
for i in range(max_tries):
try:
self.default_value_field = get_default_value()
# Atomic block to rollback transaction in case of IntegrityError.
with transaction.atomic():
super(MyModel, self).save()
break
except IntegrityError:
# default_value_field is not unique, try again with a new value.
continue
else:
# Max tries reached, raise.
raise IntegrityError('Could not save model because etc...')
else:
super(MyModel, self).save(*args, **kwargs)
Seems like django has select_for_update for queryset API. It may be the solution to my problem.
https://docs.djangoproject.com/en/dev/ref/models/querysets/#select-for-update
I would like a model to generate automatically a random alphanumeric string as its primary key when I create a new instance of it.
example:
from django.db import models
class MyTemporaryObject(models.Model):
id = AutoGenStringField(lenght=16, primary_key=True)
some_filed = ...
some_other_field = ...
in my mind the key should look something like this "Ay3kJaBdGfcadZdao03293". It's for very temporary use. In case of collision I would like it Django to try a new key.
I was wondering if there was already something out there, or a very simple solution I am not seeing (I am fairly new to python and Django). Otherwise I was thinking to do my own version of models.AutoField, would that be the right approach?
I have already found how to generate the key here, so it's not about the string generation. I would just like to have it work seamlessly with a simple Django service without adding too much complexity to the code.
EDIT:
Possible solution? What do you think?
id = models.CharField(unique=True, primary_key=True, default=StringKeyGenerator(), editable=False)
with
class StringKeyGenerator(object):
def __init__(self, len=16):
self.lenght = len
def __call__(self):
return ''.join(random.choice(string.letters + string.digits) for x in range(self.lenght))
I came up with it after going through the Django documentation one more time.
One of the simplest way to generate unique strings in python is to use uuid module. If you want to get alphanumeric output, you can simply use base64 encoding as well:
import uuid
import base64
uuid = base64.b64encode(uuid.uuid4().bytes).replace('=', '')
# sample value: 1Ctu77qhTaSSh5soJBJifg
You can then put this code in the model's save method or define a custom model field using it.
Here's how I would do it without making the field a primary key:
from django.db import IntegrityError
class MyTemporaryObject(models.Model):
auto_pseudoid = models.CharField(max_length=16, blank=True, editable=False, unique=True)
# add index=True if you plan to look objects up by it
# blank=True is so you can validate objects before saving - the save method will ensure that it gets a value
# other fields as desired
def save(self, *args, **kwargs):
if not self.auto_pseudoid:
self.auto_pseudoid = generate_random_alphanumeric(16)
# using your function as above or anything else
success = False
failures = 0
while not success:
try:
super(MyTemporaryObject, self).save(*args, **kwargs)
except IntegrityError:
failures += 1
if failures > 5: # or some other arbitrary cutoff point at which things are clearly wrong
raise
else:
# looks like a collision, try another random value
self.auto_pseudoid = generate_random_alphanumeric(16)
else:
success = True
Two problems that this avoids, compared to using the field as the primary key are that:
1) Django's built in relationship fields require integer keys
2) Django uses the presence of the primary key in the database as a sign that save should update an existing record rather than insert a new one. This means if you do get a collision in your primary key field, it'll silently overwrite whatever else used to be in the row.
Try this:
The if statement below is to make sure that the model is update able.
Without the if statement you'll update the id field everytime you resave the model, hence creating a new model everytime
from uuid import uuid4
from django.db import IntegrityError
class Book(models.Model):
id = models.CharField(primary_key=True, max_length=32)
def save(self, *args, **kwargs):
if self.id:
super(Book, self).save(*args, **kwargs)
return
unique = False
while not unique:
try:
self.id = uuid4().hex
super(Book, self).save(*args, **kwargs)
except IntegrityError:
self.id = uuid4().hex
else:
unique = True
The code snippet below uses the secrets library that comes with Python, handles id collisions, and continues to pass integrity errors when there isn't an id collision.
example of the ids 0TCKybG1qgAhRuEN , yJariA4QN42E9aLf , AZOMrzlkJ-RKh4dp
import secrets
from django.db import models, IntegrityError
class Test(models.Model):
pk = models.CharField(primary_key=True, max_length=32)
def save(self, *args, **kwargs):
unique = False
while not unique:
try:
self.pk = secrets.token_urlsafe(12)
super(Test, self).save(*args, **kwargs)
except IntegrityError as e :
# keep raising the exception if it's not id collision error
if not 'pk' in e.args[0]:
unique = True
raise e
else:
unique = True
I'm a complete beginner to Flask and I'm starting to play around with making web apps.
I have a hard figuring out how to enforce unique user names. I'm thinking about how to do this in SQL, maybe with something like user_name text unique on conflict fail, but then how to I catch the error back in Python?
Alternatively, is there a way to manage this that's built in to Flask?
That entirely depends on your database layer. Flask is very specifically not bundled with a specific ORM system, though SQL Alchemy is recommended. The good news is that SQL Alchemy has a unique constraint.
Here's how it might work:
from sqlalchemy.ext.declarative import declarative_base, InvalidRequestError
engine = #my engine
session = Session() # created by sessionmaker(bind=engine)
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String, unique=True)
# then later...
user = User()
user.name = 'Frank'
session.add(user)
try:
session.commit()
print 'welcome to the club Frank'
except InvalidRequestError:
print 'You are not Frank. Impostor!!!'
Run the part after "then later" twice. The first time you'll get a welcome message, the second time you won't.
Addendum: The closest thing that Flask has to a default authentication framework simply stores users in a dict by username. The way to check to enforce uniqueness is by manually testing eg.
if username in digest_db:
raise Exception('HEY! "{}" already exists! \
You can\'t do that'.format(username))
digest_db.add_user(username, password)
or overriding RealmDigestDB to make sure that it checks before adding:
class FlaskRealmDigestDB(authdigest.RealmDigestDB):
def add_user(self, user, password):
if user in self:
raise AttributeError('HEY! "{}" already exists! \
You can\'t do that'.format(user))
super(FlaskRealmDigestDB, self).add_user(user, password)
def requires_auth(self, f):
# yada yada
or overriding RealmDigestDB, and making it return something which does not allow duplicate assignment. eg.
class ClosedDict(dict):
def __setitem__(self, name, val):
if name in self and val != self[name]:
raise AttributeError('Cannot reassign {} to {}'.format(name, val))
super(ClosedDict, self).__setitem__(name,val)
class FlaskRealmDigestDB(authdigest.RealmDigestDB):
def newDB():
return ClosedDict()
def requires_auth(self, f):
# yada yada
I put this here as an addendum because that class does not persist data in any way, if you're planning on extending authdigest.RealmDigestDB anyway you should use something like SQLAlchemy as above.
You can use SQLAlchemy.It's a plug-in