I am creating SQLAlchemy class that represents user credentials.
I want to have field password that stores hashed value of password. Therefore I would like to override its behavior the following way:
When assigned credentials.password = value it actually stores hash of the value
When comparing credentials.password == value it actually compares with hash of the value
I have read the following part of SQLAlchemy documentation http://docs.sqlalchemy.org/en/rel_0_7/orm/mapper_config.html#using-descriptors-and-hybrids
And I think I do understand how to solve the issue number 1.
I am however unsure, how to do second point. Is there a way to do it the safe way (without breaking SQLAlchemy)?
Here is the example model:
class Credentials(Base):
__tablename__ = 'credentials'
id = Column(Integer, primary_key=True)
_password = Column('password', String)
#hybrid_property
def password(self):
return self._password
#password.setter(self):
self._password = hash(self._password)
For comparing, since you can't un-hash the password, you would need to create a custom type for the Column class, that over-rides the eq operator:
class MyPasswordType(String):
class comparator_factory(String.Comparator):
def __eq__(self, other):
return self.operate(operators.eq, hash(other))
Have a look at: http://docs.sqlalchemy.org/en/latest/core/types.html#types-operators
And http://docs.sqlalchemy.org/en/latest/core/types.html#sqlalchemy.types.TypeEngine.comparator_factory
To set you just need to pass in the value:
#password.setter
def password(self, value):
self._password = hash(value)
Related
Backstory
I have a questionnaire that asks sensitive questions most of which are true/false. The majority of the time the values are false which poses a challenge when keeping the data private at rest. When encrypting each question into a separate column, it is really easy to tell which value is true and which is false with a bit of guessing. To combat this, the questions and answers are put into a dictionary object with some salt (nonsense that changes randomly) then encrypted. Making it impossible without the key to know what the answers were.
Method
Below is an example of the model used to encrypt the data with salt at rest making it impossible to look at the data and know the contents.
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy_utils.types import JSONType
from sqlalchemy_utils.types.encrypted.encrypted_type import StringEncryptedType, AesEngine
Base = declarative_base()
class SensitiveQuestionnaire(Base):
user_id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
_data = data: dict = sa.Column(StringEncryptedType(JSONType, 'secret', AesEngine, 'pkcs5'),
nullable=False, default=lambda: {'_salt': salt_shaker()})
# values are viewed using a python property to look into the `_data` dict
#property
def sensitive_question(self) -> Optional[float]:
return self._data.get('sensitive_question')
# values are set into the `_data` dict
#sensitive_question.setter
def sensitive_question(self, value: bool) -> None:
self._data['sensitive_question'] = value
# in a real example there would be 20+ properties that map to questions
def __init__(self, **kwargs):
# Sqlalchemy does not use the __init__ method so we are free to set object defaults here
self._data = {'_salt': salt_shaker()}
for key in kwargs:
setattr(self, key, kwargs[key])
#property
def _salt(self) -> str:
return self._data['_salt']
def salt_shaker():
return ''.join([random.choice('hldjs..' for i in range(50)])
The Problem
After the SensitiveQuestionnaire object is initialized none of the changes are persisted in the database.
# GIVEN a questionnaire
questionnaire = model.SensitiveQuestionnaire(user_id=1)
db.session.add()
db.session.commit()
# WHEN updating the questionnaire and saving it to the database
questionnaire.sensitive_question= True
db.session.commit()
# THEN we get the questionnaire from the database
db_questionnaire = model.SensitiveQuestionnaire.query\
.filter(model.SensitiveQuestionnaire.user_id == 1).first()
# THEN the sensitive_question value is persisted
assert db_questionnaire.sensitive_question is True
Value from the db_questionnaire.sensitive_question is None when it should be True.
After spending the better part of the day to figure this out, the cause of the issue is how Sqlalchemy knows when there is a change. The short version is sqlalchemy uses python's __setitem__ to hook in sqlalchemy's change() method letting it know there was a change. More info can be found in sqlalchemy's docs.
The answer is to wrap the StringEncryptedType in a MultableDict Type
Mutation Tracking
Provide support for tracking of in-place changes to scalar values, which are propagated into ORM change events on owning parent objects.
From SqlAlchemy's docs: https://docs.sqlalchemy.org/en/13/orm/extensions/mutable.html
Solution
Condensed version... wrapping the StringEncryptedType in a MutableDict
_data = data: dict = sa.Column(
MutableDict.as_mutable(StringEncryptedType(JSONType, 'secret', AesEngine, 'pkcs5')),
nullable=False, default=lambda: {'_salt': salt_shaker()})
Full version from the question above
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.mutable import MutableDict
from sqlalchemy_utils.types import JSONType
from sqlalchemy_utils.types.encrypted.encrypted_type import StringEncryptedType, AesEngine
Base = declarative_base()
class SensitiveQuestionnaire(Base):
user_id: int = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
# The MutableDict.as_mutable below is what changed!
_data = data: dict = sa.Column(
MutableDict.as_mutable(StringEncryptedType(JSONType, 'secret', AesEngine, 'pkcs5')),
nullable=False, default=lambda: {'_salt': salt_shaker()})
#property
def sensitive_question(self) -> Optional[float]:
return self._data.get('sensitive_question')
# values are set into the `_data` dict
#sensitive_question.setter
def sensitive_question(self, value: bool) -> None:
self._data['sensitive_question'] = value
# in a real example there would be 20+ properties that map to questions
def __init__(self, **kwargs):
self._data = {'_salt': salt_shaker()}
for key in kwargs:
setattr(self, key, kwargs[key])
#property
def _salt(self) -> str:
return self._data['_salt']
def salt_shaker():
return ''.join([random.choice('hldjs..' for i in range(50)])
I'm using postgres with SQLAlchemy. I want to create Profile objects and have them autogenerate a GUID. However currently my profile IDs don't store any values, e.g:
profile = Profile(name='some_profile')
-> print(profile.name)
some_profile
-> print(profile.id)
None
I've looked into how others are implementing GUIDs into their models (How can I use UUIDs in SQLAlchemy?)
I understand that many people don't recommend using GUIDs as IDs, but I would like to know where I'm going wrong despite this.
Here's my current implementation:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, String
from sqlalchemy.types import TypeDecorator, CHAR
import uuid
Base = declarative_base()
class GUID(TypeDecorator):
"""Platform-independent GUID type.
Uses Postgresql's UUID type, otherwise uses
CHAR(32), storing as stringified hex values.
"""
impl = CHAR
def process_bind_param(self, value, dialect):
if value is None:
return value
elif dialect.name == 'postgresql':
return str(value)
else:
if not isinstance(value, uuid.UUID):
return "%.32x" % uuid.UUID(value).int
else:
# hexstring
return "%.32x" % value.int
def process_result_value(self, value, dialect):
if value is None:
return value
else:
if not isinstance(value, uuid.UUID):
value = uuid.UUID(value)
return value
class Profile(Base):
__tablename__ = 'profile'
id = Column(GUID(), primary_key=True, default=uuid.uuid4)
name = Column(String)
I'm still a beginner to python, but as far as I understand I'm declaring the type of my Profile id column as a GUID (setup by the GUID class). A default GUID value should therefore be successfully stored when generated in that column through uuid.uuid4().
My guess is that there isn't anything wrong with the GUID class, but instead in how I'm trying to generate the default value within the id column.
Any help would be appreciated!
Your code is correct!
After you commit the profile, you can get the valid id.
profile = Profile(name='some_profile')
-> print(profile.name)
some_profile
-> print(profile.id)
None
# commit
session.add(profile)
session.commit()
# print saved profile
-> print(profile.name)
some_profile
-> print(profile.id)
ff36e5ff-16b5-4536-bc86-8ec02a53cfc8
I am using SQLAlchemy through Flask-SQLAlchemy as the ORM for a web app.
I'd like to automatically leading and trailing strip whitespace (e.g. str.strip) when assigning to any string field.
One way to do this would be the following, but it would need to be specified for each and every string field:
class User(db.Model):
_email = db.Column('email', db.String(100), primary_key=True)
#hybrid_property
def email(self): return self._email
#email.setter
def email(self, data): self._email = data.strip()
I would like to do this more generically for every String field (without having to write the above for each).
One way would be to create a custom augmented string type that handles such processing:
from sqlalchemy.types import TypeDecorator
class StrippedString(TypeDecorator):
impl = db.String
def process_bind_param(self, value, dialect):
# In case you have nullable string fields and pass None
return value.strip() if value else value
def copy(self, **kw):
return StrippedString(self.impl.length)
You'd then use this in place of plain String in your models:
class User(db.Model):
email = db.Column(StrippedString(100), primary_key=True)
This does not work exactly the same as your own implementation in that the processing takes place when the value is to be bound to a query as a parameter, or in other words a bit later:
In [12]: u = User(email=' so.much#white.space ')
In [13]: u.email
Out[13]: ' so.much#white.space '
In [14]: session.add(u)
In [15]: session.commit()
In [16]: u.email
Out[16]: 'so.much#white.space'
I'm working on a project using Flask and a PostgreSQL database, with SQLAlchemy.
I have Group objects which have a list of User IDs who are members of the group. For some reason, when I try to add an ID to a group, it will not save properly.
If I try members.append(user_id), it doesn't seem to work at all. However, if I try members += [user_id], the id will show up in the view listing all the groups, but if I restart the server, the added value(s) is (are) not there. The initial values, however, are.
Related code:
Adding group to the database initially:
db = SQLAlchemy(app)
# ...
g = Group(request.form['name'], user_id)
db.session.add(g)
db.session.commit()
The Group class:
from flask.ext.sqlalchemy import SQLAlchemy
from sqlalchemy.dialects.postgresql import ARRAY
class Group(db.Model):
__tablename__ = "groups"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(128))
leader = db.Column(db.Integer)
# list of the members in the group based on user id
members = db.Column(ARRAY(db.Integer))
def __init__(self, name, leader):
self.name = name
self.leader = leader
self.members = [leader]
def __repr__(self):
return "Name: {}, Leader: {}, Members: {}".format(self.name, self.leader, self.members)
def add_user(self, user_id):
self.members += [user_id]
My test function for updating the Group:
def add_2_to_group():
g = Group.query.all()[0]
g.add_user(2)
db.session.commit()
return redirect(url_for('show_groups'))
Thanks for any help!
As you have mentioned, the ARRAY datatype in sqlalchemy is immutable. This means it isn’t possible to add new data into array once it has been initialised.
To solve this, create class MutableList.
from sqlalchemy.ext.mutable import Mutable
class MutableList(Mutable, list):
def append(self, value):
list.append(self, value)
self.changed()
#classmethod
def coerce(cls, key, value):
if not isinstance(value, MutableList):
if isinstance(value, list):
return MutableList(value)
return Mutable.coerce(key, value)
else:
return value
This snippet allows you to extend a list to add mutability to it. So, now you can use the class above to create a mutable array type like:
class Group(db.Model):
...
members = db.Column(MutableList.as_mutable(ARRAY(db.Integer)))
...
You can use the flag_modified function to mark the property as having changed. In this example, you could change your add_user method to:
from sqlalchemy.orm.attributes import flag_modified
# ~~~
def add_user(self, user_id):
self.members += [user_id]
flag_modified(self, 'members')
To anyone in the future: so it turns out that arrays through SQLAlchemy are immutable. So, once they're initialized in the database, they can't change size. There's probably a way to do this, but there are better ways to do what we're trying to do.
This is a hacky solution, but what you can do is:
Store the existing array temporarily
Set the column value to None
Set the column value to the existing temporary array
For example:
g = Group.query.all()[0]
temp_array = g.members
g.members = None
db.session.commit()
db.session.refresh(g)
g.members = temp_array
db.session.commit()
In my case it was solved by using the new reference for storing a object variable and assiging that new created variable in object variable.so, Instead of updating the existing objects variable it will create a new reference address which reflect the changes.
Here in Model,
Table: question
optional_id = sa.Column(sa.ARRAY(sa.Integer), nullable=True)
In views,
option_list=list(question.optional_id if question.optional_id else [])
if option_list:
question.optional_id.clear()
option_list.append(obj.id)
question.optional_id=option_list
else:
question.optional_id=[obj.id]
I have a simple player entity:
__tablename__ = 'player'
_id = Column('id', SmallInteger, primary_key=True)
_nickName = Column('nick_name', String)
def __init__(self, nickName):
self._nickName = nickName
#property
def id(self):
return self._id
#property
def nickName(self):
return self._nickName.decode(encoding='UTF-8')
#nickName.setter
def nickName(self, nickName):
self._nickName = nickName
when i do:
players = session.query(Player).filter(Player.nickName=='foo')
and i print the players var i got this:
SELECT player.id AS player_id, player.nick_name AS player_nick_name
FROM player
WHERE false
Obviously, when I add .first() at the end of the session query, the result is None.
I have tried with filter_by() and get the same result.
Any help is welcome.
While using #hybrid_property will fix this in the general case, you shouldn't need to be decoding manually at all. Just set the column type to Unicode instead of String and, assuming your server plays nice, you should correctly get back a unicode.
You also don't need the id property at all.
So all you should need for this class is:
class Player(Base):
__tablename__ = 'player'
id = Column(SmallInteger, primary_key=True)
nickName = Column(Unicode)
(Both the column names and __init__ arguments can be generated automatically.)
If there's some reason your database isn't handling Unicode correctly, well, that's a different problem that we'd love to help you fix. :)
You cannot use regular #propertys as query parameters. Use a #hybrid_property instead:
from sqlalchemy.ext.hybrid import hybrid_property
#hybrid_property
def nickName(self):
return self._nickName.decode(encoding='UTF-8')
#nickName.setter
def nickName(self, nickName):
self._nickName = nickName
This makes Player.nickName (so on the attribute on the class) useful in SQL expressions.