I'm using SQLAlchemy in project that is not a web application. It is a server application that loads number of different objects from database and modifies them locally, but don't want to save those updates to the database each time a commit is issued. I was previously working with Django ORM for some web projects and found it better suited for what I'm trying to achieve. In Django ORM I could .save() each object whenever I wanted without saving other things I may not want to save. I understand why it works like this in SQLAlchemy, but I wonder how I could do this in the Django-like way?
Update:
To make it easier to understand what I'm trying to achieve, I'll provide you an example.
This is how it works actually:
a = MyModel.query.get(1)
b = MyModel.query.get(1)
a.somefield = 1
b.somefield = 2
# this will save both of changed models
session.commit()
This is how I want it to work:
a = MyModel.query.get(1)
b = MyModel.query.get(1)
a.somefield = 1
b.somefield = 2
a.save()
# I didn't want to save b, changes of b weren't committed
I want to have greater control of what is actually saved. I want to save changes of each object every 5 minute or so.
I use something like:
class BaseModel(object):
def save(self, commit=True):
# this part can be optimized.
try:
db.session.add(self)
except FlushError:
# In case of an update operation.
pass
if commit:
db.session.commit()
def delete(self, commit=True):
db.session.delete(self)
if commit:
db.session.commit()
and then I define my models as:
class User(db.Model, BaseModel)
So, now I can do:
u = User(username='foo', password='bar')
u.save()
This is what you were planning to achieve ?
I am not sure i understand your predicament.
In Django,
foo = MyModel(field1='value1', field2='value2')
foo.save()
or alternatively
foo = MyModel.objects.create(field1='value1', field2='value2')
In SQLAlchemy,
foo = MyModel(field1='value1', field2='value2')
session.add(foo)
At this point you have only added the object to the session and it has not yet committed the transaction. You need to commit only after you have done whatever changes were required
session.commit()
take a look that this link. I think it will make the transition from Django ORM to SqlAlchemy easier.
UPDATE
For such a situation, you could use multiple sessions.
engine = create_engine("postgresql+psycopg2://user:password#localhost/test")
metadata = MetaData(bind=engine)
Session = sessionmaker(bind=engine)
session1 = Session()
session2 = Session()
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
age = Column(Integer)
def __init__(self, name, age):
self.name = name
self.age = age
def __repr__(self):
return "<User('%s','%s')>" % (self.name, self.age)
Base.metadata.create_all(engine)
Created a table 'users' in the 'test' db. Also, 2 session objects, session1 and session2, have been initialized.
a = User('foo','10')
b = User('bar', '20')
session1.add(a)
session1.add(b)
session1.commit()
The table users will now have 2 records
1: foo, 10
2: bar, 20
Fetching the 'foo' record sing session1 and 'bar' using session2.
foo = session1.query(User).filter(User.name == "foo").first()
bar = session2.query(User).filter(User.name == "bar").first()
Making changes to the 2 records
foo.age = 11
bar.age = 21
Now, if you want the changes of foo alone to carry over,
session1.commit()
and for bar,
session2.commit()
Not to stir up an old post, but
You say:
I want to save changes of each object every 5 minute or so.
So why not use a scheduler like Celery.(I use pyramid_celery)
With this you can save each object every 5 minutes, i.e. You can add a decorator:
#periodic_task(run_every=crontab(minute="*/5")
def somefunction():
#your code here
This works great, especially when you need to update your database to make sure it is up to date(in the case that there is many users using your system)
Hope this helps someone with the, saving every 5 minutes part.
Related
I am probably asking an obvious thing, but I could not find the answer.
I am learning Pony ORM, and started with creating simple database from the documentation. I made a design of a simple database diagram and code https://editor.ponyorm.com/user/nidza11/Ponz#
from pony.orm import *
db = Database()
class Person(db.Entity):
id = PrimaryKey(int, auto=True)
name = Required(str)
age = Required(int)
cars = Set('Car', cascade_delete=True)
class Car(db.Entity):
id = PrimaryKey(int, auto=True)
make = Required(str)
model = Required(str)
person = Required(Person)
db.bind("sqlite", "TestDB.sqlite", create_db = True)
db.generate_mapping(create_tables=True)
with db_session:
p1 = Person(name="John", age=24)
p2 = Person(name="Ive", age=26)
p3 = Person(name="Olivia", age = 26)
c1 = Car(make="Toyota", model = "Prius", person = p2)
p3 = Car(make="Ford", model = "Explorer", person = p3)
After running the code, database was created and populated.
I would like to make a name attribute in Person table unique. So I made a change in code
name = Required(str, unique=True)
After running the code again database was populated, but the name field is not set unique. Data was duplicated. According to DB browser name field was 'unique'. I expected an error. Such error will help me keep entity and database synchronized.
Adding a new attribute only in Person table class, but not in database, will cause an error.
Is it possible to make a change only in one place inside the code, that will reflect in database as well?
If I have to make a same change on two places, how to do it nice and clean?
I've just asked on the Pony ORM 'Telegram' group and this is the transaction that occurred there.
Let's say I have a User model with attributes id, name, email and a relationship languages.
Is it possible to create a User instance from existing data that behaves like I would have queried it with dbsession.query(User).get(42)?
What I mean in particular is that I want that an access to user.languages creates a subquery and populates the attribute.
Here a code example:
I have a class User:
class User(Base):
id = Column(Integer, primary_key=True)
name = Column(String(64))
email = Column(String(64))
languages = relationship('Language', secondary='user_languages')
I already have a lot of users stored in my DB.
And I know that I have, for example, this user in my DB:
user_dict = {
'id': 23,
'name': 'foo',
'email': 'foo#bar',
}
So I have all the attributes but the relations.
Now I want to make a sqlalchemy User instance
and kind of register it in sqlalchemy's system
so I can get the languages if needed.
user = User(**user_dict)
# Now I can access the id, name email attributes
assert user.id == 23
# but since sqlalchemy thinks it's a new model it doesn't
# lazy load any relationships
assert len(user.languages) == 0
# I want here that the languages for the user with id 23 appear
# So I want that `user` is the same as when I would have done
user_from_db = DBSession.query(User).get(23)
assert user == user_from_db
The use-case is that I have a big model with lots of complex
relationships but 90% of the time I don't need the data from those.
So I only want to cache the direct attributes plus what else I need
and then load those from the cache like above and be able to
use the sqlalchemy model like I would have queried it from the db.
From the sqlalchemy mailing list:
# to make it look like it was freshly loaded from the db
from sqlalchemy.orm.session import make_transient_to_detached
make_transient_to_detached(user)
# merge instance in session without emitting sql
user = DBSession.merge(user, load=False)
This answer was extracted from the question
I met an transaction problem when I used the python orm peewee these days. I save two book instances using this orm, and beween the two savings I raise an exception so I except that none of them are saved to database, but it doesn't work. Could anyone explain why? I am new to python, thanks.
this code is below:
from peewee import *
def get_db():
return SqliteDatabase("test.db")
class Book(Model):
id = PrimaryKeyField()
name = CharField()
class Meta:
database = get_db()
def test_transaction():
book1 = Book(name="book1")
book2 = Book(name="book2")
db = get_db()
db.create_tables([Book], safe=True)
try:
with db.transaction() as tran:
book1.save()
raise ProgrammingError("test")
book2.save()
except:
pass
for book in Book.select():
print(book.name)
if __name__ == '__main__':
test_transaction()
The problem is that when you are calling "get_db()" you are instantiating new database objects. Databases are stateful, in that they manage the active connection for a given thread. So what you've essentially got is two different databases, one that your models are associated with, and one that has your connection and transaction. When you call db.transaction() a transaction is taking place, but not on the connection you think it is.
Change the code to read as follows and it will work like you expect.
book1 = Book(name='book1')
book2 = Book(name='book2')
db = Book._meta.database
# ...
We are making a game server using SQLAlchemy.
because game servers must be very fast, we have decided to separate databases depending on user ID(integer).
so for example I did it successfully like the following.
from threading import Thread
from sqlalchemy import Column, Integer, String, DateTime, create_engine
from sqlalchemy.ext.declarative import declarative_base, DeferredReflection
from sqlalchemy.orm import sessionmaker
DeferredBase = declarative_base(cls=DeferredReflection)
class BuddyModel(DeferredBase):
__tablename__ = 'test_x'
id = Column(Integer(), primary_key=True, autoincrement=True)
value = Column(String(50), nullable=False)
and the next code will create multiple databases.
There will be test1 ~ test10 databases.
for i in range(10):
url = 'mysql://user#localhost/'
engine = create_engine(url, encoding='UTF-8', pool_recycle=300)
con = engine.connect()
con.execute('create database test%d' % i)
the following code will create 10 separate engines.
the get_engine() function will give you an engine depending on the user ID.
(User ID is integer)
engines = []
for i in range(10):
url = 'mysql://user#localhost/test%d'% i
engine = create_engine(url, encoding='UTF-8', pool_recycle=300)
DeferredBase.metadata.bind = engine
DeferredBase.metadata.create_all()
engines.append(engine)
def get_engine(user_id):
index = user_id%10
return engines[index]
by running prepare function, the BuddyModel class will be prepared, and mapped to the engine.
def prepare(user_id):
engine = get_engine(user_id)
DeferredBase.prepare(engine)
** The next code will do what I want to do exactly **
for user_id in range(100):
prepare(user_id)
engine = get_engine(user_id)
session = sessionmaker(engine)()
buddy = BuddyModel()
buddy.value = 'user_id: %d' % user_id
session.add(buddy)
session.commit()
But the problem is that when I do it in multiple threads, it just raise errors
class MetalMultidatabaseThread(Thread):
def run(self):
for user_id in range(100):
prepare(user_id)
engine = get_engine(user_id)
session = sessionmaker(engine)()
buddy = BuddyModel()
buddy.value = 'user_id: %d' % user_id
session.add(buddy)
session.commit()
threads = []
for i in range(100):
t = MetalMultidatabaseThread()
t.setDaemon(True)
t.start()
threads.append(t)
for t in threads:
t.join()
the error message is ...
ArgumentError: Class '<class '__main__.BuddyModel'>' already has a primary mapper defined. Use non_primary=True to create a non primary Mapper. clear_mappers() will remove *all* current mappers from all classes.
so.. my question is that How CAN I DO MULTIPLE-DATABASE like the above architecture using SQLAlchemy?
this is called horizontal sharding and is a bit of a tricky use case. The version you have, make a session based on getting the engine first, will work fine. There are two variants of this which you may like.
One is to use the horizontal sharding extension. This extension allows you to create a Session to automatically select the correct node.
The other is more or less what you have, but less verbose. Build a Session class that has a routing function, so you at least could share a single session and say, session.using_bind('engine1') for a query instead of making a whole new session.
I have found an answer for my question.
For building up multiple-databases depending on USER ID (integer) just use session.
Before explain this, I want to expound on the database architecture more.
For example if the user ID 114 connects to the server, the server will determine where to retrieve the user's information by using something like this.
user_id%10 # <-- 4th database
Architecture
DATABASES
- DB0 <-- save all user data whose ID ends with 0
- DB1 <-- save all user data whose ID ends with 1
.
.
.
- DB8 <-- save all user data whose ID ends with 9
Here is the answer
First do not use bind parameter.. simply make it empty.
Base = declarative_base()
Declare Model..
class BuddyModel(Base):
__tablename__ = 'test_x'
id = Column(Integer(), primary_key=True, autoincrement=True)
value = Column(String(50), nullable=False)
When you want to do CRUD ,make a session
engine = get_engine_by_user_id(user_id)
session = sessionmaker(bind=engine)()
buddy = BuddyModel()
buddy.value = 'This is Sparta!! %d' % user_id
session.add(buddy)
session.commit()
engine should be the one matched with the user ID.
I'm a complete beginner to Flask and I'm starting to play around with making web apps.
I have a hard figuring out how to enforce unique user names. I'm thinking about how to do this in SQL, maybe with something like user_name text unique on conflict fail, but then how to I catch the error back in Python?
Alternatively, is there a way to manage this that's built in to Flask?
That entirely depends on your database layer. Flask is very specifically not bundled with a specific ORM system, though SQL Alchemy is recommended. The good news is that SQL Alchemy has a unique constraint.
Here's how it might work:
from sqlalchemy.ext.declarative import declarative_base, InvalidRequestError
engine = #my engine
session = Session() # created by sessionmaker(bind=engine)
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String, unique=True)
# then later...
user = User()
user.name = 'Frank'
session.add(user)
try:
session.commit()
print 'welcome to the club Frank'
except InvalidRequestError:
print 'You are not Frank. Impostor!!!'
Run the part after "then later" twice. The first time you'll get a welcome message, the second time you won't.
Addendum: The closest thing that Flask has to a default authentication framework simply stores users in a dict by username. The way to check to enforce uniqueness is by manually testing eg.
if username in digest_db:
raise Exception('HEY! "{}" already exists! \
You can\'t do that'.format(username))
digest_db.add_user(username, password)
or overriding RealmDigestDB to make sure that it checks before adding:
class FlaskRealmDigestDB(authdigest.RealmDigestDB):
def add_user(self, user, password):
if user in self:
raise AttributeError('HEY! "{}" already exists! \
You can\'t do that'.format(user))
super(FlaskRealmDigestDB, self).add_user(user, password)
def requires_auth(self, f):
# yada yada
or overriding RealmDigestDB, and making it return something which does not allow duplicate assignment. eg.
class ClosedDict(dict):
def __setitem__(self, name, val):
if name in self and val != self[name]:
raise AttributeError('Cannot reassign {} to {}'.format(name, val))
super(ClosedDict, self).__setitem__(name,val)
class FlaskRealmDigestDB(authdigest.RealmDigestDB):
def newDB():
return ClosedDict()
def requires_auth(self, f):
# yada yada
I put this here as an addendum because that class does not persist data in any way, if you're planning on extending authdigest.RealmDigestDB anyway you should use something like SQLAlchemy as above.
You can use SQLAlchemy.It's a plug-in