I am using the following code to retrieve variables from a database that Python uses to run an automated machine. I set the variables through a PHP driven web interface. Python reads the variables and acts according to instructions.
However, during calibration of the machine, we are forced to restart python to accept any variable changes. Python isn't my first language and neither is it the first language of my colleagues. It would obviously save a lot of time if we didn't have to restart python to accept variable changes.
Our variable list class is constructed like the following;
class VariableList():
connectdb = DbConnector(host='localhost', user='a', password='b', database='c')
result = connectdb.selectDb('variablelist','varA,varB')
for row in result:
# INPUTS
varA = row[1]
varB = row[2]
What is the Pythonic way to get around this issue? Getters/Setters? #property? An example to follow would very much appreciated...
easy one. Python implementation goes like this:
class VariableList():
def __init__(self):
self.db_con = DbConnector(host='localhost', user='a', password='b', database='c')
#property
def varA(self):
return self.db_con.selectDb('variablelist','varA')
#property.setter
def varA(self, value):
self.db_con.updateDb('variablelist', value)
and also you can refactor your model with SQLAlchemy framework.
for example
from sqlalchemy (
create_engine,
Column,
Integer,
String,
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
engine = create_engine(DB_DNS,
pool_size=DB_POOL_SIZE,
max_overflow=DB_MAX_OVERFLOW,
pool_recycle=DB_POOL_RECYCLE,
isolation_level="READ UNCOMMITTED", # attention, the last one is important!
)
Session = sessionmaker(bind=engine,
autocommit=False,
expire_on_commit=False)
class MyTable(Base):
__tablename__ = MY_TABLE_NAME
id = Column(Integer, primary=True)
name = Column(String(32), nullable=True, default='', doc='user_name')
# query something
result = Session().query(MyTable).filter(CONDITION).all()
Related
I have to tie database and programming for an assignment and I have an idea for a code but need to make sure that I can use the tables I created in mySQL as my classes or objects in Python.
Example: I use SQL to create a database of houses with specific addresses and zip codes. A client says they live in zipcode x. My program should then parse through the database and return all addresses within zipcode x. Then ideally create a table in SQL with the clients results.
Not the exact assignment but it gets the basic idea across.
You're looking for an ORM. See SQLAlchemy. Example:
from sqlalchemy import Column, String, Integer, Sequence
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
create_session = sessionmaker()
Base = declarative_base()
person_autoincr_seq = Sequence('person_autoincr_seq')
class Person(Base):
__tablename__ = "person"
id = Column(
Integer,
person_autoincr_seq,
server_default=person_autoincr_seq.next_value(),
nullable = False,
primary_key = True
)
name = Column(
String,
nullable = False
)
def __init__(self, name,id=None):
if id is not None:
self.id = id
self.name = name
Using the db:
import logging as log
from contextlib import closing
engine = sqlalchemy.engine.create_engine(
"postgresql://testuser:mypassword#127.0.0.1:5432/testdb"
)
create_session.configure(bind=engine)
try:
with closing(create_session()) as db_session:
name = db_session.query(Person.name).filter_by(id=5).one()[0]
except Exception:
log.exception("Something wrong while querying db")
I have a Pyramid application that does CRUD with SQLAlchemy via pyramid_basemodel. All seems to work nicely.
I then pip installed SQLAlchemy-Continuum, to provide history for certain objects. All I did to configure it was make the following alterations to my models.py file:
import sqlalchemy as sa
from sqlalchemy import (event, Column, Index, Integer, Text, String, Date, DateTime, \
Float, ForeignKey, Table, Boolean,)
from sqlalchemy.orm import (relationship, backref, mapper, scoped_session, sessionmaker,)
from pyramid_basemodel import Base, BaseMixin, Session, save
from pyramid_fullauth.models import User
from sqlalchemy_continuum import make_versioned
from colanderalchemy import setup_schema
from zope.sqlalchemy import ZopeTransactionExtension
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
event.listen(mapper, 'mapper_configured', setup_schema)
# Continuum setup
make_versioned()
# FOR EACH VERSIONED MODEL I ADD __versioned__ = {} at the start of each model def. Eg:
class Thing(Base):
__versioned__ = {}
__tablename__ = 'thing'
id = sa.Column(Integer, primary_key=True)
related_id = sa.Column(Integer, ForeignKey('OtherThing.id'))
other_thing = sa.orm.relationship("OtherThing", backref="thing")
description = sa.Column(String(length=100))
a_date = sa.Column(Date)
some_hours = sa.Column(Integer)
b_date = sa.Column(Date)
more_hours = sa.Column(Integer)
sa.orm.configure_mappers()
(Sorry for the slightly redundant imports; I decided to totally follow the Continuum example and import sqlalchemy as sa, and switch to using that notation in the models that I versioned. I may also be doing stupid, monkey-see monkey-do stuff based on a half-understanding of different tutorials.)
This setup allowed me to run alembic revision --autogenerate and produce ModelHistory tables in the database, but when I go to some of the pages that read the now-versioned models, they give the error
sqlalchemy.exc.UnboundExecutionError: This session is not bound to a single Engine or Connection, and no context was provided to locate a binding.
For some reason it reads one model added in the same way, but then trying to update it fails with the same error.
My guess is that I need to configure whatever Continuum uses for a SQLAlchemy session to point to the existing one configured in Pyramid, but I'm not sure. Am I getting warm?
You're generating a session when you call:
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
but not binding it to an engine. Your pyramid template, pyramid_basemodel, is already generating a session for you and binding it to the engine.
Try removing the DBSession and using Session imported from pyramid_basemodel.
FWIW, if anyone these days is looking how to make SQLAlchemy-Continuum work with Pyramid, here's how you do it:
Assuming you have followed the official Pyramid tutorial is the following:
install SQLAlchemy-Continuum
add make_versioned(user_cls=None) to the top of models/__init__.py
add __versioned__ = {} to MyModel class in models/mymodel.py
And that's it!
I've created a repo that has all the needed bits in place: https://github.com/zupo/tutorial/tree/exploration/sqlalchemy-continuum
We are making a game server using SQLAlchemy.
because game servers must be very fast, we have decided to separate databases depending on user ID(integer).
so for example I did it successfully like the following.
from threading import Thread
from sqlalchemy import Column, Integer, String, DateTime, create_engine
from sqlalchemy.ext.declarative import declarative_base, DeferredReflection
from sqlalchemy.orm import sessionmaker
DeferredBase = declarative_base(cls=DeferredReflection)
class BuddyModel(DeferredBase):
__tablename__ = 'test_x'
id = Column(Integer(), primary_key=True, autoincrement=True)
value = Column(String(50), nullable=False)
and the next code will create multiple databases.
There will be test1 ~ test10 databases.
for i in range(10):
url = 'mysql://user#localhost/'
engine = create_engine(url, encoding='UTF-8', pool_recycle=300)
con = engine.connect()
con.execute('create database test%d' % i)
the following code will create 10 separate engines.
the get_engine() function will give you an engine depending on the user ID.
(User ID is integer)
engines = []
for i in range(10):
url = 'mysql://user#localhost/test%d'% i
engine = create_engine(url, encoding='UTF-8', pool_recycle=300)
DeferredBase.metadata.bind = engine
DeferredBase.metadata.create_all()
engines.append(engine)
def get_engine(user_id):
index = user_id%10
return engines[index]
by running prepare function, the BuddyModel class will be prepared, and mapped to the engine.
def prepare(user_id):
engine = get_engine(user_id)
DeferredBase.prepare(engine)
** The next code will do what I want to do exactly **
for user_id in range(100):
prepare(user_id)
engine = get_engine(user_id)
session = sessionmaker(engine)()
buddy = BuddyModel()
buddy.value = 'user_id: %d' % user_id
session.add(buddy)
session.commit()
But the problem is that when I do it in multiple threads, it just raise errors
class MetalMultidatabaseThread(Thread):
def run(self):
for user_id in range(100):
prepare(user_id)
engine = get_engine(user_id)
session = sessionmaker(engine)()
buddy = BuddyModel()
buddy.value = 'user_id: %d' % user_id
session.add(buddy)
session.commit()
threads = []
for i in range(100):
t = MetalMultidatabaseThread()
t.setDaemon(True)
t.start()
threads.append(t)
for t in threads:
t.join()
the error message is ...
ArgumentError: Class '<class '__main__.BuddyModel'>' already has a primary mapper defined. Use non_primary=True to create a non primary Mapper. clear_mappers() will remove *all* current mappers from all classes.
so.. my question is that How CAN I DO MULTIPLE-DATABASE like the above architecture using SQLAlchemy?
this is called horizontal sharding and is a bit of a tricky use case. The version you have, make a session based on getting the engine first, will work fine. There are two variants of this which you may like.
One is to use the horizontal sharding extension. This extension allows you to create a Session to automatically select the correct node.
The other is more or less what you have, but less verbose. Build a Session class that has a routing function, so you at least could share a single session and say, session.using_bind('engine1') for a query instead of making a whole new session.
I have found an answer for my question.
For building up multiple-databases depending on USER ID (integer) just use session.
Before explain this, I want to expound on the database architecture more.
For example if the user ID 114 connects to the server, the server will determine where to retrieve the user's information by using something like this.
user_id%10 # <-- 4th database
Architecture
DATABASES
- DB0 <-- save all user data whose ID ends with 0
- DB1 <-- save all user data whose ID ends with 1
.
.
.
- DB8 <-- save all user data whose ID ends with 9
Here is the answer
First do not use bind parameter.. simply make it empty.
Base = declarative_base()
Declare Model..
class BuddyModel(Base):
__tablename__ = 'test_x'
id = Column(Integer(), primary_key=True, autoincrement=True)
value = Column(String(50), nullable=False)
When you want to do CRUD ,make a session
engine = get_engine_by_user_id(user_id)
session = sessionmaker(bind=engine)()
buddy = BuddyModel()
buddy.value = 'This is Sparta!! %d' % user_id
session.add(buddy)
session.commit()
engine should be the one matched with the user ID.
I've got a case where most of the time the relationships between objects was such that pre-configuring an eager (joined) load on the relationship made sense. However now I've got a situation where I really don't want the eager load to be done.
Should I be removing the joined load from the relationship and changing all relevant queries to join at the query location (ick), or is there some way to suppress an eager load in a query once it is set up?
Below is an example where eager loading has been set up on the User->Address relationship. Can the query at the end of the program be configured to NOT eager load?
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
import sqlalchemy.orm as orm
##Set up SQLAlchemy for declarative use with Sqlite...
engine = sa.create_engine("sqlite://", echo = True)
DeclarativeBase = declarative_base()
Session = orm.sessionmaker(bind = engine)
class User(DeclarativeBase):
__tablename__ = "users"
id = sa.Column(sa.Integer, primary_key = True, autoincrement = True)
name = sa.Column(sa.String, unique = True)
addresses = orm.relationship("Address",
lazy = "joined", #EAGER LOAD CONFIG IS HERE
)
def __init__(self, Name):
self.name = Name
class Address(DeclarativeBase):
__tablename__ = "addresses"
id = sa.Column(sa.Integer, primary_key = True, autoincrement = True)
address = sa.Column(sa.String, unique = True)
FK_user = sa.Column(sa.Integer, sa.ForeignKey("users.id"))
def __init__(self, Email):
self.address = Email
##Generate data tables...
DeclarativeBase.metadata.create_all(engine)
##Add some data...
joe = User("Joe")
joe.addresses = [Address("joe#example.com"),
Address("joeyjojojs#example.net")]
s1 = Session()
s1.add(joe)
s1.commit()
## Access the data for the demo...
s2 = Session()
#How to suppress the eager load (auto-join) in the query below?
joe = s2.query(User).filter_by(name = "Joe").one() # <-- HERE?
for addr in joe.addresses:
print addr.address
You may override eagerness of properties on query-by-query basis, as far as I remember. Will this work?
from sqlalchemy.orm import lazyload
joe = (s2.query(User)
.options(lazyload('addresses'))
.filter_by(name = "Joe").one())
for addr in joe.addresses:
print addr.address
See the docs.
You can use Query.options(raiseload('*')) or Query.enable_eagerloads(False).
Query.enable_eagerloads(False) will disable all eager loading on the query. That is, even if you put a joinedload() or something, it won't be executed.
Query.options(raiseload('*')) will install a raiseload loader on every column, making sure they're not lazily loaded: an exception is raised instead. Note that this mode is fine for development and testing environments, but may be destructive in production. Make it optional like this:
Query.options(raiseload('*') if development else defaultload([]))
also note that raiseload('*') only works for top-level relationships. It won't spread on joined entities! If you request a relationship, you have to specify it twice:
session.query(User).options(
load_only('id'),
joinedload(User.addresses).options(
load_only('id'),
raiseload('*')
),
raiseload('*')
)
also, raiseload('*') only works for relationships, not columns :)
For columns, use defer(..., raiseload=True)
I would like to use autoload to use an existings database. I know how to do it without declarative syntax (model/_init_.py):
def init_model(engine):
"""Call me before using any of the tables or classes in the model"""
t_events = Table('events', Base.metadata, schema='events', autoload=True, autoload_with=engine)
orm.mapper(Event, t_events)
Session.configure(bind=engine)
class Event(object):
pass
This works fine, but I would like to use declarative syntax:
class Event(Base):
__tablename__ = 'events'
__table_args__ = {'schema': 'events', 'autoload': True}
Unfortunately, this way I get:
sqlalchemy.exc.UnboundExecutionError: No engine is bound to this Table's MetaData. Pass an engine to the Table via autoload_with=<someengine>, or associate the MetaData with an engine via metadata.bind=<someengine>
The problem here is that I don't know where to get the engine from (to use it in autoload_with) at the stage of importing the model (it's available in init_model()). I tried adding
meta.Base.metadata.bind(engine)
to environment.py but it doesn't work. Anyone has found some elegant solution?
OK, I think I figured it out. The solution is to declare the model objects outside the model/__init__.py. I concluded that __init__.py gets imported as the first file when importing something from a module (in this case model) and this causes problems because the model objects are declared before init_model() is called.
To avoid this I created a new file in the model module, e.g. objects.py. I then declared all my model objects (like Event) in this file.
Then, I can import my models like this:
from PRJ.model.objects import Event
Furthermore, to avoid specifying autoload-with for each table, I added this line at the end of init_model():
Base.metadata.bind = engine
This way I can declare my model objects with no boilerplate code, like this:
class Event(Base):
__tablename__ = 'events'
__table_args__ = {'schema': 'events', 'autoload': True}
event_identifiers = relationship(EventIdentifier)
def __repr__(self):
return "<Event(%s)>" % self.id
I just tried this using orm module.
Base = declarative_base(bind=engine)
Base.metadata.reflect(bind=engine)
Accessing tables manually or through loop or whatever:
Base.metadata.sorted_tables
Might be useful.
from sqlalchemy import MetaData,create_engine,Table
engine = create_engine('postgresql://postgres:********#localhost/db_name')
metadata = MetaData(bind=engine)
rivers = Table('rivers',metadata,autoload=True,auto_load_with=engine)
from sqlalchemy import select
s = select([rivers]).limit(5)
engine.execute(s).fetchall()
worked for me. I was getting the error because of not specifying bind when creating MetaData() object.
Check out the Using SQLAlchemy with Pylons tutorial on how to bind metadata to the engine in the init_model function.
If the meta.Base.metadata.bind(engine) statement successfully binds your model metadata to the engine, you should be able to perform this initialization in your own init_model function. I guess you didn't mean to skip the metadata binding in this function, did you?