Develop Reference

What is the best way to create a dynamic selection list from database column with Python and Flask WFForms?

Spent several hours today trying to get the unique values in my database into a dropdown selection with Flask WTForms.
The Database is MySQL, and I'm actually using SQLAlchemy for the majority of this app, but I've been unable to get unique values into the dropdown with SQLAlchemy (only seem to manage to get the string representaion in or the query itself)
other options use QuerySelectField, but this is from WTForms.ext which will be depreciated from WTForms3 so I'd rather avoid it.
The solution I have seems clunky and I feel there should be a better, more pythonic way.
My current solution is with the mysql.connector
cur = cnx.cursor()
cur.execute("SELECT markets.id AS markets_id, markets.market AS markets_market FROM markets")
r = cur.fetchall()
markets = [] # this gets passed into the SelectField() choices
for el in r:
markets.append(el[1])
#create the forms to create and update Competitor model
class CompetitorForm(FlaskForm):
competitor_name = StringField('Competitor Name', validators=[DataRequired()])
country = SelectField('Active Market(s)', validators=[DataRequired()],
choices=markets)
headquarters = StringField('HQ Country', validators=[DataRequired()])
submit = SubmitField('Add Competitor')
While this works and I can get a unique dropdown (the query itself isn't unique, but this table is just a list of countries, no duplicates), I don't feel like this is very scalable as my forms get more complex. Also the way in which I've setup the models, (Market, Competitor and then a lookup table), I need to map this back to a market ID and add to the lookup table (code for this isn't shown here as I'm only concerned with the selection dropdown for now, but i thought the context would help understand the models:
### create the competitor model ###
class Competitor(db.Model):
__tablename__ = "competitors"
id = db.Column(db.Integer,primary_key=True)
competitor_name = db.Column(db.String(64))
headquarters = db.Column(db.String(64))
active_markets = db.relationship('CompMarketLK',
backref = db.backref('active_market_id',lazy='joined'),
lazy='dynamic')
#initialise the model
def __init__(self,competitor_name,country,products,
workflow_id, headquarters,comp_type,employees):
...
def __repr__(self):
return f"Company Name: {self.competitor_name}"
### create the market model object ###
class Market(db.Model):
__tablename__ = 'markets'
id = db.Column(db.Integer,primary_key=True)
market = db.Column(db.String(64),nullable=False)
#define relationships
active_competitors = db.relationship('CompMarketLK',
backref = db.backref('competitor_id',lazy='joined'),
lazy='dynamic')
#construct model object
def __init__(self,id,market):
...
def __repr__(self):
return f"Market: {self.market}"
### create the competitor <--> market lookup table ###
class CompMarketLK(db.Model):
__tablename__ = 'comp_market_lk'
market_id = db.Column(db.Integer,
ForeignKey('markets.id'),
primary_key=True)
comp_id = db.Column(db.Integer,
ForeignKey('competitors.id'),
primary_key=True)
#construct model object
def __init__(self,market_id,comp_id):
...

this answer has most of the groundwork, with this I used the following to create the distinct list:
form.country.choices = [country.market for country in db.session.query(
Market.market).distinct()]
this gave:
def create():
form = CompetitorForm()
form.country.choices = [country.market for country in db.session.query(Market.market).distinct()]
if form.validate_on_submit():
competitor = Competitor(competitor_name=form.competitor_name.data,
country = form.country.data,
headquarters = form.headquarters.data,
)
#add to the db
db.session.add(competitor)
db.session.commit()
return redirect(url_for('core.index'))
return render_template('add_competitor.html',form=form)

SQLAlchemy's "post_update" behaves differently with objects that have been expunged from a session

I'm trying to copy rows from one DB instance to a another DB with an identical schema in a different environment. Two tables within this schema are linked in such a way that they result in mutually dependent rows. When these rows are inserted, the post_update runs afterward as expected, but the update statement sets the value of the ID field to None instead of the expected ID.
This only happens when using objects that have been expunged from a session. When using newly created objects, the post_update behaves exactly as expected.
Examples
I have a relationship set up that looks like this:
class Category(Base):
__tablename__ = 'categories'
id = Column(Integer, primary_key=True)
top_product_id = Column(Integer, ForeignKey('products.id'))
products = relationship('Product', primaryjoin='Product.category_id == Category.id', back_populates='category', cascade='all', lazy='selectin')
top_product = relationship('Product', primaryjoin='Category.top_product_id == Product.id', post_update=True, cascade='all', lazy='selectin')
class Product(Base):
__tablename__ = 'products'
id = Column(Integer, primary_key=True)
category_id = Column(Integer, ForeignKey('categories.id'))
category = relationship('Category', primaryjoin='Product.category_id == Category.id', back_populates='products', cascade='all', lazy='selectin')
If I query a category and its related products from one DB and try to write them to another, the update of top_product_id doesn't behave as expected, and sets the value to None instead. The following code:
category = source_session.query(Category).filter(Category.id == 99).one()
source_session.expunge(category)
make_transient(category)
for products in category.products:
make_transient(product)
# this step is necessary to prevent a foreign key error on the initial category insert
category.top_product_id = None
dest_session.add(category)
results in SQLAlchemy generating the following SQL:
INSERT INTO categories (name, top_product_id) VALUES (%s, %s)
('SomeCategoryName', None)
INSERT INTO products (name, category_id) VALUES (%s, %s)
('SomeProductName', 99)
UPDATE categories SET top_product_id=%s WHERE categories.id = %s
(None, 99)
But if I use newly created objects, everything works as expected.
category = Category()
product = Product()
category.name = 'SomeCategoryName'
product.name = 'SomeProductName'
product.category = category
category.top_product = product
dest_session.add(category)
results in:
INSERT INTO categories (name, top_product_id) VALUES (%s, %s)
('SomeCategoryName', None)
INSERT INTO products (name, category_id) VALUES (%s, %s)
('SomeProductName', 99)
UPDATE categories SET top_product_id=%s WHERE categories.id = %s
(1, 99)
Aside from this difference, everything behaves in the same way between these two actions. All other relationship are created properly, IDs and foreign keys are set as expected. Only the top_product_id set in the update clause created by the post_update fails to behave as expected.
As an additional troubleshooting step, I tried:
Creating new objects
Adding them to a session
Flushing the session to the DB
Expunging the objects from the session
Unseting the foreign key ID fields on the objects (to avoid initial insert error) and making the objects transient
Re-adding the objects to the session
Re-flushing to the DB
On the first flush to the DB, the top_product_id is set properly. On the second, it's set to None. So this confirms that the issue is not with differences in the sessions, but something to do with expunging objects from sessions and making them transient. There must be something that does/doesn't happen during the expunge/make transient process that leaves these objects in a fundamentally different state and prevents post_update from behaving the way it should.
Any ideas on where to go from here would be appreciated.

I assume your Base class mixes in the name column?
Your goal is to make inspect(category).committed_state look like it does for newly created objects (except maybe for id attribute). Same for each product object.
In your "newly created objects" example, category's committed_state looks like this before flushing the session:
{'id': symbol('NEVER_SET'),
'name': symbol('NO_VALUE'),
'products': [],
'top_product': symbol('NEVER_SET')}
while product's committed_state looks like this:
{'category': symbol('NEVER_SET'),
'id': symbol('NEVER_SET'),
'name': symbol('NO_VALUE')}
To get the post-update behavior, you need to both expire category.top_product_id (to prevent it from being included in the INSERT) and fudge category.top_product's committed_state (to make SQLAlchemy believe that the value has changed and therefore needs to cause an UPDATE).
First, expire category.top_product_id before making category transient:
source_session.expire(category, ["top_product_id"])
Then fudge category.top_product's committed_state (this can happen before or after making category transient):
from sqlalchemy import inspect
from sqlalchemy.orm.base import NEVER_SET
inspect(category).committed_state.update(top_product=NEVER_SET)
Full example:
from sqlalchemy import Column, ForeignKey, Integer, String, create_engine, inspect
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session, make_transient, relationship
from sqlalchemy.orm.base import NEVER_SET
class Base(object):
name = Column(String(50), nullable=False)
Base = declarative_base(cls=Base)
class Category(Base):
__tablename__ = 'categories'
id = Column(Integer, primary_key=True)
top_product_id = Column(Integer, ForeignKey('products.id'))
products = relationship('Product', primaryjoin='Product.category_id == Category.id', back_populates='category', cascade='all', lazy='selectin')
top_product = relationship('Product', primaryjoin='Category.top_product_id == Product.id', post_update=True, cascade='all', lazy='selectin')
class Product(Base):
__tablename__ = 'products'
id = Column(Integer, primary_key=True)
category_id = Column(Integer, ForeignKey('categories.id'), nullable=False)
category = relationship('Category', primaryjoin='Product.category_id == Category.id', back_populates='products', cascade='all', lazy='selectin')
source_engine = create_engine('sqlite:///')
dest_engine = create_engine('sqlite:///', echo=True)
def fk_pragma_on_connect(dbapi_con, con_record):
dbapi_con.execute('pragma foreign_keys=ON')
from sqlalchemy import event
for engine in [source_engine, dest_engine]:
event.listen(engine, 'connect', fk_pragma_on_connect)
Base.metadata.create_all(bind=source_engine)
Base.metadata.create_all(bind=dest_engine)
source_session = Session(bind=source_engine)
dest_session = Session(bind=dest_engine)
source_category = Category(id=99, name='SomeCategoryName')
source_product = Product(category=source_category, id=100, name='SomeProductName')
source_category.top_product = source_product
source_session.add(source_category)
source_session.commit()
source_session.close()
# If you want to test UPSERTs in dest_session.
# dest_category = Category(id=99, name='PrevCategoryName')
# dest_product = Product(category=dest_category, id=100, name='PrevProductName')
# dest_category.top_product = dest_product
# dest_session.add(dest_category)
# dest_session.commit()
# dest_session.close()
category = source_session.query(Category).filter(Category.id == 99).one()
# Ensure relationship attributes are initialized before we make objects transient.
_ = category.top_product
# source_session.expire(category, ['id']) # only if you want new IDs in dest_session
source_session.expire(category, ['top_product_id'])
for product in category.products:
# Ensure relationship attributes are initialized before we make objects transient.
_ = product.category
# source_session.expire(product, ['id']) # only if you want new IDs in dest_session
# Not strictly needed as long as Product.category is not a post-update relationship.
source_session.expire(product, ['category_id'])
make_transient(category)
inspect(category).committed_state.update(top_product=NEVER_SET)
for product in category.products:
make_transient(product)
# Not strictly needed as long as Product.category is not a post-update relationship.
inspect(product).committed_state.update(category=NEVER_SET)
dest_session.add(category)
# Or, if you want UPSERT (must retain original IDs in this case)
# dest_session.merge(category)
dest_session.flush()
Which produces this DML in dest_session:
INSERT INTO categories (name, id, top_product_id) VALUES (?, ?, ?)
('SomeCategoryName', 99, None)
INSERT INTO products (name, id, category_id) VALUES (?, ?, ?)
('SomeProductName', 100, 99)
UPDATE categories SET top_product_id=? WHERE categories.id = ?
(100, 99)
It seems like make_transient should reset committed_state to be as if it were a new object, but I guess not.

Static table in peewee

I want to store an enum in the database, according to this.
So let's say, I have a Gender enum and a Person model. I want to do a select like Person.select().where(Person.gender == Gender.MALE)
This can be achieved by creating a GenderField in Person as described here. But the gender won't be in the database as a table, and I want the Person to have foreign keys to the Gender table.
So how can I store static Gender data in the database, then query the Person table by the enum values?

You can, as part of the table creation process, populate the gender table:
class Gender(Model):
label = CharField(primary_key=True)
class Meta:
database = db
def create_schema():
# Use an execution context to ensure the connection is closed when
# we're finished.
with db.execution_context():
db.create_tables([Gender, ...], True)
if not Gender.select().exists():
defaults = ['MALE', 'FEMALE', 'UNSPECIFIED']
Gender.insert_many([{'label': label} for label in defaults]).execute()
The only reason to make Gender a table in this example, though, would be if you plan to dynamically add new Gender labels at run-time. If the set of values is fixed, my opinion is you're better off doing something like this:
class Person(Model):
GENDER_MALE = 'M'
GENDER_FEMALE = 'F'
GENDER_UNSPECIFIED = 'U'
name = CharField()
gender = CharField()
# Then, simply:
Person.create(name='Huey', gender=Person.GENDER_MALE)
Person.create(name='Zaizee', gender=Person.GENDER_UNSPECIFIED)

sqlalchemy upsert between tables with relationship

Question?
What is the most elegant way to insert if new and update if it exists between tables with a relationship in the same postgres DB using sqlalchemy?
Context
Users review a stage table; can accept changes that then get sent to the production table after review.
The stage table may have new records or just edited records compared to the production table.
I figured out how to do this from in peices from other questions, but all of them involve making a new object and specifying the column names e.g.
rutable.ru=rustage.ru
...ect....
My model
class rutable(db.Model):
id = db.Column(db.Integer, primary_key=True)
ru =db.Column(db.String(6), db.ForeignKey('rustage.ru'),unique=True)
class rustage(db.Model):
id = db.Column(db.Integer, primary_key=True)
ru =db.Column(db.String(6), index=True, unique=True)
rutablelink = db.relationship('rutable', lazy='dynamic', backref='ruback')
Pseudo code
#check to see if records exists
if db.session.query(exists().where(models.rutable.id == rurow)).scalar():
rustage=models.rustage.query.filter_by(id=rurow).first()
ruprod=models.rutable.query.filter_by(id=rurow).first()
form = rutable(obj=rustage)
# import pdb;pdb.set_trace()
form.populate_obj(ruprod)
else:
rustage=models.rustage.query.filter_by(id=rurow).first()
#how do i insert rustage into rutable and reset primary key?
#without creating a new obj in which i have to specify each column name?
#e.g. rutable(a=rustage.a,..... 1 million column names (i could use a form?
db.session.commit()
UPDATE
i decided to go with a form
here is what i came up with
rustage=models.rustage.query.filter_by(id=rurow).first()
if db.session.query(exists().where(models.rutable.id == rurow)).scalar():
ruprod=models.rutable.query.filter_by(id=rurow).first()
form = rutable(obj=rustage)
form.populate_obj(ruprod)
else:
ruprod=rutable(ru=rustage.ru)
form = rutable(obj=rustage)
form.populate_obj(ruprod)
rustage.reviewEdit=False
ruprod.reviewEdit=False
db.session.commit()

Put an empty field in peewee union select

I'm try to select two table that have some common field. in raw MySQL query, i can write this:
SELECT t1.id, t1.username, t1.date FROM table1 as 't1' UNION SELECT t2.id, "const_txt", t2.date FROM table2 as 't2'
In that query ,the username field is not in table2 and I set const_txt instead.
So, in peewee, i want to union two table that have the same above situation.
class PeeweeBaseModel(Model):
class Meta:
database = my_db
class Table1(PeeweeBaseModel):
id = PrimaryKeyField()
username = CharField(255)
date = DateTimeField()
#other fields ...
class Table2(PeeweeBaseModel):
id = PrimaryKeyField()
date = DateTimeField()
#other fields ...
and then , union two model. something like this:
u = (
Table1(
Table1.id,
Table1.username,
Table1.date
).select()
|
Table2(
Table2.id,
"const_text_instead_real_field_value",
Table2.date
).select()
).select().execute()
But the const_text is not accepted by a field and ignore in result query.
the question is: How can I define a field that does not exist in my table and set it manually in query?
(And I prefer not using SQL() function.)
thanks.

you can use SQL() in SELECT statement.
u = (
Table1(
Table1.id,
Table1.username,
Table1.date
).select()
|
Table2(
Table2.id,
SQL(" '' AS username "),
Table2.date
).select()
).select().execute()