Develop Reference

Filtering a PostgreSQL function that returns a table in sqlalchemy

I am trying to solve the following two problems:
Constructing an ORM model to use in SQLAlchemy with a table that is generated from a user defined function in PostgreSQL
Filtering said table (either in ORM model or not)
I have defined a function that returns a table called transformed. It takes a user_id as an input and returns 3 columns. Looks something like this below (this is extremely simplified, the query is obviously a little more complicated).
CREATE OR REPLACE FUNCTION transformed (user_id integer)
RETURNS TABLE (
id integer,
user varchar,
description varchar,
)
AS $body$
SELECT
id,
user,
description
FROM table
WHERE id = $1
$body$
LANGUAGE sql;
def build_query(user_id):
base_query: Query = select(
Column("id", Integer),
Column("user", String),
Column("description", String),
).select_from(func.transformed)_data(user_id))
return base_query
I have no problem getting the three columns. The challenge is to filter and sort them.
query = build_query(user_id).filter_by(description = "Foo")
This returns the following error: sqlalchemy.exc.InvalidRequestError: Entity namespace for "transformed(:transformed_1) has no property "description".
I would love to know how to either a) properly filter or b) turn the pure columns I am defining into some sort of ORM model that I can then use to filter on the properties of the ORM model.

SQLAlchemy is pretty awesome. You can define the function using DDLEvents and then call the function using func and define the output with table_valued there is a different method for a scalar but its in the same docs.
import sys
from sqlalchemy import (
create_engine,
Integer,
String,
event,
column,
)
from sqlalchemy.schema import (
Column,
DDL,
)
from sqlalchemy.sql import select, func
from sqlalchemy.orm import declarative_base, Session
Base = declarative_base()
username, password, db = sys.argv[1:4]
engine = create_engine(f"postgresql+psycopg2://{username}:{password}#/{db}", echo=True)
class User(Base):
__tablename__ = "users"
id = Column(Integer, primary_key=True)
name = Column(String(8), index=True)
event.listen(
User.__table__,
"after_create",
DDL("""CREATE OR REPLACE FUNCTION transformed (int) RETURNS TABLE (id integer, username varchar)
AS $$ SELECT id, name AS username FROM users WHERE id = $1 $$
LANGUAGE sql;
"""))
Base.metadata.create_all(engine)
with Session(engine) as session, session.begin():
for name in ['User 1', 'User 2', 'User 3']:
session.add(User(name=name))
with Session(engine) as session, session.begin():
# Get a single row with id 1
fn = func.transformed(1).table_valued(
column("id", Integer),
column("username", String))
# This filtering is redundant because we selected
# a user by its id but it demonstrates referencing the column
# we defined above.
q = select(fn).where(fn.c.username.like('User %'))
for id, user in session.execute(q).all():
print(id, user)
This prints out
1 User 1
The actual query looks like this, via echo=True:
SELECT anon_1.id, anon_1.username
FROM transformed(%(transformed_1)s) AS anon_1
WHERE anon_1.username LIKE %(username_1)s

There is one possible solution but it's very similar to simply constructing the query via raw text:
query = build_query(user_id).where(text("id = 4"))
Not a huge fan since it's not easy to expand on this dynamically.

SQLAlchemy: Could not locate any relevant foreign key columns for primary join condition

I have built two tables in my postgres CLI (test2_table: id, name, age, profession, city, country). The second is referenced to my 1st table's id and has a column called wage. When I run the syntax everything is fine.
SELECT name, age, profession, city, country, wage FROM test2_table JOIN salary ON salary.test2_id = test2_table.id;
This works - a table with wages is printed for me to see.
My only problem is when I try the same thing to my SQLAlchemy database class. When I try to join the classes together I get an error:
sqlalchemy.exc.ArgumentError: Could not locate any relevant foreign key columns for primary join condition 'test2_table.id = salary.id' on relationship Salary.wages.
Ensure that referencing columns are associated with a ForeignKey or ForeignKeyConstraint, or are annotated in the join condition with the foreign() annotation.
my database classes:
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Test_db_02(Base):
__tablename__ = 'test2_table'
id = Column('id', Integer, primary_key=True)
name = Column('name', String(40))
age = Column('age', Integer)
profession = Column('profession', String(60))
city = Column('city', String(60))
country = Column('country', String(40))
class Salary(Base):
__tablename__ = 'salary'
id = Column('id', Integer, primary_key=True)
wage = Column('wage', String(20))
test2_id = Column('test2_id', Integer, ForeignKey('test2_table.id'))
wages = relationship("Test_db_02", backref="salary", primaryjoin="Test_db_02.id == Salary.id")
I have a simple report page that prompts the user to chose from a drop-down option selector. Before I tried to join the two tables togteher, my queries were working well, now after my attempt to join I get the error. The reports #app has been scaled down for this example.
Reports #app.route
Session = sessionmaker(bind=engine)
session = Session()
#app.route('/reports', methods=['GET', 'POST'])
if request.method == 'GET':
return render_template('reports.html')
else:
if request.form.get("report_options") == "all_db":
db_entry = session.query(Test_db_02).order_by(Test_db_02.id)
db_entry = db_entry.all()
return render_template('reports.html', db_entry=db_entry)
elif request.form.get("report_options") == "name":
db_entry = session.query(Test_db_02).order_by(Test_db_02.id)
data_name = db_entry.all()
return render_template('reports.html', data_name=data_name)
elif request.form.get("report_options") == "media_prof":
db_entry = session.query(Test_db_02).join(test2_table.salary)
media_prof = db_entry.all()
return render_template('reports.html', media_prof=media_prof)
If I am honest I have read through the sqlalchemy documentation on relationships, joins & foreign keys (and watched some YouTube tutorials) but it still seems a bit confusing..
The main challenge for me is to be able to join two tables together. Once I have accomplished that, I will attempt to iterate through them with the flask/jinja set up.

Your primaryjoin condition in Salary.wages is wrong. Presumably, Salary.idis an autoincrement primary key column and as you have no ForeignKey constraining values of Salary.id to values of Test_db_02.id, its unlikely that you want to include that column in the join condition as you have done:
primaryjoin="Test_db_02.id == Salary.id"
Its more likely that you want to relate the two models through the Salary.test2_id attribute, as you have the ForeignKey constraint on that column:
primaryjoin="Test_db_02.id == Salary.test2_id"

SQLAlchemy + Postgres (print created types, e.g. enum types)

I would like to be able to get info about what types will be created during SQLAlchemy's create_all(). Yes, they can be printed if I set up echo-ing of generated SQL, but how can i print it without actually hitting database? For example, I have a model:
class MyModel(Base):
id = Column(Integer, primary_key=True)
indexed_field = Column(String(50))
enum_field = Column(Enum(MyEnum))
__table_args__ = (
Index("my_ix", indexed_field),
)
where MyEnum is:
class MyEnum(enum.Enum):
A = 0
B = 1
I can get CREATE TABLE statement and all CREATE INDEX statements like this:
from sqlalchemy.schema import CreateTable, CreateIndex
print(str(CreateTable(MyModel.__table__).compile(postgres_engine)))
for idx in MyModel.__table__.indexes:
print(str(CreateIndex(idx)).compile(postgres_engine))
Result will be something like that:
CREATE TABLE my_model (
id SERIAL NOT NULL,
indexed_field VARCHAR(50),
enum_field myenum,
PRIMARY KEY (id)
)
CREATE INDEX my_ix ON my_model (indexed_field)
Notice the line enum_field myenum. How can I get generated SQL for CREATE TYPE myenum... statement?

I've found the answer!
from sqlalchemy.dialects.postgresql.base import PGInspector
PGInspector(postgres_engine).get_enums()
It returns a list of all created enums, which IMO is even better than raw sql, documentation is here.

SQLAlchemy mapping joined tables' columns to one object

I have three tables: UserTypeMapper, User, and SystemAdmin. In my get_user method, depending on the UserTypeMapper.is_admin row, I then query either the User or SystemAdmin table. The user_id row correlates to the primary key id in the User and SystemAdmin tables.
class UserTypeMapper(Base):
__tablename__ = 'user_type_mapper'
id = Column(BigInteger, primary_key=True)
is_admin = Column(Boolean, default=False)
user_id = Column(BigInteger, nullable=False)
class SystemAdmin(Base):
__tablename__ = 'system_admin'
id = Column(BigInteger, primary_key=True)
name = Column(Unicode)
email = Column(Unicode)
class User(Base):
__tablename__ = 'user'
id = Column(BigInteger, primary_key=True)
name = Column(Unicode)
email = Column(Unicode)
I want to be able to get any user – system admin or regular user – from one query, so I do a join, on either User or SystemAdmin depending on the is_admin row. For example:
DBSession.query(UserTypeMapper, SystemAdmin).join(SystemAdmin, UserTypeMapper.user_id==SystemAdmin.id).first()
and
DBSession.query(UserTypeMapper, User).join(User, UserTypeMapper.user_id==User.id).first()
This works fine; however, I then would like to be access these, like so:
>>> my_admin_obj.is_admin
True
>>> my_admin_obj.name
Bob Smith
versus
>>> my_user_obj.is_admin
False
>>> my_user_obj.name
Bob Stevens
Currently, I have to specify: my_user_obj.UserTypeMapper.is_admin and my_user_obj.User.name. From what I've been reading, I need to map the tables so that I don't need to specify which table the attribute belongs to. My problem is that I do not understand how I can specify this given that I have two potential tables that the name attribute, for example, may come from.
This is the example I am referring to: Mapping a Class against Multiple Tables
How can I achieve this? Thank you.

You have discovered why "dual purpose foreign key", is an antipattern.
There is a related problem to this that you haven't quite pointed out; there's no way to use a foreign key constraint to enforce the data be in a valid state. You want to be sure that there's exactly one of something for each row in UserTypeMapper, but that 'something' is not any one table. formally you want a functional dependance on
user_type_mapper → (system_admin× 1) ∪ (user× 0)
But most sql databses won't allow you to write a foreign key constraint expressing that.
It looks complicated because it is complicated.
instead, lets consider what we really want to say; "every system_admin should be a user; or
system_admin → user
In sql, that would be written:
CREATE TABLE user (
id INTEGER PRIMARY KEY,
name VARCHAR,
email VARCHAR
);
CREATE TABLE system_admin (
user_id INTEGER PRIMARY KEY REFERENCES user(id)
);
Or, in sqlalchemy declarative style
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String)
email = Column(String)
class SystemAdmin(Base):
__tablename__ = 'system_admin'
user_id = Column(ForeignKey(User.id), primary_key=True)
What sort of questions does this schema allow us to ask?
"Is there a SystemAdmin by the name of 'john doe'"?
>>> print session.query(User).join(SystemAdmin).filter(User.name == 'john doe').exists()
EXISTS (SELECT 1
FROM "user" JOIN system_admin ON "user".id = system_admin.user_id
WHERE "user".name = :name_1)
"How many users are there? How many sysadmins?"
>>> print session.query(func.count(User.id), func.count(SystemAdmin.user_id)).outerjoin(SystemAdmin)
SELECT count("user".id) AS count_1, count(system_admin.user_id) AS count_2
FROM "user" LEFT OUTER JOIN system_admin ON "user".id = system_admin.user_id
I hope you can see why the above is prefereable to the design you describe in your question; but in the off chance you don't have a choice (and only in that case, if you still feel what you've got is better, please refine your question), you can still cram that data into a single python object, which will be very difficult to work with, by providing an alternate mapping to the tables; specifically one which follows the rough structure in the first equation.
We need to mention UserTypeMapper twice, once for each side of the union, for that, we need to give aliases.
>>> from sqlalchemy.orm import aliased
>>> utm1 = aliased(UserTypeMapper)
>>> utm2 = aliased(UserTypeMapper)
For the union bodies join each alias to the appropriate table: Since SystemAdmin and User have the same columns in the same order, we don't need to describe them in detail, but if they are at all different, we need to make them "union compatible", by mentioning each column explicitly; this is left as an exercise.
>>> utm_sa = Query([utm1, SystemAdmin]).join(SystemAdmin, (utm1.user_id == SystemAdmin.id) & (utm1.is_admin == True))
>>> utm_u = Query([utm2, User]).join(User, (utm2.user_id == User.id) & (utm2.is_admin == False))
And then we join them together...
>>> print utm_sa.union(utm_u)
SELECT anon_1.user_type_mapper_1_id AS anon_1_user_type_mapper_1_id, anon_1.user_type_mapper_1_is_admin AS anon_1_user_type_mapper_1_is_admin, anon_1.user_type_mapper_1_user_id AS anon_1_user_type_mapper_1_user_id, anon_1.system_admin_id AS anon_1_system_admin_id, anon_1.system_admin_name AS anon_1_system_admin_name, anon_1.system_admin_email AS anon_1_system_admin_email
FROM (SELECT user_type_mapper_1.id AS user_type_mapper_1_id, user_type_mapper_1.is_admin AS user_type_mapper_1_is_admin, user_type_mapper_1.user_id AS user_type_mapper_1_user_id, system_admin.id AS system_admin_id, system_admin.name AS system_admin_name, system_admin.email AS system_admin_email
FROM user_type_mapper AS user_type_mapper_1 JOIN system_admin ON user_type_mapper_1.user_id = system_admin.id AND user_type_mapper_1.is_admin = 1 UNION SELECT user_type_mapper_2.id AS user_type_mapper_2_id, user_type_mapper_2.is_admin AS user_type_mapper_2_is_admin, user_type_mapper_2.user_id AS user_type_mapper_2_user_id, "user".id AS user_id, "user".name AS user_name, "user".email AS user_email
FROM user_type_mapper AS user_type_mapper_2 JOIN "user" ON user_type_mapper_2.user_id = "user".id AND user_type_mapper_2.is_admin = 0) AS anon_1
While it's theoretically possible to wrap this all up into a python class that looks a bit like standard sqlalchemy orm stuff, I would certainly not do that. working with non-table mappings, especially when they are more than simple joins (this is a union), is lots of work for zero payoff.

delete not cascaded to table in sqlalchemy

I am developing an extension to an existing app which uses sqlalchemy 0.6.
The app has sqlalchemy tables created the non-declarative way. I am trying to create in my extension a new table with a foreign key column pointing at the primary key of the main table in the application database and I am creating it declaratively.
This all works fine, with the table created once the extension is loaded, and with no complaints at all. My table prints out and demonstrates that new rows have been added ok.
What I want and think is possible (but don't know as I have never used sql or any other database) is for the corresponding row in my table to be deleted when the row in the app's main table with the corresponding foreign key is deleted.
So far, and with many permutations having been tried, nothing has worked. I thought that with a backref set and with a relation defined with delete being cascaded, there shouldn't be a problem. Because the new table is defined in an extension which should just plugin, I don't want to edit the code in the main app at all, at least that is my goal. One of the problems that I have, though, is that the main app table that I want to reference, has no member variables defined in its class, does not declare its primary key in its mapper and only has the primary key declared in the table. This makes it difficult to create a relation(ship) clause, the first argument of which must be to a class or mapper (in this case neither of which have the primary key declared).
Is there any way of achieving this?
ps - here is some of the code that I am using. LocalFile is the declarative class. All the connection details are taken care of by the main application.
if not self.LocalFile.__table__.exists(bind=Engine):
self.LocalFile__table__.create(bind=Engine)
Here is the LocalFile class - Base is a declarative base class with bind=Engine passed in the constructor:
class LocalFile(Base):
__tablename__ = 'local_file'
_id = Column(Integer, Sequence('local_file_sequence', start=1, increment=1), primary_key=True)
_filename = Column(String(50), nullable=False)
_filepath = Column(String(128), nullable=False)
_movieid = Column(Integer, ForeignKey(db.tables.movies.c.movie_id, onupdate='CASCADE', ondelete='CASCADE'))
#movies = relation(db.Movie, backref="local_file", cascade="all")
#property
def filename(self):
return self._filename
#filename.setter
def filename(self, filename):
self._filename = filename
#property
def filepath(self):
return self._filepath
#filepath.setter
def filepath(self, filepath):
self._filepath = filepath
#property
def movieid(self):
return self._movieid
#movieid.setter
def movieid(self, movieid):
self._movieid = movieid
#property
def id(self):
return self._id
#id.setter
def id(self, id):
self._id = id
filename = synonym('_filename', descriptor=filename)
movieid = synonym('_movieid', descriptor=movieid)
filepath = synonym('_filepath', descriptor=filepath)
id = synonym('_id', descriptor=id)
def __init__(self, filename, filepath, movieid):
self._filename = filename
self._filepath = filepath
self._movieid = movieid
def __repr__(self):
return "<User('%s','%s', '%s')>" % (self.filename, self.filepath, self.movieid)
Edit:
The backend is sqlite3. Below is the code from the creation of the table produced by using the echo command (thanks for pointing that out, it's very useful - already I suspect that the existing application is generating far more sql than is necessary).
Following the reported sql table creation is the code generated when a row is removed. I personally can't see any statement that references the possible deletion of a row in the local file table, but I know very little sql currently. Thanks.
2011-12-29 16:29:18,530 INFO sqlalchemy.engine.base.Engine.0x...0650
CREATE TABLE local_file (
_id INTEGER NOT NULL,
_filename VARCHAR(50) NOT NULL,
_filepath VARCHAR(128) NOT NULL,
_movieid INTEGER,
PRIMARY KEY (_id),
FOREIGN KEY(_movieid) REFERENCES movies (movie_id) ON DELETE CASCADE ON UPDATE CASCADE
)
2011-12-29T16:29:18: I: sqlalchemy.engine.base.Engine.0x...0650(base:1387):
CREATE TABLE local_file (
_id INTEGER NOT NULL,
_filename VARCHAR(50) NOT NULL,
_filepath VARCHAR(128) NOT NULL,
_movieid INTEGER,
PRIMARY KEY (_id),
FOREIGN KEY(_movieid) REFERENCES movies (movie_id) ON DELETE CASCADE ON UPDATE CASCADE
)
2011-12-29 16:29:18,534 INFO sqlalchemy.engine.base.Engine.0x...0650 ()
2011-12-29T16:29:18: I: sqlalchemy.engine.base.Engine.0x...0650(base:1388): ()
2011-12-29 16:29:18,643 INFO sqlalchemy.engine.base.Engine.0x...0650 COMMIT
2011-12-29T16:29:18: I: sqlalchemy.engine.base.Engine.0x...0650(base:1095): COMMIT
for row in table produces the following for the two tables:
the local file table:
(, u' 310 To Yuma')
(, u' Ravenous')
the movie table in the existing app:
(, u'IMDb - 3:10 to Yuma')
(, u'Ravenous')
The code when deleting a row is so long that I cannot include it here (200 lines or so - isn't that a little too many for deleting one row?), but it makes no reference to deleting a row in the localfile table. There are statements like:
2011-12-29 17:09:17,141 INFO sqlalchemy.engine.base.Engine.0x...0650 UPDATE movies SET poster_md5=?, updated=? WHERE movies.movie_id = ?
2011-12-29T17:09:17: I: sqlalchemy.engine.base.Engine.0x...0650(base:1387): UPDATE movies SET poster_md5=?, updated=? WHERE movies.movie_id = ?
2011-12-29 17:09:17,142 INFO sqlalchemy.engine.base.Engine.0x...0650 (None, '2011-12-29 17:09:17.141019', 2)
2011-12-29T17:09:17: I: sqlalchemy.engine.base.Engine.0x...0650(base:1388): (None, '2011-12-29 17:09:17.141019', 2)
2011-12-29 17:09:17,150 INFO sqlalchemy.engine.base.Engine.0x...0650 DELETE FROM posters WHERE posters.md5sum = ?
2011-12-29T17:09:17: I: sqlalchemy.engine.base.Engine.0x...0650(base:1387): DELETE FROM posters WHERE posters.md5sum = ?
2011-12-29 17:09:17,157 INFO sqlalchemy.engine.base.Engine.0x...0650 (u'083841e14b8bb9ea166ea4b2b976f03d',)

In SQLite you must turn on support for foreign keys explicitly or it just ignores any SQL related to foreign keys.
engine = create_engine(database_url)
def on_connect(conn, record):
conn.execute('pragma foreign_keys=ON')
from sqlalchemy import event
event.listen(engine, 'connect', on_connect)