Develop Reference

SQLAlchemy getting a row using filter() but not with get() function

I'm using SQLAlchemy==1.3.18 and Flask==1.1.2
My problem is that I want to update a row that's inserted using a Stored Procedure. Here is my process:
My engine to the DB:
from sqlalchemy import create_engine
engine = create_engine('mysql+pymysql://db_user:db_password#db_host:db_port')
My model
class Employee(declararive_base()):
__tablename__ = "EMPLOYEE"
id = Column(Integer, primary_key=True)
name = Column(String(30))
last_name = Column(String(30))
I execute a Stored Procedure using this function:
def call_procedure(function_name, params):
connection = cloudsql.Engine.raw_connection()
try:
cursor = connection.cursor()
cursor.callproc(function_name, params)
results = list(cursor.fetchall())
cursor.close()
connection.commit()
return results
finally:
connection.close()
Then, I recover the id of the inserted record and set the value on this variable new_employee_id.
After that I try to execute this: session.query(Employee).get(new_employee_id) and I get None.
But if I do the query using session.query(Employee).filter(Employee.id == new_employee_id).all()
I get the record.
Any ideas about this?
I'm thinking that maybe the get method works over the index and since the table is not reindexed again at that moment, the query cannot get the record. What happens with the filter is that it searches on the whole table and can get the record.
Update: My model class was added to this question.
Update2: I added how I create my engine on SQLAlchemy.

Forcing python to accept variable changes from database without restarting code

I am using the following code to retrieve variables from a database that Python uses to run an automated machine. I set the variables through a PHP driven web interface. Python reads the variables and acts according to instructions.
However, during calibration of the machine, we are forced to restart python to accept any variable changes. Python isn't my first language and neither is it the first language of my colleagues. It would obviously save a lot of time if we didn't have to restart python to accept variable changes.
Our variable list class is constructed like the following;
class VariableList():
connectdb = DbConnector(host='localhost', user='a', password='b', database='c')
result = connectdb.selectDb('variablelist','varA,varB')
for row in result:
# INPUTS
varA = row[1]
varB = row[2]
What is the Pythonic way to get around this issue? Getters/Setters? #property? An example to follow would very much appreciated...

easy one. Python implementation goes like this:
class VariableList():
def __init__(self):
self.db_con = DbConnector(host='localhost', user='a', password='b', database='c')
#property
def varA(self):
return self.db_con.selectDb('variablelist','varA')
#property.setter
def varA(self, value):
self.db_con.updateDb('variablelist', value)
and also you can refactor your model with SQLAlchemy framework.
for example
from sqlalchemy (
create_engine,
Column,
Integer,
String,
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
engine = create_engine(DB_DNS,
pool_size=DB_POOL_SIZE,
max_overflow=DB_MAX_OVERFLOW,
pool_recycle=DB_POOL_RECYCLE,
isolation_level="READ UNCOMMITTED", # attention, the last one is important!
)
Session = sessionmaker(bind=engine,
autocommit=False,
expire_on_commit=False)
class MyTable(Base):
__tablename__ = MY_TABLE_NAME
id = Column(Integer, primary=True)
name = Column(String(32), nullable=True, default='', doc='user_name')
# query something
result = Session().query(MyTable).filter(CONDITION).all()

How to do a proper upsert using sqlalchemy on postgresql?

I would like to do an upsert using the "new" functionality added by postgresql 9.5, using sqlalchemy core. While it is implemented, I'm pretty confused by the syntax, which I can't adapt to my needs.
Here is a sample code of what I would like to be able to do :
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
__tablename__ = 'test'
a_id = Column('id',Integer, primary_key=True)
a = Column("a",Integer)
engine = create_engine('postgres://name:password#localhost/test')
User().metadata.create_all(engine)
meta = MetaData(engine)
meta.reflect()
table = Table('test', meta, autoload=True)
conn = engine.connect()
from sqlalchemy.dialects.postgresql import insert as psql_insert
stmt = psql_insert(table).values({
table.c['id']: bindparam('id'),
table.c['a']: bindparam('a'),
})
stmt = stmt.on_conflict_do_update(
index_elements=[table.c['id']],
set_={'a': bindparam('a')},
)
list_of_dictionary = [{'id':1, 'a':1, }, {'id':2, 'a':2,}]
conn.execute(stmt, list_of_dictionary)
I basically want to insert a bulk of rows, and if one id is already taken, I want to update it with the value I initially wanted to insert.
However sqlalchemy throw me this error :
CompileError: bindparam() name 'a' is reserved for automatic usage in the VALUES or SET clause of this insert/update statement. Please use a name other than column name when using bindparam() with insert() or update() (for example, 'b_a').
While it is a known issue (see https://groups.google.com/forum/#!topic/sqlalchemy/VwiUlF1cz_o), I didn't found any proper answer that does not require to modify either the keys of list_of_dictionary or the name of your columns.
I want to know if there is a way of constructing stmt in a way to have a consistent behavior that does not depends on whether the keys of the variable list_of_dictionary are the name of the columns of the inserted table (my code works without error in those cases).

this does the trick for me:
from sqlalchemy import create_engine
from sqlalchemy import MetaData, Table
from sqlalchemy.dialects import postgresql
from sqlalchemy.inspection import inspect
def upsert(engine, schema, table_name, records=[]):
metadata = MetaData(schema=schema)
metadata.bind = engine
table = Table(table_name, metadata, schema=schema, autoload=True)
# get list of fields making up primary key
primary_keys = [key.name for key in inspect(table).primary_key]
# assemble base statement
stmt = postgresql.insert(table).values(records)
# define dict of non-primary keys for updating
update_dict = {
c.name: c
for c in stmt.excluded
if not c.primary_key
}
# cover case when all columns in table comprise a primary key
# in which case, upsert is identical to 'on conflict do nothing.
if update_dict == {}:
warnings.warn('no updateable columns found for table')
# we still wanna insert without errors
insert_ignore(table_name, records)
return None
# assemble new statement with 'on conflict do update' clause
update_stmt = stmt.on_conflict_do_update(
index_elements=primary_keys,
set_=update_dict,
)
# execute
with engine.connect() as conn:
result = conn.execute(update_stmt)
return result

For anyone looking for an ORM solution, the following worked for me:
def upsert(
sa_sessionmaker: Union[sessionmaker, scoped_session],
model: DeclarativeMeta,
get_values: Dict[str, Any],
update_values: Dict[str, Any],
) -> Any:
"""Upserts (updates if exists, else inserts) a SQLAlchemy model object.
Note that get_values must uniquely identify a single model object (row) for this
function to work.
Args:
sa_sessionmaker: SQLAlchemy sessionmaker to connect to the database.
model: Model declarative metadata.
get_values: Arguments used to try to retrieve an existing object.
update_values: Desired attributes for the object fetched via get_values,
or the new object if nothing was fetched.
Returns:
Model object subject to upsert.
"""
with sa_sessionmaker() as session:
instance = session.query(model).filter_by(**get_values).one_or_none()
if instance:
for attr, new_val in update_values.items():
setattr(instance, attr, new_val)
else:
create_kwargs = get_values | update_values
session.add(model(**create_kwargs))
session.commit()
instance = session.query(model).filter_by(**get_values).one_or_none()
return instance
A few remarks:
If the primary key of the object is known, using Session.merge() is likely a better alternative than the function above. In that sense, the function above assumes that the primary key is not known (and hence not part of get_values)
sa_sessionmaker is a factory for Session objects (see the docs)
model takes a SQLAlchemy declarative metadata (i.e., a "table" see the docs)
Python >= 3.9 required for the implementation above. If your environment requires a previous version of Python, replace create_kwargs = get_values | update_values with create_kwargs = {**get_values, **update_values}

Is there a way to use SQL Tables as classes or objects in python?

I have to tie database and programming for an assignment and I have an idea for a code but need to make sure that I can use the tables I created in mySQL as my classes or objects in Python.
Example: I use SQL to create a database of houses with specific addresses and zip codes. A client says they live in zipcode x. My program should then parse through the database and return all addresses within zipcode x. Then ideally create a table in SQL with the clients results.
Not the exact assignment but it gets the basic idea across.

You're looking for an ORM. See SQLAlchemy. Example:
from sqlalchemy import Column, String, Integer, Sequence
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
create_session = sessionmaker()
Base = declarative_base()
person_autoincr_seq = Sequence('person_autoincr_seq')
class Person(Base):
__tablename__ = "person"
id = Column(
Integer,
person_autoincr_seq,
server_default=person_autoincr_seq.next_value(),
nullable = False,
primary_key = True
)
name = Column(
String,
nullable = False
)
def __init__(self, name,id=None):
if id is not None:
self.id = id
self.name = name
Using the db:
import logging as log
from contextlib import closing
engine = sqlalchemy.engine.create_engine(
"postgresql://testuser:mypassword#127.0.0.1:5432/testdb"
)
create_session.configure(bind=engine)
try:
with closing(create_session()) as db_session:
name = db_session.query(Person.name).filter_by(id=5).one()[0]
except Exception:
log.exception("Something wrong while querying db")

Sqlalchemy if table does not exist

I wrote a module which is to create an empty database file
def create_database():
engine = create_engine("sqlite:///myexample.db", echo=True)
metadata = MetaData(engine)
metadata.create_all()
But in another function, I want to open myexample.db database, and create tables to it if it doesn't already have that table.
EG of the first, subsequent table I would create would be:
Table(Variable_TableName, metadata,
Column('Id', Integer, primary_key=True, nullable=False),
Column('Date', Date),
Column('Volume', Float))
(Since it is initially an empty database, it will have no tables in it, but subsequently, I can add more tables to it. Thats what i'm trying to say.)
Any suggestions?

I've managed to figure out what I intended to do. I used engine.dialect.has_table(engine, Variable_tableName) to check if the database has the table inside. IF it doesn't, then it will proceed to create a table in the database.
Sample code:
engine = create_engine("sqlite:///myexample.db") # Access the DB Engine
if not engine.dialect.has_table(engine, Variable_tableName): # If table don't exist, Create.
metadata = MetaData(engine)
# Create a table with the appropriate Columns
Table(Variable_tableName, metadata,
Column('Id', Integer, primary_key=True, nullable=False),
Column('Date', Date), Column('Country', String),
Column('Brand', String), Column('Price', Float),
# Implement the creation
metadata.create_all()
This seems to be giving me what i'm looking for.

Note that in 'Base.metadata' documentation it states about create_all:
Conditional by default, will not attempt to recreate tables already
present in the target database.
And if you can see that create_all takes these arguments: create_all(self, bind=None, tables=None, checkfirst=True), and according to documentation:
Defaults to True, don't issue CREATEs for tables already present in
the target database.
So if I understand your question correctly, you can just skip the condition.

The accepted answer prints a warning that engine.dialect.has_table() is only for internal use and not part of the public API. The message suggests this as an alternative, which works for me:
import os
import sqlalchemy
# Set up a connection to a SQLite3 DB
test_db = os.getcwd() + "/test.sqlite"
db_connection_string = "sqlite:///" + test_db
engine = create_engine(db_connection_string)
# The recommended way to check for existence
sqlalchemy.inspect(engine).has_table("BOOKS")
See also the SQL Alchemy docs.

For those who define the table first in some models.table file, among other tables.
This is a code snippet for finding the class that represents the table we want to create ( so later we can use the same code to just query it )
But together with the if written above, I still run the code with checkfirst=True
ORMTable.__table__.create(bind=engine, checkfirst=True)
models.table
class TableA(Base):
class TableB(Base):
class NewTableC(Base):
id = Column('id', Text)
name = Column('name', Text)
form
Then in the form action file:
engine = create_engine("sqlite:///myexample.db")
if not engine.dialect.has_table(engine, table_name):
# Added to models.tables the new table I needed ( format Table as written above )
table_models = importlib.import_module('models.tables')
# Grab the class that represents the new table
# table_name = 'NewTableC'
ORMTable = getattr(table_models, table_name)
# checkfirst=True to make sure it doesn't exists
ORMTable.__table__.create(bind=engine, checkfirst=True)

engine.dialect.has_table does not work for me on cx_oracle.
I am getting AttributeError: 'OracleDialect_cx_oracle' object has no attribute 'default_schema_name'
I wrote a workaround function:
from sqlalchemy.engine.base import Engine
def orcl_tab_or_view_exists(in_engine: Engine, in_object: str, in_object_name: str,)-> bool:
"""Checks if Oracle table exists in current in_engine connection
in_object: 'table' | 'view'
in_object_name: table_name | view_name
"""
obj_query = """SELECT {o}_name FROM all_{o}s WHERE owner = SYS_CONTEXT ('userenv', 'current_schema') AND {o}_name = '{on}'
""".format(o=in_object, on=in_object_name.upper())
with in_engine.connect() as connection:
result = connection.execute(obj_query)
return len(list(result)) > 0

This is the code working for me to create all tables of all model classes defined with Base class
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
class YourTable(Base):
__tablename__ = 'your_table'
id = Column(Integer, primary_key = True)
DB_URL="mysql+mysqldb://<user>:<password>#<host>:<port>/<db_name>"
scoped_engine = create_engine(DB_URL)
Base = declarative_base()
Base.metadata.create_all(scoped_engine)