I'm currently trying to transfer a program designed for a MySQL database onto a MS-SQL database and I've run into some trouble. I discovered that MySQL does not have case sensitivity by default as MS-SQL has. This has lead to some problems with code similar to that listed below.
class Employee(Base):
__tablename__ = "Employees"
Id = Column(Integer(unsigned=True),
primary_key=True, nullable=False, unique=True)
DisplayName = Column(String(64),
nullable=False)
#more columns
def get_employees(sql_session, param, columns=None, partial_match=True):
if not columns:
columns = [Employee.Id, Employee.DisplayName]
clauses = []
if partial_match:
clauses.append(Employee.DisplayName.startswith(param))
whereclause = and_(*clauses)
stmt = select(columns, whereclause)
return sql_session.execute(stmt)
I know of the SQL keyword COLLATE but I'm not sure how to implement that, or if it's even the best option to use in this situation. What recommendations would you give to create a case insensitive LIKE query using SQLAlchemy?
Python 2.7.7
SQLAlchemy 0.7.7
That's a bit odd, in my experience MS SQL Server is case insensitive by default although you can optionally set it to case sensitive using the database's collation setting.
You can use COLLATE with SqlAlchemy (see here), so you should be able to do (I have not tried this myself):
clauses.append(Employee.DisplayName.startswith(collate(param, 'SQL_Latin1_General_CP1_CI_AS')))
SQL Server also supports regex-like pattern matching with LIKE queries, so alternatively you could make use of this in your param value e.g. '[vV]alue%'
Related
I am newbie in python/sqlachemy world. I am trying to fetch data from PostgreSQL using sqlachemy in flask, where I need to fetch only selected column instead of streaming all the column from database through network. One of the approach is using session (like below) which is working fine,
session.query(User.user_id, User.name).all()
But for some reason I would like to stick with Model.query method instead of using sessions. So I ended up to use something like below,
User.query.options(load_only(User.user_id, User.name)).all()
Above code snippet doesn't filters the selected column, instead it gives back all the column. It looks like sqlachemy doesn't respects the load_only arguments. Why is that behaviour and any solution to achieve my use case in Model.query instead of using sessions?
My user model looks like below,
class User(db.Model):
__tablename__ = 'user_info'
user_id = Column(String(250), primary_key=True)
name = Column(String(250))
email = Column(String(250))
address = Column(String(512))
Version Info
Python - 3.7
sqlachemy - 1.3.11
Edit 1: Though I added load_only attributes, It is generating the following query,
SELECT user.user_id AS user_user_id, user.name AS user_name, user.email AS user_email, user.address AS user_address FROM user_info
I'm using SQLAlchemy for MySQL.
The common example of SQLAlchemy is
Defining model classes by the table structure. (class User(Base))
Migrate to the database by db.create_all (or alembic, etc)
Import the model class, and use it. (db.session.query(User))
But what if I want to use raw SQL file instead of defined model classes?
I did read automap do similar like this, but I want to get mapper object from raw SQL file, not created database.
Is there any best practice to do this?
This is an example of DDL
-- ddl.sql
-- This is just an example, so please ignore some issues related to a grammar
CREATE TABLE `card` (
`card_id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'card',
`card_company_id` bigint(20) DEFAULT NULL COMMENT 'card_company_id',
PRIMARY KEY (`card_id`),
KEY `card_ix01` (`card_company_id`),
KEY `card_ix02` (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='card table'
And I want to do like
Base = raw_sql_base('ddl.sql') # Some kinda automap_base but from SQL file
# engine, suppose it has two tables 'user' and 'address' set up
engine = create_engine("mysql://user#localhost/program")
# reflect the tables
Base.prepare(engine)
# mapped classes are now created with names by sql file
Card = Base.classes.card
session = Session(engine)
session.add(Card(card_id=1, card_company_id=1))
session.commit() # Insert
SQLAlchemy is not an SQL parser, but the exact opposite; its reflection works against existing databases only. In other words you must execute your DDL and then use reflection / automap to create the necessary Python models:
from sqlalchemy.ext.automap import automap_base
# engine, suppose it has two tables 'user' and 'address' set up
engine = create_engine("mysql://user#localhost/program")
# execute the DDL in order to populate the DB
with open('ddl.sql') as ddl:
engine.execute(ddl)
Base = automap_base()
# reflect the tables
Base.prepare(engine, reflect=True)
# mapped classes are now created with names by sql file
Card = Base.classes.card
session = Session(engine)
session.add(Card(card_id=1, card_company_id=1))
session.commit() # Insert
This of course may fail, if you have already executed the same DDL against your database, so you would have to handle that case as well. Another possible caveat is that some DB-API drivers may not like executing multiple statements at a time, if your ddl.sql happens to contain more than one CREATE TABLE statement etc.
...but I want to get mapper object from raw SQL file.
Ok, in that case what you need is the aforementioned parser. A cursory search produced two candidates:
sqlparse: Generic, but the issue tracker is a testament to how nontrivial parsing SQL is. Is often confused, for example parses ... COMMENT 'card', `card_company_id` ... as a keyword and an identifier list, not as a keyword, a literal, punctuation, and an identifier (or even better, the column definitions as their own nodes).
mysqlparse: A MySQL specific solution, but with limited support for just about anything, and it seems abandoned.
Parsing would be just the first step, though. You'd then have to convert the resulting trees to models.
How does one add a UniqueConstraint which is Case Insensitive using SQLalchemy?
In some databases, string columns are case-insensitive by default (MySQL, SQL Server), so you wouldn't need to do anything extra.
In others, you can create a functional index that enforces the case-insensitive unique constraint:
Index('myIndex', func.lower(mytable.c.myColumn), unique=True)
You can also specify a case-insensitive collation for the column if the database supports it. For instance SQLite has a 'NOCASE' collation:
myColumn = Column(String(255), collation='NOCASE', nullable=False)
See http://docs.sqlalchemy.org/en/latest/core/type_basics.html?highlight=collation#sqlalchemy.types.String.params.collation
You may also specify a user-defined type for your column if your database provides a suitable one. PostgreSQL has a citext data type which is case-insensitive. See https://github.com/mahmoudimus/sqlalchemy-citext
Finally you can customize the DDL to create the constraint for your specific database.
To add upon #jspcal's answer, if the model is defined using a class, then one you would have to either instantiate it independently after declaring the model or use the text construct.
i.e.
from sqlalchemy.sql.expressions import func
class User(Base):
__tablename__ = 'user'
username = Column('username', String(24), nullable=False)
Index('user_username_index', func.lower(User.username), unique=True)
using the text construct:
from sqlalchemy.sql.expressions import text
class User(Base):
__tablename__ = 'user'
__table_args__ = (
Index('ix_user_name', text('LOWER(username)')),
)
username = Column('username', String(24), nullable=False)
NB: table_args needs to be a tuple or dict, hence the need for that trailing comma inside the parenthesis.
That will create an index on username column of table user in lowercase form. Therefore, data stored in this column is unique and case insensitive.
I am accessing Postgre database using SQLAlchemy models. In one of models I have Column with UUID type.
id = Column(UUID(as_uuid=True), default=uuid.uuid4(), nullable=False, unique=True)
and it works when I try to insert new row (generates new id).
Problem is when I try to fetch Person by id I try like
person = session.query(Person).filter(Person.id.like(some_id)).first()
some_id is string received from client
but then I get error LIKE (Programming Error) operator does not exist: uuid ~~ unknown.
How to fetch/compare UUID column in database through SQLAlchemy ?
don't use like, use =, not == (in ISO-standard SQL, = means equality).
Keep in mind that UUID's are stored in PostgreSQL as binary types, not as text strings, so LIKE makes no sense. You could probably do uuid::text LIKE ? but it would perform very poorly over large sets because you are effectively ensuring that indexes can't be used.
But = works, and is far preferable:
mydb=>select 'd796d940-687f-11e3-bbb6-88ae1de492b9'::uuid = 'd796d940-687f-11e3-bbb6-88ae1de492b9';
?column?
----------
t
(1 row)
So let's say I'm using Python 2.5's built-in default sqlite3 and I have a Django model class with the following code:
class SomeEntity(models.Model):
some_field = models.CharField(max_length=50, db_index=True, unique=True)
I've got the admin interface setup and everything appears to be working fine except that I can create two SomeEntity records, one with some_field='some value' and one with some_field='Some Value' because the unique constraint on some_field appears to be case sensitive.
Is there some way to force sqlite to perform a case insensitive comparison when checking for uniqueness?
I can't seem to find an option for this in Django's docs and I'm wondering if there's something that I can do directly to sqlite to get it to behave the way I want. :-)
Yes this can easily be done by adding a unique index to the table with the following command:
CREATE UNIQUE INDEX uidxName ON mytable (myfield COLLATE NOCASE)
If you need case insensitivity for nonASCII letters, you will need to register your own COLLATION with commands similar to the following:
The following example shows a custom collation that sorts “the wrong way”:
import sqlite3
def collate_reverse(string1, string2):
return -cmp(string1, string2)
con = sqlite3.connect(":memory:")
con.create_collation("reverse", collate_reverse)
cur = con.cursor()
cur.execute("create table test(x)")
cur.executemany("insert into test(x) values (?)", [("a",), ("b",)])
cur.execute("select x from test order by x collate reverse")
for row in cur:
print row
con.close()
Additional python documentation for sqlite3 shown here
Perhaps you can create and use a custom model field; it would be a subclass of CharField but providing a db_type method returning "text collate nocase"
For anyone in 2021, with the help of Django 4.0 UniqueConstraint expressions you could add a Meta class to your model like this:
class Meta:
constraints = [
models.UniqueConstraint(
Lower('<field name>'),
name='<constraint name>'
),
]