I'm having trouble with SQLAlchemy when doing a raw SQL which checks against multiple values.
my_sess.execute(
"SELECT * FROM table WHERE `key`='rating' AND uid IN :uids",
params=dict(uids=some_list)
).fetchall()
There are 2 scenarios for this query, one that works and one that doesn't. If some_list = [1], it throws me an SQL error that I have a syntax error near ). But if some_list = [1, 2], the query executes successfully.
Any reason why this would happen?
No, SQL parameters only ever deal with scalar values. You'll have to generate the SQL here; if you need raw SQL, use:
statement = "SELECT * FROM table WHERE `key`='rating' AND uid IN ({})".format(
', '.join([':i{}'.format(i) for i in range(len(some_list))]))
my_sess.execute(
statement,
params={'i{}'.format(i): v for i, v in enumerate(some_list)})
).fetchall()
e.g. generate enough parameters to hold all values in some_list with string formatting, then generate matching parameters to fill them.
Better still would be to use a literal_column() object to do all the generating for you:
from sqlalchemy.sql import literal_column
uid_in = literal_column('uid').in_(some_list)
statement = "SELECT * FROM able WHERE `key`='rating' AND {}".format(uid_in)
my_sess.execute(
statement,
params={'uid_{}'.format(i): v for i, v in enumerate(some_list)})
).fetchall()
but then you perhaps could just generate the whole statement using the `sqlalchemy.sql.expression module, as this would make supporting multiple database dialects much easier.
Moreover, the uid_in object already holds references to the right values for the bind parameters; instead of turning it into a string as we do with the str.format() action above, SQLAlchemy would have the actual object plus the associated parameters and you would no longer have to generate the params dictionary either.
The following should work:
from sqlalchemy.sql import table, literal_column, select
tbl = table('table')
key_clause = literal_column('key') == 'rating'
uid_clause = literal_column('uid').in_(some_list)
my_sess.execute(select('*', key_clause & uid_clause, [tbl]))
where the sqlalchemy.sql.select() takes a column spec (here hard-coded to *), a where clause (generated from the two clauses with & to generate a SQL AND clause) and a list of selectables; here your one sqlalchemy.sql.table() value.
Quick demo:
>>> from sqlalchemy.sql import table, literal_column, select
>>> some_list = ['foo', 'bar']
>>> tbl = table('table')
>>> key_clause = literal_column('key') == 'rating'
>>> uid_clause = literal_column('uid').in_(some_list)
>>> print select('*', key_clause & uid_clause, [tbl])
SELECT *
FROM "table"
WHERE key = :key_1 AND uid IN (:uid_1, :uid_2)
but the actual object tree generated from all this contains the actual values for the bind parameters too, so my_sess.execute() can access these directly.
Related
I'm trying to create a DataFrame through a sql query with pandas read_sql_query method. The query has a where clause that includes a like operation but it also includes a = operation that depends on a variable. The issue is that python is interpreting the % in the like operation as a place holder, just like in the = variable operation which is something I DO want.
Here's an example of it:
sql_string = """ SELECT a,b from table WHERE a = %(variable)s
AND b like '%fixed_chars%' """
params = {'variable':'AA'}
df = pandas.read_sql_query(sql_string, params=params, con=connection)
The error that I get is TypeError: not enough arguments for format string since it interprets the % you usually use as wildcard in mysql as the place holder in python.
In this case, you'll have to use two % for those not being formatting placeholders:
sql_string = "SELECT a,b from table WHERE a = %(variable)s AND \
b like '%%fixed_chars%%'"
Hope this helps!
I am using the Canari Framework to build a series of transforms within Malteog that scan a MySQL database and return the results as entities.
I've got the entire thing working bar the LIKE operator within the SQL statement. Here's the code:
#===========================================================================
# Request and set target from graph
target = (request.entity.value)
#===========================================================================
# Extract message column and check against target entity
statement = (
"SELECT * FROM tableT.TTO WHERE Text LIKE %(tq)s"
)
cursor.execute(statement, { 'tq': target })
results = cursor.fetchall()
#===========================================================================
# Create response entities based on SQL output
for r in results:
e = getTTO(target)
e.From = r[0]
e.Text = r[1]
e.date = r[2]
e.iconurl = 'http://local.paterva.com/phrase.png'
response += e
It works, but it's only returning matches that are exact. I need it to check the TEXT column to see if the entries contain any mention of the variable "target". I spent the best part of a day getting the python variable passed in, but I can't get it to return anything other than an exact match.
The SQL LIKE operator needs special "wildcard" markers ("%" in MySQL) before and/or after the value (depending on if you want a "starts with", "endswith" or "contains" search). You have to add those wildcards around your argument, ie:
cursor.execute(statement, {'tq': "%{}%".format(target)})
Also note that cursor.fetchall() will load your whole resultset in memory, which is asking for MemoryError. cursor objects are iterables and it's safer to use them that way:
cursor.execute(statement, {'tq': "%{}%".format(target)})
for row in cursor:
do_something_with(row)
I have troubles using a simple sql statement with the operator IN through pymssql.
Here is a sample :
import pymssql
conn = pymssql.connect(server='myserver', database='mydb')
cursor = conn.cursor()
req = "SELECT * FROM t1 where id in (%s)"
cursor.execute(req, tuple(range(1,10)))
res = cursor.fetchall()
Surprisingly only the first id is returned and I can't figure out why.
Does anyone encounter the same behavior ?
You're trying to pass nine ID values to the query and you only have one placeholder. You can get nine placeholders by doing this:
ids = range(1,10)
placeholders = ','.join('%s' for i in ids)
req = "SELECT * FROM t1 where id in ({})".format(placeholders)
cursor.execute(req, ids)
res = cursor.fetchall()
As an aside, you don't necessarily need a tuple here. A list will work fine.
It looks like you are only passing SELECT * FROM t1 where id in (1). You call execute with the tuple but the string only has one formatter. To pass all values, call execute like this:
cursor.execute(req, (tuple(range(1,10)),))
This will pass the tuple as first argument to the string to format.
EDIT: Regarding the executeone/many() thing, if you call executemany and it returns the last instead of the first id, it seems that execute will run the query 10 times as it can format the string with 10 values. The last run will then return the last id.
I'm trying to do a very simple search from a list of terms
terms = ['term1', 'term2', 'term3']
How do I programmatically go through the list of terms and construct the conditions from the list of terms so that I can make the query using filter and or_ or _and?
query.filter(or_(#something constructed from terms))
If you have a list of terms and want to find rows where a field matches one of them, then you could use the in_() method:
terms = ['term1', 'term2', 'term3']
query.filter(Cls.field.in_(terms))
If you want to do something more complex, then or_() and and_() take ClauseElement objects as parameters. ClauseElement and its subclasses basically represent the SQL AST of your query. Typically, you create clause elements by invoking a comparison operator on Column or InstrumentedAttribute objects:
# Create the clause element
clause = (users_table.columns['name'] == "something")
# you can also use the shorthand users_table.c.name
# The clause is a binary expression ...
print(type(clause))
# <class 'sqlalchemy.sql.expression._BinaryExpression'>
# ... that compares a column for equality with a bound value.
print(type(clause.left), clause.operator, type(clause.right))
# <class 'sqlalchemy.schema.Column'>, <built-in function eq>,
# <class 'sqlalchemy.sql.expression._BindParamClause'>
# str() compiles it to SQL
print(str(clause))
# users.name = ?
# You can also do that with ORM attributes
clause = (User.name == "something")
print(str(clause))
# users.name = ?
You can handle clause elements representing your conditions like any Python objects, put them into lists, compose them into other clause elements, etc. So you can do something like this:
# Collect the separate conditions to a list
conditions = []
for term in terms:
conditions.append(User.name == term)
# Combine them with or to a BooleanClauseList
condition = or_(*conditions)
# Can now use the clause element as a predicate in queries
query = query.filter(condition)
# or to view the SQL fragment
print(str(condition))
# users.name = ? OR users.name = ? OR users.name = ?
Assuming that your terms variable contains valid SQL statement fragments, you can simply pass terms preceded by an asterisk to or_ or and_:
>>> from sqlalchemy.sql import and_, or_
>>> terms = ["name='spam'", "email='spam#eggs.com'"]
>>> print or_(*terms)
name='spam' OR email='spam#eggs.com'
>>> print and_(*terms)
name='spam' AND email='spam#eggs.com'
Note that this assumes that terms contains only valid and properly escaped SQL fragments, so this is potentially unsafe if a malicious user can access terms somehow.
Instead of building SQL fragments yourself, you should let SQLAlchemy build parameterised SQL queries using other methods from sqlalchemy.sql. I don't know whether you have prepared Table objects for your tables or not; if so, assume that you have a variable called users which is an instance of Table and it describes your users table in the database. Then you can do the following:
from sqlalchemy.sql import select, or_, and_
terms = [users.c.name == 'spam', users.c.email == 'spam#eggs.com']
query = select([users], and_(*terms))
for row in conn.execute(query):
# do whatever you want here
Here, users.c.name == 'spam' will create an sqlalchemy.sql.expression._BinaryExpression object that records that this is a binary equality relation between the name column of the users table and a string literal that contains spam. When you convert this object to a string, you will get an SQL fragment like users.name = :1, where :1 is a placeholder for the parameter. The _BinaryExpression object also remembers the binding of :1 to 'spam', but it won't insert it until the SQL query is executed. When it is inserted, the database engine will make sure that it is properly escaped. Suggested reading: SQLAlchemy's operator paradigm
If you only have the database table but you don't have a users variable that describes the table, you can create it yourself:
from sqlalchemy import Table, MetaData, Column, String, Boolean
metadata = MetaData()
users = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
Column('email', String),
Column('active', Integer)
)
Alternatively, you can use autoloading which queries the database engine for the structure of the database and builds users automatically; obviously this is more time-consuming:
users = Table('users', metadata, autoload=True)
I had the same issue in"SQLAlchemy: an efficient/better select by primary keys?":
terms = ['one', 'two', 'three']
clauses = or_( * [Table.field == x for x in terms] )
query = Session.query(Table).filter(clauses)
You can use the "Conjunctions" documentation to combine conditions And, Or and not.
I'm trying to create an SQLite 3 database from Python. I have a few types I'd like to insert into each record: A float, and then 3 groups of n floats, currently a tuple but could be an array or list.. I'm not well-enough versed in Python to understand all the differences. My problem is the INSERT statement.
DAS = 12345
lats = (42,43,44,45)
lons = (10,11,12,13)
times = (1,2,3,4,5,6,7,8,9)
import sqlite3
connection = sqlite3.connect("test.db")
cursor = connection.cursor()
cursor.execute( "create table foo(DAS LONG PRIMARY KEY,lats real(4),lons real(4), times real(9) )" )
I'm not sure what comes next. Something along the lines of:
cmd = 'INSERT into foo values (?,?,?,?), ..."
cursor.execute(cmd)
How should I best build the SQL insert command given this data?
The type real(4) does not mean an array/list/tuple of 4 reals; the 4 alters the 'real' type. However, SQLite mostly ignores column types due to its manifest typing, but they can still affect column affinity.
You have a few options, such as storing the text representation (from repr) or using four columns, one for each.
You can modify this with various hooks provided by the Python's SQLite library to handle some of the transformation for you, but separate columns (with functions to localize and handle various statements, so you don't repeat yourself) is probably the easiest to work with if you need to search/etc. in SQL on each value.
If you do store a text representation, ast.literal_eval (or eval, under special conditions) will convert back into a Python object.
Something like this:
db = sqlite3.connect("test.db")
cursor = db.cursor()
cursor.execute("insert into foo values (?,?,?,?)", (val1, val2, val3, val4))
db.commit() # Autocommit is off by default (and rightfully so)!
Please note, that I am not using string formatting to inject actual data into the query, but instead make the library do this work for me. That way the data is quoted and escaped correctly.
EDIT: Obviously, considering your database schema, it doesn't work. It is impractical to attempt to store a collection-type value in a single field of a sqlite database. If I understand you correctly, you should just create a separate column for every value you are storing in the single row. That will be a lot of columns, sure, but that's the most natural way to do it.
(A month later), two steps:
1. flatten e.g. DAS lats lons times to one long list, say 18 long
2. generate "Insert into tablename xx (?,?,... 18 question marks )" and execute that.
Test = 1
def flatten( *args ):
""" 1, (2,3), [4,5] -> [1 2 3 4 5] """
# 1 level only -- SO [python] [flatten] zzz
all = []
for a in args:
all.extend( a if hasattr( a, "__iter__" ) else [a] )
if Test: print "test flatten:", all
return all
def sqlinsert( db, tablename, *args ):
flatargs = flatten( *args ) # one long list
ncol = len(flatargs)
qmarks = "?" + (ncol-1) * ",?"
insert = "Insert into tablename %s values (%s)" % (tablename, qmarks)
if Test: print "test sqlinsert:", insert
if db:
db.execute( insert, flatargs )
# db.executemany( insert, map( flatargs, rows ))
return insert
#...............................................................................
if __name__ == "__main__":
print sqlinsert( None, "Table", "hidiho", (4,5), [6] )