I have a list of ids in python. For example:
x = [1,2,3,4,5,6]
And i want to select a list of records in my (mysql ) data-base under the condition that the ids of these records are in x. something like below:
SELECT * FROM mytable WHERE id IN x
but I don't know who I can do this in python. I have seen some examples using %s in their sql string. However this does not work when the variable is a list. does anyone know how I can do this?
Thanks
Try something like this:
'(%s)' % ','.join(map(str,x))
This will give you a string that you could use to send to MySql as a valid IN clause:
(1,2,3,4,5,6)
Well, if all of those are known to be numbers of good standing, then you can simply call
"SELECT * FROM mytable WHERE ID IN ({0})".format(','.join(x))
If you know that they are numbers but any of them might have been from the user, then I might use:
"SELECT * FROM mytable WHERE ID IN ({0})".format(','.join(list(map(int,x))))
format will perform the replacement at the appropriate index. join is used so that you don't have the []. list converts everything to a list, map applies a function to a list/tuple/iterable. In Python 3, however, map returns a generator, which isn't what you need. You need a list. So, list(map(x,y)) will return the list form of map(x,y).
Related
I trying to use sqlite3 query like this one:
books = c.execute("SELECT * FROM books WHERE id IN (?)", session["cart"])
Where session["cart"] is a dynamic list.
For example:
session["cart"] = ['1', '7']
I get an error as given in title while trying to execute it.
How may I insert whole list in the question mark placeholder?
I can't use multiple question marks, because their number may vary.
Does anyone has any clue? Or maybe better idea how to achieve so?
You will have to format your python list into something sql can read, this is easily done with str.join
stringified_list = ", ".join(session["cart"])
books = c.execute("SELECT * FROM books WHERE id IN (?)", stringified_list )
in case you are unfamiliar, str.join(iterable) will make a string with all the string representations of the items in the iterable separated by the str you called join on, i.e. '---'.join([1,2,3])
will produce "1---2---3" (a string)
I'm having a problem executing this SQL statement with a python list injection. I'm new to teradata SQL, and I'm not sure if this is the appropriate syntax for injecting a list into the where clause.
conn = teradatasql.connect(host='PROD', user='1234', password='1234', logmech='LDAP')
l = ["Comp-EN Routing", "Comp-COLLABORATION"]
l2 = ["PEO", "TEP"]
l3 = ["TCV"]
crsr = conn.cursor()
query = """SELECT SOURCE_ORDER_NUMBER
FROM DL_.BV_DETAIL
WHERE (LEVEL_1 IN ? AND LEVEL_2 IN ?) or LEVEL_3 IN ?"""
crsr.executemany(query, [l,l2,l3])
conn.autocommit = True
I keep getting this error
Version 17.0.0.2] [Session 308831600] [Teradata Database] [Error 3939] There is a mismatch between the number of parameters specified and the number of parameters required.
Late to answer this, but if I found the question someone else will in the future too.
executemany in teradatasql requires that second parameter to be a "sequence of sequences". The most common type of sequence we generally use in Python is a list. Essentially you need a list that contains, for each element in the list, a list.
In your case this may look like:
myListOfLists=[['level1valueA','level1valueA','level3valueA'],['level1valueB','level1valueB','level3valueB']]
Your SQL statement will be executed twice, once for each list in your list.
In your case though I suspect you are wanting to find any combination of the values that you have stored in your three lists which is entirely different ball of wax and is going to take some creativity (generate a list of list with all possible combinations and submit to executemany OR construct a SQL statement that can take in multiple comma delimited lists of values, form a cartesian product, and test for hits)
Want to add some regarding SELECT statement and executemany method: to retrieve all records returned by your query you will need to call .nextset() followed by .fetchall() as many times as it will become False. First .fetchall() will give you only first result (first list of parameters specified).
...
with teradatasql.connect(connectionstring) as conn:
with conn.cursor() as cur:
cur.executemany("SELECT COL1 FROM THEDATABASE.THETABLE WHERE COL1 = ?;",[['A'],['B']])
result=cur.fetchall() # will bring you only rows matching 'A'
if (cur.nextset()):
result2=cur.fetchall() # results for 'B'
...
I am working with python lists that have varying number of items and need to build out a dynamic string (database insert statements) based on the number of items.
For example:
l1 = ['one','two','three']
The desired output is something like this. Was hoping people had more pythonic suggestions on how to approach this in the string.
"insert into {table} values ('{c1}', '{c2}', '{c3}')".format(table='tabName', c1=l1[0], c2=l1[1], c3=l1[2])
But the above obviously wont work if I now have a list with a 4th element.
How would you suggest that I build out the string if i have a variable number of items in my list (and thus variable number of database columns to insert into)?
Thanks in advance for any advice.
What you're trying to do is a very bad idea, as well as impossible to do the way you're attempting it.
Usually, you don't want SQL queries to be dynamic at all; you want a fixed string like this:
sql = 'INSERT INTO tabName VALUES(?, ?, ?)'
db.execute(sql, l1)
Also, you usually want to add the column names, not rely on the column ordering (according to SQL standards, the order completely arbitrary, although in practice nearly every database will always use the order in which the columns were created).
In rare cases (e.g., when you're building a database administration tool), you do need dynamic SQL, but you still want to use parameters. You do this by dynamically putting parameters into the query, and then passing the list. Something like this:
params = ', '.join('?' for _ in l1)
sql = 'INSERT INTO {table} VALUES({params})'.format(table=tabName, params=params)
db.execute(sql, l1)
(But of course if tabName is a user-entered string, this is at least as dangerous as not using parameters.)
What is the correct method to have the list (countryList) be available via %s in the SQL statement?
# using psycopg2
countryList=['UK','France']
sql='SELECT * from countries WHERE country IN (%s)'
data=[countryList]
cur.execute(sql,data)
As it is now, it errors out after trying to run "WHERE country in (ARRAY[...])". Is there a way to do this other than through string manipulation?
Thanks
For the IN operator, you want a tuple instead of list, and remove parentheses from the SQL string.
# using psycopg2
data=('UK','France')
sql='SELECT * from countries WHERE country IN %s'
cur.execute(sql,(data,))
During debugging you can check that the SQL is built correctly with
cur.mogrify(sql, (data,))
To expland on the answer a little and to address named parameters, and converting lists to tuples:
countryList = ['UK', 'France']
sql = 'SELECT * from countries WHERE country IN %(countryList)s'
cur.execute(sql, { # You can pass a dict for named parameters rather than a tuple. Makes debugging hella easier.
'countryList': tuple(countryList), # Converts the list to a tuple.
})
You could use a python list directly as below. It acts like the IN operator in SQL and also handles a blank list without throwing any error.
data=['UK','France']
sql='SELECT * from countries WHERE country = ANY (%s)'
cur.execute(sql,(data,))
source:
http://initd.org/psycopg/docs/usage.html#lists-adaptation
Since the psycopg3 question was marked as a duplicate, I'll add the answer to that here too.
In psycopg3, you can not use in %s with a tuple, like you could in psycopg2. Instead you have to use ANY() and wrap your list inside another list:
conn.execute("SELECT * FROM foo WHERE id = ANY(%s)", [[10,20,30]])
Docs: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html#you-cannot-use-in-s-with-a-tuple
I am trying to select multiple columns, but not all of the columns, from the database. All of the columns I want to select are going to start with "word".
So in pseudocode I'd like to do this:
SELECT "word%" from searchterms where onstate = 1;
More or less. I am not finding any documentation on how to do this - is it possible in MySQL? Basically, I am trying to store a list of words in a single row, with an identifier, and I want to associate all of the words with that identifier when I pull the records. All of the words are going to be joined as a string and passed to another function in an array/dictionary with their identifier.
I am trying to make as FEW database calls as possible to keep speedy code.
Ok, here's another question for you guys:
There are going to be a variable number of columns with the name "word" in them. Would it be faster to do a separate database call for each row, with a generated Python query per row, or would it be faster to simply SELECT *, and only use the columns I needed? Is it possible to say SELECT * NOT XYZ?
No, SQL doesn't provide you with any syntax to do such a select.
What you can do is ask MySQL for a list of column names first, then generate the SQL query from that information.
SELECT column_name
FROM information_schema.columns
WHERE table_name = 'your_table'
AND column_name LIKE 'word%'
let's you select the column names. Then you can do, in Python:
"SELECT * FROM your_table WHERE " + ' '.join(['%s = 1' % name for name in columns])
Instead of using string concatenation, I would recommend using SQLAlchemy instead to do the SQL generating for you.
However, if all you are doing is limit the number of columns there is no need to do a dynamic query like this at all. The hard work for the database is selecting the rows; it makes little difference to send you 5 columns out of 10, or all 10.
In that case just use a "SELECT * FROM ..." and use Python to pick out the columns from the result set.
No, you cannot dynamically produce the list of columns to be selected. It will have to be hardcoded in your final query.
Your current query would produce a result set with one column and the value of that column would be the string "word%" in all rows that satisfy the condition.
You can generate the list of column names first by using
SHOW COLUMNS IN tblname LIKE "word%"
Then loop through the cursor and generate SQL statement uses all the columns from the query above.
"SELECT {0} FROM searchterms WHERE onstate = 1".format(', '.join(columns))
This could be helpful: MySQL wildcard in select
In conclusion it is not possible in MySQL directly.
What you could do as a dirty workaround is get all the column names from the table with an initial query (http://dev.mysql.com/doc/refman/5.0/en/show-columns.html) and then compare in python if the name matches your pattern. Afterwards you could do the MySQL select statement with the found column names like this:
SELECT word1, word2, word3 from searchterms where onstate = 1;