I'm having a problem executing this SQL statement with a python list injection. I'm new to teradata SQL, and I'm not sure if this is the appropriate syntax for injecting a list into the where clause.
conn = teradatasql.connect(host='PROD', user='1234', password='1234', logmech='LDAP')
l = ["Comp-EN Routing", "Comp-COLLABORATION"]
l2 = ["PEO", "TEP"]
l3 = ["TCV"]
crsr = conn.cursor()
query = """SELECT SOURCE_ORDER_NUMBER
FROM DL_.BV_DETAIL
WHERE (LEVEL_1 IN ? AND LEVEL_2 IN ?) or LEVEL_3 IN ?"""
crsr.executemany(query, [l,l2,l3])
conn.autocommit = True
I keep getting this error
Version 17.0.0.2] [Session 308831600] [Teradata Database] [Error 3939] There is a mismatch between the number of parameters specified and the number of parameters required.
Late to answer this, but if I found the question someone else will in the future too.
executemany in teradatasql requires that second parameter to be a "sequence of sequences". The most common type of sequence we generally use in Python is a list. Essentially you need a list that contains, for each element in the list, a list.
In your case this may look like:
myListOfLists=[['level1valueA','level1valueA','level3valueA'],['level1valueB','level1valueB','level3valueB']]
Your SQL statement will be executed twice, once for each list in your list.
In your case though I suspect you are wanting to find any combination of the values that you have stored in your three lists which is entirely different ball of wax and is going to take some creativity (generate a list of list with all possible combinations and submit to executemany OR construct a SQL statement that can take in multiple comma delimited lists of values, form a cartesian product, and test for hits)
Want to add some regarding SELECT statement and executemany method: to retrieve all records returned by your query you will need to call .nextset() followed by .fetchall() as many times as it will become False. First .fetchall() will give you only first result (first list of parameters specified).
...
with teradatasql.connect(connectionstring) as conn:
with conn.cursor() as cur:
cur.executemany("SELECT COL1 FROM THEDATABASE.THETABLE WHERE COL1 = ?;",[['A'],['B']])
result=cur.fetchall() # will bring you only rows matching 'A'
if (cur.nextset()):
result2=cur.fetchall() # results for 'B'
...
Related
I have a Oracle SQL query:
SELECT * from table1 WHERE deliveredDate = ? AND productID IN (?,?,?,...);
I would like to pass a single variable to deliveredDate and a list with length unknown to the productID using cx_Oracle and Python
From the Oracle Using Bind guide (https://cx-oracle.readthedocs.io/en/latest/user_guide/bind.html) I understand that you can bind either the single variable or list of items, but I'm not sure if we can bind both.
Please help me with this issue.
Thank you.
Of course you can, but convert the notation for bind variables from ? to :-preceeded integers such as
import pandas as pd
import cx_Oracle
import datetime
conn = cx_Oracle.connect('un/pwd#ip:port/db')
cur = conn.cursor()
sql = """
SELECT *
FROM table1
WHERE deliveredDate = :0 AND productID IN (:1,:2)
"""
cur.execute(sql,[datetime.datetime(2022, 5, 3),1,2])
res = cur.fetchall()
print(res)
The key part of your question was the 'unknown length' for the IN clause. The cx_Oracle documentation Binding Multiple Values to a SQL WHERE IN Clause shows various solutions each with some pros & cons depending on size of the list and the number of times the statement will be executed. For most cases you will not want to bind to a single placeholder in your statement IN list because of performance implications. If there is an upper bound on the size of the IN list, then put that many placeholders and bind None for all unknown values. The doc example explains it better:
cursor.execute("""
select employee_id, first_name, last_name
from employees
where last_name in (:name1, :name2, :name3, :name4, :name5)""",
name1="Smith", name2="Taylor", name3=None, name4=None, name5=None)
for row in cursor:
print(row)
(This uses keyword parameters to match the bind placeholders, but you can use a list instead).
Other solutions are shown in that doc link.
I need conditionaly update Oracle table from my Python code. It's a simple piece of code, but I encountered cx_Oracle.DatabaseError: ORA-01036: illegal variable name/number with following attempts
id_as_list = ['id-1', 'id-2'] # list of row IDs in the DB table
id_as_list_of_tuples = [('id-1'), ('id-2')] # the same as list of tuples
sql_update = "update my_table set processed = 1 where object_id = :1"
# then when I tried any of following commands, result was "illegal variable name/number"
cursor.executemany(sql_update, id_as_list) # -> ends with error
cursor.executemany(sql_update, id_as_list_of_tuples) # -> ends with error
for id in id_as_list:
cursor.execute(sql_update, id) # -> ends with error
Correct solution was to use list of dictionaries and the key name in the SQL statement:
id_as_list_of_dicts = [{'id': 'id-1'}, {'id': 'id-2'}]
sql_update = "update my_table set processed = 1 where object_id = :id"
cursor.executemany(sql_update, id_as_list_of_dicts) # -> works
for id in id_as_list_of_dicts:
cursor.execute(sql_update, id) # -> also works
I've found some helps and tutorials like this and they all used ":1, :2,..." syntax (but on the other hand I haven't found any example with update and cx_Oracle). Although my issue has been solved with help of dictionaries I wonder if it's common way of update or if I do something wrong in the ":1, :2,..." syntax.
Oracle 12c, Python 3.7, cx_Oracle 7.2.1
You can indeed bind with dictionaries but the overhead of creating the dictionaries can be undesirable. You need to make sure you create a list of sequences when using executemany(). So in your case, you want something like this instead:
id_as_list = [['id-1'], ['id-2']] # list of row IDs in the DB table
id_as_list_of_tuples = [('id-1',), ('id-2',)] # the same as list of tuples
In the first instance you had a list of strings. Strings are sequences in their own right so in that case cx_Oracle was expecting 4 bind variables (the number of characters in each string).
In the second instance you had the same data as the first instance -- as you were simply including parentheses around the strings, not creating tuples! You need the trailing comma as shown in my example to create tuples as you thought you were creating!
I have a DB with ID/Topic/Definition columns. When a select query is made, with possibly hundreds of parameters, I would like the fetchall call to also return the topic of any non-existent rows with a default text (i.e. "Not Found").
I realize this could be done in a loop, but that would query the DB every cycle and have a significant performance hit. With the parameters joined by "OR" in a single select statement the search is nearly instantaneous.
Is there a way to get a return of the query (topic) with default text for non-existent rows in SQLite?
Table Structure (named "dictionary")
ID|Topic|Definition
1|wd1|def1
2|wd3|def3
Sample Query
SELECT Topic,Definition FROM dictionary WHERE Topic = "wd1" or Topic = "wd2" or topic = "wd3"'
Desired Return
[(wd1, def1), (wd2, "Not Found"), (wd3, def3)]
To get data like wd2 out of a query, such data must be in the database in the first place.
You could put it into a temporary table, or use a common table expression.
To include rows without a match, use an outer join:
WITH IDs(ID) AS ( VALUES ('wd1'), ('wd2'), ('wd3') )
SELECT Topic,
IFNULL(Definition, 'Not Found') AS Definition
FROM IDs
LEFT JOIN dictionary USING (ID);
Both methods return a list of the returned items of the query, did I miss something here, or they have identical usages indeed?
Any differences performance-wise?
If you are using the default cursor, a MySQLdb.cursors.Cursor, the entire result set will be stored on the client side (i.e. in a Python list) by the time the cursor.execute() is completed.
Therefore, even if you use
for row in cursor:
you will not be getting any reduction in memory footprint. The entire result set has already been stored in a list (See self._rows in MySQLdb/cursors.py).
However, if you use an SSCursor or SSDictCursor:
import MySQLdb
import MySQLdb.cursors as cursors
conn = MySQLdb.connect(..., cursorclass=cursors.SSCursor)
then the result set is stored in the server, mysqld. Now you can write
cursor = conn.cursor()
cursor.execute('SELECT * FROM HUGETABLE')
for row in cursor:
print(row)
and the rows will be fetched one-by-one from the server, thus not requiring Python to build a huge list of tuples first, and thus saving on memory.
Otherwise, as others have already stated, cursor.fetchall() and list(cursor) are essentially the same.
cursor.fetchall() and list(cursor) are essentially the same. The different option is to not retrieve a list, and instead just loop over the bare cursor object:
for result in cursor:
This can be more efficient if the result set is large, as it doesn't have to fetch the entire result set and keep it all in memory; it can just incrementally get each item (or batch them in smaller batches).
list(cursor) works because a cursor is an iterable; you can also use cursor in a loop:
for row in cursor:
# ...
A good database adapter implementation will fetch rows in batches from the server, saving on the memory footprint required as it will not need to hold the full result set in memory. cursor.fetchall() has to return the full list instead.
There is little point in using list(cursor) over cursor.fetchall(); the end effect is then indeed the same, but you wasted an opportunity to stream results instead.
A (MySQLdb/PyMySQL-specific) difference worth noting when using a DictCursor is that list(cursor) will always give you a list, while cursor.fetchall() gives you a list unless the result set is empty, in which case it gives you an empty tuple. This was the case in MySQLdb and remains the case in the newer PyMySQL, where it will not be fixed for backwards-compatibility reasons. While this isn't a violation of Python Database API Specification, it's still surprising and can easily lead to a type error caused by wrongly assuming that the result is a list, rather than just a sequence.
Given the above, I suggest always favouring list(cursor) over cursor.fetchall(), to avoid ever getting caught out by a mysterious type error in the edge case where your result set is empty.
You could use list comprehensions to bring the item in your tuple into a list:
conn = mysql.connector.connect()
cursor = conn.cursor()
sql = "SELECT column_name FROM db.table_name;"
cursor.execute(sql)
results = cursor.fetchall()
# bring the first item of the tuple in your results here
item_0_in_result = [_[0] for _ in results]
I have a list of ids in python. For example:
x = [1,2,3,4,5,6]
And i want to select a list of records in my (mysql ) data-base under the condition that the ids of these records are in x. something like below:
SELECT * FROM mytable WHERE id IN x
but I don't know who I can do this in python. I have seen some examples using %s in their sql string. However this does not work when the variable is a list. does anyone know how I can do this?
Thanks
Try something like this:
'(%s)' % ','.join(map(str,x))
This will give you a string that you could use to send to MySql as a valid IN clause:
(1,2,3,4,5,6)
Well, if all of those are known to be numbers of good standing, then you can simply call
"SELECT * FROM mytable WHERE ID IN ({0})".format(','.join(x))
If you know that they are numbers but any of them might have been from the user, then I might use:
"SELECT * FROM mytable WHERE ID IN ({0})".format(','.join(list(map(int,x))))
format will perform the replacement at the appropriate index. join is used so that you don't have the []. list converts everything to a list, map applies a function to a list/tuple/iterable. In Python 3, however, map returns a generator, which isn't what you need. You need a list. So, list(map(x,y)) will return the list form of map(x,y).