I need conditionaly update Oracle table from my Python code. It's a simple piece of code, but I encountered cx_Oracle.DatabaseError: ORA-01036: illegal variable name/number with following attempts
id_as_list = ['id-1', 'id-2'] # list of row IDs in the DB table
id_as_list_of_tuples = [('id-1'), ('id-2')] # the same as list of tuples
sql_update = "update my_table set processed = 1 where object_id = :1"
# then when I tried any of following commands, result was "illegal variable name/number"
cursor.executemany(sql_update, id_as_list) # -> ends with error
cursor.executemany(sql_update, id_as_list_of_tuples) # -> ends with error
for id in id_as_list:
cursor.execute(sql_update, id) # -> ends with error
Correct solution was to use list of dictionaries and the key name in the SQL statement:
id_as_list_of_dicts = [{'id': 'id-1'}, {'id': 'id-2'}]
sql_update = "update my_table set processed = 1 where object_id = :id"
cursor.executemany(sql_update, id_as_list_of_dicts) # -> works
for id in id_as_list_of_dicts:
cursor.execute(sql_update, id) # -> also works
I've found some helps and tutorials like this and they all used ":1, :2,..." syntax (but on the other hand I haven't found any example with update and cx_Oracle). Although my issue has been solved with help of dictionaries I wonder if it's common way of update or if I do something wrong in the ":1, :2,..." syntax.
Oracle 12c, Python 3.7, cx_Oracle 7.2.1
You can indeed bind with dictionaries but the overhead of creating the dictionaries can be undesirable. You need to make sure you create a list of sequences when using executemany(). So in your case, you want something like this instead:
id_as_list = [['id-1'], ['id-2']] # list of row IDs in the DB table
id_as_list_of_tuples = [('id-1',), ('id-2',)] # the same as list of tuples
In the first instance you had a list of strings. Strings are sequences in their own right so in that case cx_Oracle was expecting 4 bind variables (the number of characters in each string).
In the second instance you had the same data as the first instance -- as you were simply including parentheses around the strings, not creating tuples! You need the trailing comma as shown in my example to create tuples as you thought you were creating!
Related
I wrote a python script to update currency exchange rates using API calls. I successfully parsed the json results and extracted individual exchange rates as floats. I am however struggling with formatting/implementing an SQL table update loop.
Here is a code snippet that is tripping me up, assume that val = an actual exchange rate variable that is supplied from the API fetching/parsing section of the code:
mycursor = mydb.cursor()
sql = "UPDATE currencies SET coefficient = %s"
val = 0.03137
mycursor.execute(sql, val)
mydb.commit()
Running this gives me the following error:
Could not process parameters: float(0.03137), it must be of type list, tuple or dict:
I do not even know what to search for in order to reach an explanation that I understand and that helps me implement what I want correctly.
From: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html
Note
In Python, a tuple containing a single value must include a comma. For
example, ('abc') is evaluated as a scalar while ('abc',) is evaluated
as a tuple.
Thus in my example the expression that works is:
val = (0.03137, )
I have created a python class, and one of my methods is meant to take in either a single ID number or a list of ID numbers. The function will then use the ID numbers to query from a table in BigQuery using a .sql script. Currently, the function works fine for a single ID number using the following:
def state_data(self, state, id_number):
if state == 'NY':
sql_script = self.sql_scripts['get_data_ny']
else:
sql_script = self.sql_scripts['get_data_rest']
sql_script = sql_script.replace('##id_number##', id_number)
I'm having issues with passing in multiple ID numbers at once. There are 3 different ways that I've tried without success:
The above method, passing in the multiple ID numbers as a tuple to use with WHERE ID_NUM IN('##id_number##'). This doesn't work, as when the .sql script gets called, a syntax error is returned, as parentheses and quotes are automatically added. For example, the SQL statement attempts to run as WHERE ID_NUM IN('('123', '124')'). This would run fine without one of the two sets of parentheses and quotes, but no matter what I try to pass in, they always get added.
The second technique I have tried is to create a table, populate it with the passed in ID numbers, and then join with the larger table in BQ. It goes as follows:
CREATE OR REPLACE TABLE ID_Numbers
(
ID_Number STRING
);
INSERT INTO ID_Numbers (ID_Number)
VALUES ('##id_number##');
-- rest of script is a simple left join of the above created table with the BQ table containing the data for each ID
This again works fine for single ID numbers, but passing in multiple VALUES (in this case ID Numbers) would require a ('##id_number##') per unique ID. One thing that I have not yet attempted - to assign a variable to each unique ID and pass each one in as a new VALUE. I am not sure if this technique will work.
The third technique I've tried is to include the full SQL query in the function, rather than calling a .sql script. The list of ID numbers get passed in as tuple, and the query goes as follows:
id_nums = tuple(id_number)
query = ("""SELECT * FROM `data_table`
WHERE ID_NUM IN{}""").format(id_nums)
This technique also does not work, as I get the following error:
AttributeError: 'QueryJob' object has no attribute 'format'.
I've attempted to look into this error but I cannot find anything that helps me out effectively.
Finally, I'll note that none of the posts asking the same or similar questions have solved my issues so far.
I am looking for any and all advice for a way that I can successfully pass a variable containing multiple ID numbers into my function that ultimately calls and runs a BQ query.
You should be able to use *args to get the id_numbers as a sequence and f-strings and str.join() to build the SQL query:
class MyClass:
def state_data(self, state, *id_numbers):
print(f"{state=}")
query = f"""
SELECT * FROM `data_table`
WHERE ID_NUM IN ({", ".join(str(id_number) for id_number in id_numbers)})
"""
print(query)
my_class = MyClass()
my_class.state_data("some state", 123)
my_class.state_data("some more state", 123, 124)
On my machine, this prints:
➜ sql python main.py
state='some state'
SELECT * FROM `data_table`
WHERE ID_NUM IN (123)
state='some more state'
SELECT * FROM `data_table`
WHERE ID_NUM IN (123, 124)
I'm having a problem executing this SQL statement with a python list injection. I'm new to teradata SQL, and I'm not sure if this is the appropriate syntax for injecting a list into the where clause.
conn = teradatasql.connect(host='PROD', user='1234', password='1234', logmech='LDAP')
l = ["Comp-EN Routing", "Comp-COLLABORATION"]
l2 = ["PEO", "TEP"]
l3 = ["TCV"]
crsr = conn.cursor()
query = """SELECT SOURCE_ORDER_NUMBER
FROM DL_.BV_DETAIL
WHERE (LEVEL_1 IN ? AND LEVEL_2 IN ?) or LEVEL_3 IN ?"""
crsr.executemany(query, [l,l2,l3])
conn.autocommit = True
I keep getting this error
Version 17.0.0.2] [Session 308831600] [Teradata Database] [Error 3939] There is a mismatch between the number of parameters specified and the number of parameters required.
Late to answer this, but if I found the question someone else will in the future too.
executemany in teradatasql requires that second parameter to be a "sequence of sequences". The most common type of sequence we generally use in Python is a list. Essentially you need a list that contains, for each element in the list, a list.
In your case this may look like:
myListOfLists=[['level1valueA','level1valueA','level3valueA'],['level1valueB','level1valueB','level3valueB']]
Your SQL statement will be executed twice, once for each list in your list.
In your case though I suspect you are wanting to find any combination of the values that you have stored in your three lists which is entirely different ball of wax and is going to take some creativity (generate a list of list with all possible combinations and submit to executemany OR construct a SQL statement that can take in multiple comma delimited lists of values, form a cartesian product, and test for hits)
Want to add some regarding SELECT statement and executemany method: to retrieve all records returned by your query you will need to call .nextset() followed by .fetchall() as many times as it will become False. First .fetchall() will give you only first result (first list of parameters specified).
...
with teradatasql.connect(connectionstring) as conn:
with conn.cursor() as cur:
cur.executemany("SELECT COL1 FROM THEDATABASE.THETABLE WHERE COL1 = ?;",[['A'],['B']])
result=cur.fetchall() # will bring you only rows matching 'A'
if (cur.nextset()):
result2=cur.fetchall() # results for 'B'
...
I basically have this test2 file that has domain information that I want to use so I strip the additional stuff and get just the domain names as new_list.
What I want to then do is query a database with these domain names, pull the name and severity score and then (the part I'm really having a hard time with) getting a stored list (or tuple) that I can use that consists of the pulled domains and severity score.
It is a psql database for reference but my problem lies more on the managing after the query.
I'm still really new to Python and mainly did a bit of Java so my code probably looks terrible, but I've tried converting to strings and tried appending to a list at the end but I am quite unsuccessful with most of it.
def get_new():
data = []
with open('test2.txt', 'r') as file:
data = [line.rstrip('\n') for line in open('test2.txt')]
return data
new_list = get_new()
def db_query():
cur = connect.cursor()
query ="SELECT name, de.severity_score FROM domains d JOIN ips i ON i.domain_id = d.id JOIN domains_extended de ON de.domain_id = d.id WHERE name = '"
for x in new_list:
var = query + x + "'"
cur.execute(var)
get = cur.fetchall()
# STORE THE LOOPED QUERIES INTO A VARIABLE OF SOME KIND (problem area)
print(results)
cur.close()
connect.close()
db_query()
Happy place: Takes domain names from file, uses those domain names a part of the query parameters to get severity score associated, then stores it into a variable of some sort so that I can use those values later (in a loop or some logic).
I've tried everything I could think of and ran into errors with it being a query that I'm trying to store, lists won't combine, etc.
I would make sure that your get_new() function is returning what yous expect from that file. Just iterate on your new_data list.
There is no reference to results in your db_query() function (perhaps it is global like new_data[]) but try printing the result of the query, that is, print(get) in your for loop and see what comes out. If this works then you can create a list to append to.
Well first off in your code you are resetting your get variable in each loop. So after fixing that by initializing get = [] above your loop then adding get.extend(cur.fetchall()) into your loop instead of the current statement. You could then do something like domainNames = [row[0] for row in get] . If get is loading properly though getting the values out of it should be no problem.
How can I dynamically build an update query where the number and field names I'm updating are different without having to set up a different update query for every single update statement.
I'm new to python, but in other languages I've seen it be done with a key value pair where you would dynamically do something like:
UPDATE my_table SET key1 = value1 WHERE key2 = value2
Then you'd pass an array of key value pairs to the function and off you go.
Is the best way to do this in python to just create an update string and pass in the fields? Something like:
"UPDATE my_table SET {} = ? WHERE {} = ?".format(key1, key2)
Then I guess you'd have to separately pass in the parameters to pyodbc.executemany?
But then I'm not sure how you'd handle a variable number of fields to update. I'm sure there's a way to do this easily, so hopefully someone can clue me in.
Yes, with vanilla Python and pyodbc the basic process you've described is correct. If you have a list of column names to update
>>> col_names_to_set = ['LastName', 'FirstName']
you can build the required items for the SET clause,
>>> set_tokens = ','.join([f'{x}=?' for x in col_names_to_set])
>>> set_tokens
'LastName=?,FirstName=?'
include that in your SQL command text,
>>> sql = f"UPDATE TableName SET {set_tokens} WHERE ..."
>>> sql
'UPDATE TableName SET LastName=?,FirstName=? WHERE ...'
and then pass that statement to executemany along with your list of tuples containing the values to be updated.
Note that for safety you probably should enclose the column names in the delimiters for your SQL dialect, e.g., for SQL Server (T-SQL):
>>> set_tokens = ','.join([f'[{x}]=?' for x in col_names_to_set])
>>> set_tokens
'[LastName]=?,[FirstName]=?'