How can I dynamically build an update query where the number and field names I'm updating are different without having to set up a different update query for every single update statement.
I'm new to python, but in other languages I've seen it be done with a key value pair where you would dynamically do something like:
UPDATE my_table SET key1 = value1 WHERE key2 = value2
Then you'd pass an array of key value pairs to the function and off you go.
Is the best way to do this in python to just create an update string and pass in the fields? Something like:
"UPDATE my_table SET {} = ? WHERE {} = ?".format(key1, key2)
Then I guess you'd have to separately pass in the parameters to pyodbc.executemany?
But then I'm not sure how you'd handle a variable number of fields to update. I'm sure there's a way to do this easily, so hopefully someone can clue me in.
Yes, with vanilla Python and pyodbc the basic process you've described is correct. If you have a list of column names to update
>>> col_names_to_set = ['LastName', 'FirstName']
you can build the required items for the SET clause,
>>> set_tokens = ','.join([f'{x}=?' for x in col_names_to_set])
>>> set_tokens
'LastName=?,FirstName=?'
include that in your SQL command text,
>>> sql = f"UPDATE TableName SET {set_tokens} WHERE ..."
>>> sql
'UPDATE TableName SET LastName=?,FirstName=? WHERE ...'
and then pass that statement to executemany along with your list of tuples containing the values to be updated.
Note that for safety you probably should enclose the column names in the delimiters for your SQL dialect, e.g., for SQL Server (T-SQL):
>>> set_tokens = ','.join([f'[{x}]=?' for x in col_names_to_set])
>>> set_tokens
'[LastName]=?,[FirstName]=?'
Related
I need conditionaly update Oracle table from my Python code. It's a simple piece of code, but I encountered cx_Oracle.DatabaseError: ORA-01036: illegal variable name/number with following attempts
id_as_list = ['id-1', 'id-2'] # list of row IDs in the DB table
id_as_list_of_tuples = [('id-1'), ('id-2')] # the same as list of tuples
sql_update = "update my_table set processed = 1 where object_id = :1"
# then when I tried any of following commands, result was "illegal variable name/number"
cursor.executemany(sql_update, id_as_list) # -> ends with error
cursor.executemany(sql_update, id_as_list_of_tuples) # -> ends with error
for id in id_as_list:
cursor.execute(sql_update, id) # -> ends with error
Correct solution was to use list of dictionaries and the key name in the SQL statement:
id_as_list_of_dicts = [{'id': 'id-1'}, {'id': 'id-2'}]
sql_update = "update my_table set processed = 1 where object_id = :id"
cursor.executemany(sql_update, id_as_list_of_dicts) # -> works
for id in id_as_list_of_dicts:
cursor.execute(sql_update, id) # -> also works
I've found some helps and tutorials like this and they all used ":1, :2,..." syntax (but on the other hand I haven't found any example with update and cx_Oracle). Although my issue has been solved with help of dictionaries I wonder if it's common way of update or if I do something wrong in the ":1, :2,..." syntax.
Oracle 12c, Python 3.7, cx_Oracle 7.2.1
You can indeed bind with dictionaries but the overhead of creating the dictionaries can be undesirable. You need to make sure you create a list of sequences when using executemany(). So in your case, you want something like this instead:
id_as_list = [['id-1'], ['id-2']] # list of row IDs in the DB table
id_as_list_of_tuples = [('id-1',), ('id-2',)] # the same as list of tuples
In the first instance you had a list of strings. Strings are sequences in their own right so in that case cx_Oracle was expecting 4 bind variables (the number of characters in each string).
In the second instance you had the same data as the first instance -- as you were simply including parentheses around the strings, not creating tuples! You need the trailing comma as shown in my example to create tuples as you thought you were creating!
This question already has answers here:
How to prepare SQL query dynamically (column names too) avoiding SQL injection
(2 answers)
MySQL parameterized queries
(7 answers)
Closed 3 years ago.
I try to use UPDATE statement in pymysql to update some column's value where self.key, self.value, self.id are three variables.
cur.execute("UPDATE environment_history SET {key}=%s WHERE id=%s".format(key=self.key),
(self.value, self.id)
)
While in the above code, this leaves an opportunity for SQL Injection. We'll just have to post some data like this
{"id":"23151","key":"uuid='ac211';#","value":"abcde"}
This would update all rows, and this could be more dangerous.
I have to stop this from happening.
I've tried some unpractical solutions, one of them is:
cur.execute("UPDATE environment_history SET %s=%s WHERE id=%s",
(self.key,self.value, self.id)
)
However, pymysql will escape column name to something like\'uuid\'.
And this is an SQL Syntax error since SET col_name=3 and SET `col_name`=3 are correct.
Then, how to avoid this kind of SQL Injection?
Trying to replace and escape self.key manually was the only option I can ever come up with, but that doesn't sounds like an elegant way.
My database is MySQL.
Any ideas? Thanks!
From my experience with dynamic queries like that, I'd suggest you to keep a table/key definitions somewhere and check if those are valid.
Basic idea would be to keep a dict of valid keys for tables, for example:
valid_keys = {
'table1': ['key1', 'key2', 'key3'],
'table2': ['key1', 'key2'],
...
}
and doing a simple check, for example:
keys = valid_keys[table] if table in valid_keys.keys() else None
if keys != None and key in keys:
# do your query stuff
You can place that check in a method/function, and make it even simpler, without need to repeat yourself:
# checks table and key validity
def query_data_valid(table, key):
keys = valid_keys[table] if table in valid_keys.keys() else None
if keys != None and key in keys:
return True
return False
...
# block in your query calling method
if query_data_valid(table, key):
# do your query stuff
I would suggest filtering self.key down to a known value, only then use it as a column. For example:
keywords = ["uuid", "other", "keywords"]
if self.key in keywords:
column_name = self.key
else:
column_name = keywords[0]
This doesn't answer your question, but a general rule. As explained in this post you need to use parameterized queries.
A parameterized query is a query in which placeholders used for parameters and the parameter values supplied at execution time. That means parameterized query gets compiled only once.
Incorrect (with security issues)
c.execute("SELECT * FROM foo WHERE bar = %s AND baz = %s" % (param1, param2))
Correct (with escaping)
c.execute("SELECT * FROM foo WHERE bar = {0} AND baz = {1}".format(param1, param2))
I have a script with the following:
UPDATE table
SET column to update = ?
WHERE condition = ?", "text to insert", "text to test condition"
For some reason SQL is not executing or even reading this line. When I misspell any of the reserved words or column names I do not get an error.
HOWEVER, when I have
UPDATE table
SET column to update = "text to insert"
WHERE Name = "text to test condition"
SQL behaves as expected.
The problem is the second method,which works, is not adequate for my needs. Am I missing something?
Thanks
Since this is tagged with pyodbc, I'm assuming you're trying to do run a query with parameters. Your code should probably read something like this:
pyodbc.execute(
"""
UPDATE table
SET column_to_update = ?
WHERE other_column = ?
""",
"text to put in column_to_update",
"text to test condition in other_column",
)
Please note that parameters marked with a ? must be tied to a data typed object such as a column, so they can be bound. See:
https://github.com/mkleehammer/pyodbc/wiki/Getting-started#parameters
Good luck!
I'm going to assume that you are trying to run a SQL query from some client code by passing in variables with the query, though I'm not sure of this or what language you might be using - please clarify and add tags so we can help.
SQL Server does not use ? for parameter placeholders like in other DBMS queries (say, SQLite which uses ? in the way you are trying to use them). Instead, you need to either let an ORM tool declare variables for you, and use those, or explicitly declare your own parameters and pass them in. You haven't told us the development environment you're using or what method of SQL connection and ORM (if any) you're using, but here's a quick example using the very excellent Dapper ORM in C# from here, given an open connection conn:
string val = "my value";
conn.Execute("insert MyTable (val) values(#val)", new {val});
// or: conn.Execute("insert MyTable (val) values(#val)", new {val = val});"
conn.Execute("update MyTable set val = #val where Id = #id", new {val, id = 1});
In the background, Dapper handles the mapping and creation of variables, such that the actual SQL script received by SQL Server is something closer to this:
-- first query:
declare #val nvarchar(max) = 'my value';
insert MyTable (val) values(#val);
-- second query:
declare #val nvarchar(max) = 'my value', #id int = 1;
update MyTable set val = #val where Id = #id
On the other hand, if you are just going to execute a raw query directly with a SqlConnection, try something like this (equivalent to the first query above):
// Assumes an open connection conn
string val = "my value";
using (SqlCommand cmd = conn.CreateCommand())
{
cmd.CommandText = "insert MyTable (val) values(#val)";
cmd.Parameters.AddWithValue("#val", val); // This creates the #val declaration for you when the query is executed
cmd.ExecuteNonQuery();
}
Whatever you do, parameterize your parameters, and beware of SQL injection!
Hope that helps. If you'd like a clearer example, please give us some code to show how you're passing the query to the SQL Connection for execution.
How could i use psycopg2 query parameters with on conflict?
for example, i have this data:
#This is the data for the insert query
q_values = {'customer_app_id': '35', 'email': 'my#email.com', 'access_counter': 1, 'company_name': 'twitter', 'status': 'start'}
#This is the data for the on conflict query part
conflict_values = {'access_counter': +1, 'status': 'check', 'status_info': 'already exist'}
This is the query:
insert into access (customer_app_id,email,access_counter,company_name,status)
values (%s,%s,%s,%s,%s) ON CONFLICT (customer_app_id,email) DO UPDATE SET
%s,%s,%s RETURNING *
Then i run this line:
q_values = q_values.update(conflict_values)
cursor.execute(query, q_values)
First, how at all to run on conflict with query parameters ?
Second, the update i am doing with the dict is not good since if it will be duplicate keys it will merge them, and then number of parameters wont be equal to number of values.
and access_counter +1 - on conflict i'm trying to increase the access _counter by 1
please can you help ?
Thank you!
To run on conflict with query parameter:
The %s has to be replaced by %(key value that you want to access)s. This has to be done since you are using a dictionary. So in this case the query would look as follows-
insert into access (customer_app_id,email,access_counter,company_name,status)
values (%(customer_app_id)s,%(email)s,%(access_counter)s,%(company_name)s,%(status)s) ON CONFLICT
To update the access counter:
A separate dictionary need not be used. Rather ON CONFLICT you can directly make changes in the database. So in this case the final query would look as follows-
insert into access (customer_app_id,email,access_counter,company_name,status)
values (%(customer_app_id)s,%(email)s,%(access_counter)s,%(company_name)s,%(status)s) ON CONFLICT (customer_app_id,email) DO UPDATE SET access_counter = access.access_counter + 1
In case you want to update one of your column values to a new value sent in the INSERT query then you will have to use EXCLUDED keyword.
Example(this example is not related to the question):
INSERT INTO user_details (user_id, username,email,last_login)
VALUES (%s, %s, %s,%s)
ON CONFLICT (user_id) DO UPDATE
SET last_login = EXCLUDED.last_login
As you are passing a dictionary of values (instead of a list or a tuple), I think you need to use %(name)s placeholders, not %s.
This means you can give the placeholders alternative names, so they don't conflict in the q_values dictionary.
See the documentation for details.
This question already has answers here:
imploding a list for use in a python MySQLDB IN clause
(8 answers)
Closed 1 year ago.
I want to insert a list in my database but I can't.
Here is an example of what I need:
variable_1 = "HELLO"
variable_2 = "ADIOS"
list = [variable_1,variable_2]
INSERT INTO table VALUES ('%s') % list
Can something like this be done? Can I insert a list as a value?
When I try it, an error says that is because of an error in MySQL syntax
The answer to your original question is: No, you can't insert a list like that.
However, with some tweaking, you could make that code work by using %r and passing in a tuple:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1, variable_2]
print "INSERT INTO table VALUES %r;" % (tuple(varlist),)
Unfortunately, that style of variable insertion leaves your code vulnerable to SQL injection attacks.
Instead, we recommend using Python's DB API and building a customized query string with multiple question marks for the data to be inserted:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1,variable_2]
var_string = ', '.join('?' * len(varlist))
query_string = 'INSERT INTO table VALUES (%s);' % var_string
cursor.execute(query_string, varlist)
The example at the beginning of the SQLite3 docs shows how to pass arguments using the question marks and it explains why they are necessary (essentially, it assures correct quoting of your variables).
Your question is not clear.
Do you want to insert the list as a comma-delimited text string into a single column in the database? Or do you want to insert each element into a separate column? Either is possible, but the technique is different.
Insert comma-delimited list into one column:
conn.execute('INSERT INTO table (ColName) VALUES (?);', [','.join(list)])
Insert into separate columns:
params = ['?' for item in list]
sql = 'INSERT INTO table (Col1, Col2. . .) VALUES (%s);' % ','.join(params)
conn.execute(sql, list)
both assuming you have established a connection name conn.
A few other suggestions:
Try to avoid INSERT statements that do not list the names and order of the columns you're inserting into. That kind of statement leads to very fragile code; it breaks if you add, delete, or move columns around in your table.
If you're inserting a comma-separted list into a single-field, that generally violates principals of database design and you should use a separate table with one value per record.
If you're inserting into separate fields and they have names like Word1 and Word2, that is likewise an indication that you should be using a separate table instead.
Never use direct string substitution to create SQL statements. It will break if one of the values is, for example o'clock. It also opens you to attacks by people using SQL injection techniques.
You can use json.dumps to convert a list to json and write the json to db.
For example:
insert table example_table(column_name) values(json.dumps(your_list))