I have more than 2 tables in an sqlite3 database. Im trying to INSERT data into one table and update a few columns in another table. Is that possible?
I have tried to user executemany() and executescript() in many shapes and forms. I have recieved a bunch of error messages, and its mostly because execute will not accept my parameters.
with sqlite3.connect('database.db') as conn:
cur = conn.cursor()
cur.executescript("""INSERT INTO prev_users (
imei_prev,
user_id_prev,
start_date,
end_date,
type_prev)
VALUES (?,?,?,?,?);
UPDATE phones
SET owner = (?), date = (?), state = (?)
WHERE imei = (?);
""",(user['imei'], user['user_id'], user['date'], date, user['type'], "None", date, "In Progress", user['imei']))
executescript() will not accept parameters because, well, it doesn't take parameters as an argument, it only takes a "script". From the doc:
executescript(sql_script)
This is a nonstandard convenience method for executing multiple SQL statements at once. It issues a COMMIT statement first, then
executes the SQL script it gets as a parameter.
sql_script can be an instance of str.
For example, it might be used for creating several tables in one script.
executemany() runs one sql against a sequence of parameters.
Neither method is the right tool for the job. You'll likely have to split it into two calls to execute().
Related
I'm currently working with Python and Pandas to query data from my databse. I have one query to get some customer information (this is obviously not the real query, it's simplified without joins etc.):
def customer_query(con, date):
stmt = """
SELECT
first_name
last_name
dob
FROM
customer
"""
return pd.read_sql(
stmt,
con
)
I do it with pandas so that I can easily export the dataset to csv without any hassle.
The requirements changed and I need to generate two different datasets based on circumstances around the customer.
The original query still holds, we still need first_name, last_name etc, so I don't want to create two entirely separate queries.
I want to add a where-clause to my query like so:
def customer_query(con, date):
stmt = """
SELECT
first_name
last_name
dob
FROM
customer
where id in (:sub)
"""
return pd.read_sql(
stmt,
con,
params={"sub": "SELECT customer_id FROM different_table_1"},
)
I cannot just put the subquery in the statement with a parameter. What I wish to do is put the subquery as a parameter.
This way I could pass the subquery as an argument and generate two different datasets. That doesn't work with pandas though.
The only thing I can come up with is to execute the subquery on it's own, grab the customer ids from it and pass those to my "customer_query" function. This isn't as nice as executing everything in one SQL statement, but I don't have any other idea. I also refrain from the idea to build the sql statement with f strings or something.
EDIT:
I forgot to mention that I'm connection to an Oracle DB and the "con" object is a cx_Oracle connection.
Apparently my thought of solution is not valid since cx_Oracle does not really support passing a list as a parameter.
You may be better off creating two queries so the optimizer tunes each one. Like everything "it depends". How often will they run? Does the connection remain open (there is also the statement cache to think about), etc, etc.
In Oracle SQL, bind values are not used to build up SQL statements (hence their security benefits).
This does not work – the update has no effect:
command = "select content from blog where slug = 'meow'; update account_balance set balance=200 where id=1; select 1 from blog;"
content = db.engine.scalar(command)
Switching the statements performs the update and select successfully:
command = "update account_balance set balance=200 where id=1; select content from blog where slug = 'meow';"
content = db.engine.scalar(command)
Why does the first not work? It works in Pgadmin. I enabled autocommit with Flask-Sqlalchemy.
I am doing a workshop on SQL injection, so please dont rewrite the solution!
The way SQLAlchemy's autocommit works is that it inspects the issued statements, trying to detect whether or not data is modified:
..., SQLAlchemy implements its own “autocommit” feature which works completely consistently across all backends. This is achieved by detecting statements which represent data-changing operations, i.e. INSERT, UPDATE, DELETE, as well as data definition language (DDL) statements such as CREATE TABLE, ALTER TABLE, and then issuing a COMMIT automatically if no transaction is in progress. The detection is based on the presence of the autocommit=True execution option on the statement. If the statement is a text-only statement and the flag is not set, a regular expression is used to detect INSERT, UPDATE, DELETE, as well as a variety of other commands for a particular backend
Since multiple result sets are not supported at SQLAlchemy level, in your first example the detection simply omits issuing a COMMIT because the first statement is a SELECT, where as in your second example it is an UPDATE. No attempt to detect data modifying statements from multiple statements takes place.
If you look at PGExecutionContext.should_autocommit_text(), you'll see that it does a regex match against AUTOCOMMIT_REGEXP. In other words it matches only at the beginning of the text.
If you want to create a table using SELECT INTO:
As explained above engine.execute('select * into a from b') doesn't work. Instead you can do as follows:
conn = engine.raw_connection()
cursor = conn.cursor()
cursor.execute('select * into a from b')
conn.commit()
You should use db.engine.execute(...).first() if you want to execute everything and get only first row.
My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?
First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.
In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.
This does not work – the update has no effect:
command = "select content from blog where slug = 'meow'; update account_balance set balance=200 where id=1; select 1 from blog;"
content = db.engine.scalar(command)
Switching the statements performs the update and select successfully:
command = "update account_balance set balance=200 where id=1; select content from blog where slug = 'meow';"
content = db.engine.scalar(command)
Why does the first not work? It works in Pgadmin. I enabled autocommit with Flask-Sqlalchemy.
I am doing a workshop on SQL injection, so please dont rewrite the solution!
The way SQLAlchemy's autocommit works is that it inspects the issued statements, trying to detect whether or not data is modified:
..., SQLAlchemy implements its own “autocommit” feature which works completely consistently across all backends. This is achieved by detecting statements which represent data-changing operations, i.e. INSERT, UPDATE, DELETE, as well as data definition language (DDL) statements such as CREATE TABLE, ALTER TABLE, and then issuing a COMMIT automatically if no transaction is in progress. The detection is based on the presence of the autocommit=True execution option on the statement. If the statement is a text-only statement and the flag is not set, a regular expression is used to detect INSERT, UPDATE, DELETE, as well as a variety of other commands for a particular backend
Since multiple result sets are not supported at SQLAlchemy level, in your first example the detection simply omits issuing a COMMIT because the first statement is a SELECT, where as in your second example it is an UPDATE. No attempt to detect data modifying statements from multiple statements takes place.
If you look at PGExecutionContext.should_autocommit_text(), you'll see that it does a regex match against AUTOCOMMIT_REGEXP. In other words it matches only at the beginning of the text.
If you want to create a table using SELECT INTO:
As explained above engine.execute('select * into a from b') doesn't work. Instead you can do as follows:
conn = engine.raw_connection()
cursor = conn.cursor()
cursor.execute('select * into a from b')
conn.commit()
You should use db.engine.execute(...).first() if you want to execute everything and get only first row.
This does not work – the update has no effect:
command = "select content from blog where slug = 'meow'; update account_balance set balance=200 where id=1; select 1 from blog;"
content = db.engine.scalar(command)
Switching the statements performs the update and select successfully:
command = "update account_balance set balance=200 where id=1; select content from blog where slug = 'meow';"
content = db.engine.scalar(command)
Why does the first not work? It works in Pgadmin. I enabled autocommit with Flask-Sqlalchemy.
I am doing a workshop on SQL injection, so please dont rewrite the solution!
The way SQLAlchemy's autocommit works is that it inspects the issued statements, trying to detect whether or not data is modified:
..., SQLAlchemy implements its own “autocommit” feature which works completely consistently across all backends. This is achieved by detecting statements which represent data-changing operations, i.e. INSERT, UPDATE, DELETE, as well as data definition language (DDL) statements such as CREATE TABLE, ALTER TABLE, and then issuing a COMMIT automatically if no transaction is in progress. The detection is based on the presence of the autocommit=True execution option on the statement. If the statement is a text-only statement and the flag is not set, a regular expression is used to detect INSERT, UPDATE, DELETE, as well as a variety of other commands for a particular backend
Since multiple result sets are not supported at SQLAlchemy level, in your first example the detection simply omits issuing a COMMIT because the first statement is a SELECT, where as in your second example it is an UPDATE. No attempt to detect data modifying statements from multiple statements takes place.
If you look at PGExecutionContext.should_autocommit_text(), you'll see that it does a regex match against AUTOCOMMIT_REGEXP. In other words it matches only at the beginning of the text.
If you want to create a table using SELECT INTO:
As explained above engine.execute('select * into a from b') doesn't work. Instead you can do as follows:
conn = engine.raw_connection()
cursor = conn.cursor()
cursor.execute('select * into a from b')
conn.commit()
You should use db.engine.execute(...).first() if you want to execute everything and get only first row.