I have the following query in python-mysql:
self.cursor.execute('SELECT apple_id FROM main_catalog WHERE apple_id=%s', apple_id)
if not self.cursor.fetchone():
cursor.execute('''INSERT INTO main_catalog
(apple_id, provider_id, show_title, artwork_apple_url, title,
itunes_url, network_name, length, episode_production_number, synopsis)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)''',
apple_id, provider_obj.pk, show_title, artwork_apple_url, episode_title,
itunes_url, network_name, length, episode_production_number, synopsis)
This seems incredible verbose. Is there a way to write this more compactly and easier to read/understand?
I suggest you use pytho-sql or sqlpuzzle to be avoid of strings containing log SQL statements
catalog = Table('main_catalog')
catalog.insert(columns=[
catalog.apple_id, user.provide_id],
values=[['Foo', 'foo'], ['Bar', 'bar']])
)
Using this library you can build the query from a dict:
columns = [Column(catalog, col) for col in data.keys()]
sql = catalog.insert(
columns=columns,
values=[data.values()]
)
Well, so far there's no logical operations you can minimize into this peace of code. It's basically a query which inserts values into a table which has many fields. I doubt there's something you can do about it.
Related
sql = """INSERT INTO (Product_details) VALUES (%s, %s, %s, %s, %s, %s)"""
val = [name, Spec ,Ratings ,Delivery ,Discount ,Price ] # list of data into list
for values in val: #loping the variables in the list and adding it to database
engine.execute(sql, values)
It looks like you're using Python - Try ? instead of %s - sometimes the parameter marker is not what you would expect it to be so do check which one you need to use for the language you're embedding the SQL in
I'm working on a DB and I'm having trouble when using pymysql to INSERT some values
cur.execute("""INSERT INTO orders (name, size, type, is_done) VALUES (%s, %s, %s, %s)"""
% (name, size, type, is_done))
Where name, size and type are strings and is_done is a bool
It gives me the typical error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near, so I suppose the problem is a ', but how can I solve it?
Edit
I should also add that the name value is retrieved from a MySQL DB
The current accepted solution has a SQL injection vulnerability. You are not supposed to format the string with the % operator - just pass the tuple of arguments as a second argument, and the library will deal with the rest.
cur.execute("INSERT INTO orders (name, size, type, is_done) VALUES (%s, %s, %s, %s)",
(name, size, type, is_done))
Also see this answer and pymysql documentation.
I have found the problem, which was that instead of
cur.execute("""INSERT INTO orders (name, size, type, is_done)
VALUES (%s, %s, %s, %s)"""
% (name, size, type, is_done))
I should have done
cur.execute("""INSERT INTO orders (name, size, type, is_done)
VALUES ("%s", "%s", "%s", "%s")"""
% (name, size, type, is_done))
if you don't input value for id. You have an error. Try this query.
cur.execute("insert into orders values(%s, %s, %s, %s, %s)", (None, name, size, type, is_done))
"%s" and "None" for id column. This query running my code.
Note: Don't forget commit()
This might be a rather silly question but what am I doing wrong here? It creates the table but the INSERT INTO doesn't work, I guess I'm doing something wrong with the placeholders?
conn = psycopg2.connect("dbname=postgres user=postgres")
cur = conn.cursor()
escaped_name = "TOUR_2"
cur.execute('CREATE TABLE %s(id serial PRIMARY KEY, day date, elapsed_time varchar, net_time varchar, length float, average_speed float, geometry GEOMETRY);' % escaped_name)
cur.execute('INSERT INTO %s (day,elapsed_time, net_time, length, average_speed, geometry) VALUES (%s, %s, %s, %s, %s, %s)', (escaped_name, day ,time_length, time_length_net, length_km, avg_speed, myLine_ppy))
conn.commit()
cur.close()
conn.close()
The INSERT INTO call doesn't work, it gives me
cur.execute('INSERT INTO %s (day,elapsed_time, net_time, length, average_speed,
geometry) VALUES (%s, %s, %s, %s, %s, %s)'% (escaped_name, day ,time_length,
time_length_net, length_km, avg_speed, myLine_ppy))
psycopg2.ProgrammingError: syntax error at or near ":"
LINE 1: ...h, average_speed, geometry) VALUES (2013/09/01 , 2:56:59, 02...
Can someone help me on this one? Thanks a bunch!
You are using Python string formatting and this is a Very Bad Idea (TM). Think SQL-injection. The right way to do it is to use bound variables:
cur.execute('INSERT INTO %s (day, elapsed_time, net_time, length, average_speed, geometry) VALUES (%s, %s, %s, %s, %s, %s)', (escaped_name, day, time_length, time_length_net, length_km, avg_speed, myLine_ppy))
where the tuple of parameters is given as second argument to execute(). Also you don't need to escape any value, psycopg2 will do the escaping for you. In this particular case is also suggested to not pass the table name in a variable (escaped_name) but to embed it in the query string: psycopg2 doesn't know how to quote table and column names, only values.
See psycopg2 documentation:
https://www.psycopg.org/docs/usage.html#passing-parameters-to-sql-queries
If you want to programmatically generate the SQL statement, the customary way is to use Python formatting for the statement and variable binding for the arguments. For example, if you have the table name in escaped_name you can do:
query = "INSERT INTO %s (col1, ...) VALUES (%%s, ...)" % escaped_name
curs.execute(query, args_tuple)
Obviously, to use placeholders in your query you need to quote any % that introduce a bound argument in the first format.
Note that this is safe if and only if escaped_name is generated by your code ignoring any external input (for example a table base name and a counter) but it is at risk of SQL injection if you use data provided by the user.
To expand on #Matt's answer, placeholders do not work for identifiers like table names because the name will be quoted as a string value and result in invalid syntax.
If you want to generate such a query dynamically, you can use the referred to pyscopg2.sql module:
from psycopg2.sql import Identifier, SQL
cur.execute(SQL("INSERT INTO {} VALUES (%s)").format(Identifier('my_table')), (10,))
As of psycopg2 v2.7 there is a supported way to do this: see the psycopg2.sql docs.
I am reading in some raw files and inserting them into a DB. This will involve MILLIONS of records, and for each record, I have multiple inserts (many tables). When I was testing locally, it was going quickly, but for the whole dataset, I need to work with a remote database. It is painfully slow this way, which I assume is because of all the trips over the network to delete/insert.
I am using the MySQLdb module (python) and at the moment, I have things such as the following:
# setup connection
con = mdb.connect('remote.host', 'database_user', '123456789', 'database_name');
... read files, loop through records, etc...
# clear out data related to current record
cur.execute("DELETE FROM articles WHERE article_id = %s", article.id)
cur.execute("DELETE FROM authors WHERE article_id = %s", article.id)
cur.execute("DELETE FROM addresses WHERE article_id = %s", article.id)
cur.execute("DELETE FROM citation_references WHERE article_id = %s", article.id)
cur.execute("DELETE FROM citation_patents WHERE article_id = %s", article.id)
# insert the article
cur.execute("INSERT INTO articles (article_id, doctype, keywords, language, title) VALUES (%s, %s, %s, %s, %s, %s)" , (article.id, article.doctype, ';'.join(article.keywords), article.language, article.title))
# insert all the authors
for au in article.authors:
cur.execute("INSERT INTO isi_authors (article_id, name_first, name_last, email) VALUES (%s, %s, %s, %s)", (article.id, au.first_name, au.last_name, au.email))
... other loops like the authors to insert 10-20 citations per article, multiple addresses, etc ...
From what I can tell, MySQLdb does not allow me to send multiple queries at once. There must be a way for me to avoid the network delays. Any ideas?
At least MySQLdb 1.2.3 seems to allow multiple queries out of the box, you just have to call cursor.nextset() to cycle through the returned result sets.
db = conn.cursor()
db.execute('SELECT 1; SELECT 2;')
more = True
while more:
print db.fetchall()
more = db.nextset()
If you want to be absolutely sure the support for this is enabled, and/or disable the support, you can use something like this:
MYSQL_OPTION_MULTI_STATEMENTS_ON = 0
MYSQL_OPTION_MULTI_STATEMENTS_OFF = 1
conn.set_server_option(MYSQL_OPTION_MULTI_STATEMENTS_ON)
# Multiple statement execution here...
conn.set_server_option(MYSQL_OPTION_MULTI_STATEMENTS_OFF)
If there's an error while executing one of the queries, mysql won't execute any queries after that point. The db.execute() call will throw the exception if it originated from the first query, otherwise the appropriate db.nextset() will do it, so you can fetch the result sets from the successfull queries before getting the exception.
Use executemany. Here's an example from the manual:
c.executemany(
"""INSERT INTO breakfast (name, spam, eggs, sausage, price)
VALUES (%s, %s, %s, %s, %s)""",
[
("Spam and Sausage Lover's Plate", 5, 1, 8, 7.95 ),
("Not So Much Spam Plate", 3, 2, 0, 3.95 ),
("Don't Wany ANY SPAM! Plate", 0, 4, 3, 5.95 )
] )
In your case it would look something like this:
sql = "INSERT INTO isi_authors (article_id, name_first, name_last, email) VALUES (%s, %s, %s, %s)"
params = [(article.id, au.first_name, au.last_name, au.email) for au in article.authors]
cur.executemany(sql, params)
From the documentation of executemany:
This method improves performance on multiple-row INSERT and
REPLACE. Otherwise it is equivalent to looping over args with
execute().
mySQL INSERT syntax does allow it. Compare 1) and 2)
1. INSERT INTO tbl_name (a,b,c) VALUES(1,2,3);
2. INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
In the second case you're inserting three rows at once.
http://dev.mysql.com/doc/refman/5.5/en/insert.html
Hopefully it will give you ideas.
PS: this is language-independent way
I am trying to accomplish something like the following:
cursor = db.cursor()
cursor.execute('INSERT INTO media_files (%s, %s, %s, %s ... ) VALUES (%s, %s, %s, %s, ...)', (fieldlist, valuelist))
cursor.commit()
I have 2 lists, fieldlist and valuelist which each contain the same number of items. What is the best way to generate a dynamic MySQL query statement where the collumns are stored in fieldlist and the values are stored in valuelist?
cursor.execute('INSERT INTO media_files (%s) VALUES (%%s, %%s, %%s, %%s, ...)' % ','.join(fieldlist), valuelist)
To make it clearer:
sql = 'INSERT INTO media_files (%s) VALUES (%%s, %%s, %%s, %%s, ...)' % ','.join(fieldlist)
cursor.execute(sql, valuelist)
The cursor expects parameters to be passed as a single sequence, so you need to combine - in order - the field and value lists.
itertools.chain() does exactly that however it returns a generator and I'm not sure if cursor.execute() will accept that as it's param sequence. Try it. If it fails, wrap it with list()
import itertools
sql = 'INSERT INTO media_files (%s, %s, %s, %s ... ) VALUES (%s, %s, %s, %s, ...)'
cursor.execute(sql, itertools.chain(fieldlist, valuelist))
EDIT:
This solution will not work. This would cause the field names to be escaped and wrapped with quotes which would cause an sql syntax error.
I'll leave this answer as it might serve as a useful example but look to #Trent's answer for the solution.