I have a simple problem when I update a MySQL table. I want to change the columns REMOVED in my table where there is a matching ID. I'm using the code below:
c = mydb.cursor(buffered=True)
for i,row in df.iterrows():
sql = "UPDATE `bas` SET removed = %s WHERE id = &s"
print(row)
print(sql)
c.execute(str(sql), tuple(row))
mydb.commit()
This code returns the following: ProgrammingError: Not all parameters were used in the SQL statement
Is there an obvious problem with the sql query?
I would like to get names from one db and initiate a new table with it. I want to add more analysis, but this is my starting point where I'm already struggling and I have no idea where I made the mistake.
mydb = db_login()
# get team
team = pd.read_sql('SELECT * FROM team', con=mydb)
names = team.name.to_list()
this will output something like ['name1', 'name2' ...]
mycursor = mydb.cursor()
mycursor.execute("DROP TABLE IF EXISTS team_analyse")
mycursor.execute("CREATE TABLE team_analyse (name VARCHAR(50))") #todo add all needed columns
sqlFormula = "INSERT INTO team_analyse (name) VALUES (%s)" #todo initial team commit
mycursor.executemany(sqlFormula, names)
mydb.commit()
In the end I get the following error:
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
You should be using a single INSERT INTO ... SELECT here:
INSERT INTO team_analyse (name)
SELECT name
FROM team
Your updated Python script:
mycursor = mydb.cursor()
sql = """INSERT INTO team_analyse (name)
SELECT name
FROM team"""
mycursor.execute(sql)
The error message you are seeing is telling you that you passed a list as the parameters to bind to the statement, but not all parameters were used. Actually, the single insert statement you were trying to run only has a single parameter. But in any case, it is unnecessary to bring the result set from the team table into Python's memory. Instead, use my suggestion and let MySQL do the heavy lifting.
I want to use a variable for the name of my table in Cassandra query.
It's possible to do this :
self.session.execute(
"""
INSERT INTO table_name (col1, col2)
VALUES (%s, %s)
""",
("1", "2")
)
But, for example, I would like to do something like this
self.session.execute(
"""
INSERT INTO %s (col1, col2)
VALUES (%s, %s)
""",
("table_name","1", "2")
)
It's look the wrong way ...
And sure I want keep safety about sql injection
Thanks
You need to use prepared queries instead of that approach - when you use %s placeholder, then the query is not prepared the Cassandra gets an additional overhead from parsing your query every time. It's better to avoid this if you use something like this - define a map that will hold prepared queries, and in the code use the function that will prepare query that isn't in this cache. The code will look as following (not tested):
queries = {}
def get_query(session, stmt):
global queries
query = queries.get(stmt)
if query is None:
query = session.prepare(stmt)
queries[stmt]=query
return query
and then in the code:
stmt = "select * from %s.%s where id =?".format(ks, table)
query = get_query(stmt)
session.execute(query, [params])
See documentation for more details.
I need to do a daily load from a source db into a new db.
The new db table columns are identical structure to the source table select statement. The new db table is in a separate db so the cursor object is unique per db connection and I can't just do a select into query.
So, for example, if I do a select on some table in some source db:
# Assuming we already connected to the db and have a cursor object...
sql_query = "SELECT val_bin, val_id, val_sel from table"
cursor.execute(sql_query)
I now have the objects I need in the cursor object from the select.
Then to insert, normally I would just grab each value and do an insert statement for each. For example:
for row in cursor.fetchall():
insert_query = "insert into new_table (val_bin, val_id, val_sel) VAULES (%s, %d, %s) % row[0], row[1], row[2]"
destination_cursor.execute(insert_query)
destination_db.commit()
However this seems tedious and slow to loop through everything.
Is there a way I can just insert the entire returned cursor object from the select statement into the new db table? The destination table schema matches exactly with what returned from the select.
If it's not possible, that's fine I am just trying to make this easier and more efficient.
You may want to use .executemany instead of .execute if the goal to insert all the results from a given cursor into a table. The following is an illustration:
cursor = connection.cursor()
destination_cursor = connection.cursor()
sql_query = "SELECT val_bin, val_id, val_sel from table"
cursor.execute(sql_query)
insert_query = "insert into new_table (val_bin, val_id, val_sel) VALUES (%s, %s, %s)"
destination_cursor.executemany(insert_query, cursor)
destination_db.commit()
I hope this proves useful.
I am trying to store some data generated by a python script in a MySQL database. Essentially I am using the commands:
con = oursql.connect(user="user", host="host", passwd="passwd",
db="testdb")
c = con.cursor()
c.executemany(insertsimoutput, zippedsimoutput)
con.commit()
c.close()
where,
insertsimoutput = '''insert into simoutput
(repnum,
timepd,
...) values (?, ?, ...?)'''
About 30,000 rows are inserted and there are about 15 columns. The above takes about 7 minutes. If I use MySQLdb instead of oursql, it takes about 2 seconds. Why this huge difference? Is this supposed to be done some other way in oursql, our oursql is just plain slow? If there is a better way to insert this data with oursql, I would appreciate if you can let me know.
Thank you.
The difference is that MySQLdb does some hackery to your query while oursql does not...
Taking this:
cursor.executemany("INSERT INTO sometable VALUES (%s, %s, %s)",
[[1,2,3],[4,5,6],[7,8,9]])
MySQLdb translates it before running into this:
cursor.execute("INSERT INTO sometable VALUES (1,2,3),(4,5,6),(7,8,9)")
But if you do:
cursor.executemany("INSERT INTO sometable VALUES (?, ?, ?)",
[[1,2,3],[4,5,6],[7,8,9]])
In oursql, it gets translated into something like this pseudocode:
stmt = prepare("INSERT INTO sometable VALUES (?, ?, ?)")
for params in [[1,2,3],[4,5,6],[7,8,9]]:
stmt.execute(*params)
So if you want to emulate what mysqldb is doing but benefit from prepared statements and other goodness with oursql, you need to do this:
from itertools import chain
data = [[1,2,3],[4,5,6],[7,8,9]]
one_val = "({})".format(','.join("?" for i in data[0]))
vals_clause = ','.join(one_val for i in data)
cursor.execute("INSERT INTO sometable VALUES {}".format(vals_clause),
chain.from_iterable(data))
I bet oursql will be faster when you do this :-)
Also, if you think its ugly, you are right. But just remember MySQL db is doing something uglier internally - its using regular expressions to parse your INSERT statement and break off the parameterized part and THEN doing what I suggested you do for oursql.
I would say to check if oursql supports a bulk insert sql command to boost performance.
Oursql does support bulk insert statements. I've written code to do so, using the sqlalchemy wrapper.
For pure oursql, something like this should be fine:
with open('tmp.csv', 'wb') as tmp:
for item in zippedsimoutput:
tmp.write("{0}\n".format(item))
c.execute("""LOAD DATA LOCAL INFILE 'tmp.csv' INTO TABLE flags FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n' ;""")
Note that the rows must be in the same order as the columns on the database.