I want to use a variable for the name of my table in Cassandra query.
It's possible to do this :
self.session.execute(
"""
INSERT INTO table_name (col1, col2)
VALUES (%s, %s)
""",
("1", "2")
)
But, for example, I would like to do something like this
self.session.execute(
"""
INSERT INTO %s (col1, col2)
VALUES (%s, %s)
""",
("table_name","1", "2")
)
It's look the wrong way ...
And sure I want keep safety about sql injection
Thanks
You need to use prepared queries instead of that approach - when you use %s placeholder, then the query is not prepared the Cassandra gets an additional overhead from parsing your query every time. It's better to avoid this if you use something like this - define a map that will hold prepared queries, and in the code use the function that will prepare query that isn't in this cache. The code will look as following (not tested):
queries = {}
def get_query(session, stmt):
global queries
query = queries.get(stmt)
if query is None:
query = session.prepare(stmt)
queries[stmt]=query
return query
and then in the code:
stmt = "select * from %s.%s where id =?".format(ks, table)
query = get_query(stmt)
session.execute(query, [params])
See documentation for more details.
Related
I'm inserting data in a database with INSERT INTO
The main problem is that I'm inserting around 170k points of data and I'm looping a code like that:
for row in data:
SQL='''INSERT INTO Table
VALUES ({},{},{})
'''.format(row[0],row[1],row[2])
cur.execute(SQL)
cur.commit()
cur.close()
con.close()
This code is extremely slow, is there a faster way to do it?
I was thinking if there is a way to insert a whole column of my matrix data at once.
Try this. Basically you can achieve it using executemany() method.
import mysql.connector
mydb = mysql.connector.connect(
.....
)
mycursor = mydb.cursor()
val = []
for row in data:
val.append((row[0],row[1],row[2]))
sql = "INSERT INTO table (x,y,z) VALUES (%s, %s, %s)"
mycursor.executemany(sql, val)
mydb.commit()
Support may vary by DBMS (which you do not specify), but you can use a prepared statement, using your DBMS's paramstyle string in the VALUES clause, and pass a list of rows to the executemany() method. See the docs at https://www.python.org/dev/peps/pep-0249/#cursor-methods
I would like to get names from one db and initiate a new table with it. I want to add more analysis, but this is my starting point where I'm already struggling and I have no idea where I made the mistake.
mydb = db_login()
# get team
team = pd.read_sql('SELECT * FROM team', con=mydb)
names = team.name.to_list()
this will output something like ['name1', 'name2' ...]
mycursor = mydb.cursor()
mycursor.execute("DROP TABLE IF EXISTS team_analyse")
mycursor.execute("CREATE TABLE team_analyse (name VARCHAR(50))") #todo add all needed columns
sqlFormula = "INSERT INTO team_analyse (name) VALUES (%s)" #todo initial team commit
mycursor.executemany(sqlFormula, names)
mydb.commit()
In the end I get the following error:
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
You should be using a single INSERT INTO ... SELECT here:
INSERT INTO team_analyse (name)
SELECT name
FROM team
Your updated Python script:
mycursor = mydb.cursor()
sql = """INSERT INTO team_analyse (name)
SELECT name
FROM team"""
mycursor.execute(sql)
The error message you are seeing is telling you that you passed a list as the parameters to bind to the statement, but not all parameters were used. Actually, the single insert statement you were trying to run only has a single parameter. But in any case, it is unnecessary to bring the result set from the team table into Python's memory. Instead, use my suggestion and let MySQL do the heavy lifting.
I need to do a daily load from a source db into a new db.
The new db table columns are identical structure to the source table select statement. The new db table is in a separate db so the cursor object is unique per db connection and I can't just do a select into query.
So, for example, if I do a select on some table in some source db:
# Assuming we already connected to the db and have a cursor object...
sql_query = "SELECT val_bin, val_id, val_sel from table"
cursor.execute(sql_query)
I now have the objects I need in the cursor object from the select.
Then to insert, normally I would just grab each value and do an insert statement for each. For example:
for row in cursor.fetchall():
insert_query = "insert into new_table (val_bin, val_id, val_sel) VAULES (%s, %d, %s) % row[0], row[1], row[2]"
destination_cursor.execute(insert_query)
destination_db.commit()
However this seems tedious and slow to loop through everything.
Is there a way I can just insert the entire returned cursor object from the select statement into the new db table? The destination table schema matches exactly with what returned from the select.
If it's not possible, that's fine I am just trying to make this easier and more efficient.
You may want to use .executemany instead of .execute if the goal to insert all the results from a given cursor into a table. The following is an illustration:
cursor = connection.cursor()
destination_cursor = connection.cursor()
sql_query = "SELECT val_bin, val_id, val_sel from table"
cursor.execute(sql_query)
insert_query = "insert into new_table (val_bin, val_id, val_sel) VALUES (%s, %s, %s)"
destination_cursor.executemany(insert_query, cursor)
destination_db.commit()
I hope this proves useful.
How to insert data into a mySQL table when the data is stored in a variable?
I have lots of data that will be stored in variables and wish for it to be inserted into the database table.
Without any more details, The simple generic answer is: print/format your variable into the query string that you'll be executing on a database. Example:
sql_string = "INSERT INTO table_name VALUES('%s', '%s')" % (python_var1, python_var2)
or
sql_string = "INSERT INTO table_name VALUES('{}','{}')".format(python_var1, python_var2)
then execute that on your db. Exact query obviously needs to match your DB schema
cursor.execute("""INSERT INTO device (name, MODEL, MEMORY, COLOUR, PRICE, PRICE_UP, PRICE_DOWN, PRICE_DIFF) VALUES (r{0}, r{1}, r{2}, r{3}, r{4}, r{5}, r{6}, r{7})""".format(*device_data))
I am trying to use SQL with prepared statements in Python. Python doesn't have its own mechanism for this so I try to use SQL directly:
sql = "PREPARE stmt FROM ' INSERT INTO {} (date, time, tag, power) VALUES (?, ?, ?, ?)'".format(self.db_scan_table)
self.cursor.execute(sql)
Then later, in the loop:
sql = "EXECUTE stmt USING \'{}\', \'{}\', {}, {};".format(d, t, tag, power)
self.cursor.execute(sql)
And in the loop I get:
MySQL Error [1064]: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''2014-12-25', '12:31:46', 88000000, -6.64' at line 1
What's going on?
Using prepared statements with MySQL in Python is explained e.g at http://zetcode.com/db/mysqlpython/ -- look within that page for Prepared statements.
In your case, that would be, e.g:
sql = ('INSERT INTO {} (date, time, tag, power) VALUES '
'(%s, %s, %s, %s)'.format(self.db_scan_table))
and later, "in the loop" as you put it:
self.cursor.execute(sql, (d, t, tag, power))
with no further string formatting -- the MySQLdb module does the prepare and execute parts on your behalf (and may cache things to avoid repeating work needlessly, etc, etc).
Do consider, depending on the nature of "the loop" you mention, that it's possible that a single call to .execute_many (with a sequence of tuples as the second argument) could take the place of the whole loop (unless you need more processing within that loop beyond just the insertion of data into the DB).
Added: a better alternative nowadays may be to use mysql's own Connector/Python and the explicit prepare=True option in the .cursor() factory -- see http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html . This lets you have a specific cursor on which statements are prepared (with the "more efficient than using PREPARE and EXECUTE" binary protocol, according to that mysql.com page) and another one for statements that are better not prepared; "explicit is better than implicit" is after all one of the principles in "The Zen of Python" (import this from an interactive prompt to read all those principles). mysqldb doing things implicitly (and it seems the current open-source version doesn't use prepared statements) can't be as good an architecture as Connector/Python's more explicit one.
import mysql.connector
db_con=mysql.connector.connect(host='',
database='',
user='',
password='')
cursor = db_con.cursor(prepared=True,)
#cursor = db_con.cursor(prepared=True)#IT MAY HAVE PROBLEM
sql = """INSERT INTO table (xy,zy) VALUES (%s, %s)"""
input=(1,2)
cursor.execute(sql , input)
db_con.commit()
SELECT STMT
sql = """SELECT * FROM TABLE WHERE XY=%s ORDER BY id DESC LIMIT 1 """
ID=1
input=(ID,)
#input=(ID)# IT MAY HAS PROBLEM
cursor.execute(sql, input)
data = cursor.fetchall()
rowsNumber=cursor.rowcount
Python does support prepared statements:
sql = "INSERT INTO {} (date, time, tag, power) VALUES (%s, %s, %s, %s);"
sql = sql.format(self.db_scan_table)
self.cursor.execute(sql, (d, t, tag, power))
(You should ensure self.db_scan_table is not vulnerable to SQL injection)
This assumes your paramstyle is 'format', which it should be for MySQL.