What I'm trying to achieve is placing data generated with a python script into a MySQL database. So far I have been able to generate the data with Python, and print it, but I'm not too sure how I can place that into the mysql table. I've read that you can use MySQLdb as a connector between the two, and currently I have been able to place data into the table, but only data that I manually type.
Hopefully the code makes sense, but I am trying to place the values from BidSize, AskSize, BidPrice, and AskPrice into the table.
from wrapper_v3 import IBWrapper, IBclient
from swigibpy import Contract as IBcontract
if __name__=="__main__":
"""
This simple example returns streaming price data
"""
callback = IBWrapper()
client=IBclient(callback)
ibcontract = IBcontract()
ibcontract.secType = "FUT"
ibcontract.expiry="201612"
ibcontract.symbol="GE"
ibcontract.exchange="GLOBEX"
ans=client.get_IB_market_data(ibcontract)
print "Bid size, Ask size; Bid price; Ask price"
print ans
import MySQLdb as mdb
con = mdb.connect('localhost', 'testuser', 'test623', 'testdb');
with con:
cur = con.cursor()
cur.execute("DROP TABLE IF EXISTS Histdata")
cur.execute("CREATE TABLE Histdata(Id INT PRIMARY KEY AUTO_INCREMENT, \
Name VARCHAR(25), BidSize VARCHAR(20), AskSize VARCHAR(20), BidPrice VARCHAR(20), AskPrice VARCHAR(20))")
cur.execute("INSERT INTO Histdata(Name,BidSize,AskSize,BidPrice,AskPrice) VALUES('test','test','test','test','test'))")
Parameterize your query like this:
# assuming ans is a list
bid_size, ask_size, bid_price, ask_price = ans
cur.execute('''
INSERT INTO Histdata
(Name,BidSize,AskSize,BidPrice,AskPrice)
VALUES(%s,%s,%s,%s,%s)''', ('test', bid_size, ask_size, bid_price, ask_price))
As Fabricator correctly pointed out, please parametrize your queries to prevent SQL injections.
Yet, the reason why you are not seeing the data written to your table is because you did not commit the transaction. Your cursor is adding queries to the transaction, but it is only after you instruct a commit that the data is actually written to the database.
Add con.commit() after adding your cursor queries to finalize the transaction.
In older versions of the MySQLdb adapter for Python data would be committed automatically. But this is considered "bad practice" as the programmer's control over transactions should be honored. This flow has also been laid out in Python's DB API, which all database adapters are asked to honor.
As a side note: Don't forget to close both your cursor and your connection when you are done:
cursor.close()
conn.close()
Otherwise your connection will stay open in an idle state and you could potentially deplete your pool of connection slots.
Related
I am going to insert data into a table which is in Oracle using by python. It did not give me any error but also it did not insert anything into my table in Oracle SQL Developer. Also I wrote a query to get select rom my table in python environment it gave me output.
My CREATE TABLE code:
cursor = conn.cursor()
cursor.execute("CREATE TABLE INSERT_TEST(FISRT_COLUMN VARCHAR2(30),SECOND_COLUMN NUMBER(10,2))")
My INSERT code:
cursor = conn.cursor()
data = [
('Mina', 20),
('Minoo', 32),
('Sara', 22),
('Ehssan', 25),
('Ladan', 55)
]
for row in data:
cursor.execute("""
insert into INSERT_TEST (FISRT_COLUMN, SECOND_COLUMN)
values (:1, :2)""", row)
My SELECT query:
re = cursor.execute("select * from INSERT_TEST")
for row in re:
print(row)
My output in python environment:
('Mina', 20.0) ('Minoo', 32.0) ('Sara', 22.0) ('Ehssan', 25.0) ('Ladan', 55.0)
My output in Oracle SQL Developer is empty!
I think you need a conn.commit() at the end of your program. commit belongs to the connection class, not to the cursor.
I understand it depends on table in oracle SQL developer, a the end of my code I have to write cursor.execute('commit') to show that it end and insert all of the data into my table.
Check the documentation for your database driver.
For some database drivers, the default behaviour is to COMMIT any uncommitted data when you close the transaction and the connection; so, make sure you close the connection when you have finished.
For other database drivers, the default behaviour is to ROLLBACK any uncommitted data when you close the transaction and the connection; in this case, you will need to explicitly COMMIT the data before you close the connection otherwise it will ROLLBACK your changes.
The Oracle database does not let other sessions (even belonging to the same user) see uncommitted data so, if you INSERT from Python but do not COMMIT (either explicitly or, if the driver supports it, implicitly when you close the connection) then you will be able to see it from the Python session where you created the data but it will be hidden when you view it in another session (i.e. from SQL Developer).
I use pyodbc in my python flask Project for the SQLite DB connection.
I know and understand SQL Injections but this is my first time dealing with it.
I tried to execute some
I have a function which concatenates the SQL String in my database.py file:
def open_issue(self, data_object):
cursor = self.conn.cursor()
# data_object is the issue i get from the user
name = data_object["name"]
text = data_object["text"]
rating_sum = 0
# if the user provides an issue
if name:
# check if issue is already in db
test = cursor.execute(f'''SELECT name FROM issue WHERE name = "{name}"''')
data = test.fetchall()
# if not in db insert
if len(data) == 0:
# insert the issue
cursor.executescript(f'''INSERT INTO issue (name, text, rating_sum)
VALUES ("{name}", "{text}", {rating_sum})''')
else:
print("nothing inserted!")
In the api.py file the open_issue() function gets called:
#self.app.route('/open_issue')
def insertdata():
# data sent from client
# data_object = flask.request.json
# unit test dictionary
data_object = {"name": "injection-test-table",
"text": "'; CREATE TABLE 'injected_table-1337';--"}
DB().open_issue(data_object)
The "'; CREATE TABLE 'injected_table-1337';--" sql injection has not created the injected_table-1337, instead it got inserted normally like a string into the text column of the injection-test-table.
So i don't really know if i am safe for the standard ways of SQL injection (this project will only be hosted locally but good security is always welcome)
And secondary: are there ways with pyodbc to check if a string contains sql syntax or symbols, so that nothing will get inserted in my example or do i need to check the strings manually?
Thanks a lot
As it turns out, with SQLite you are at much less risk of SQL injection issues because by default neither Python's built-in sqlite3 module nor the SQLite ODBC driver allow multiple statements to be executed in a single .execute call (commonly known as an "anonymous code block"). This code:
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = f"SELECT * FROM table1 WHERE txt='{thing}'"
crsr.execute(sql)
throws this for sqlite3
sqlite3.Warning: You can only execute one statement at a time.
and this for SQLite ODBC
pyodbc.Error: ('HY000', '[HY000] only one SQL statement allowed (-1) (SQLExecDirectW)')
Still, you should follow best practices and use a proper parameterized query
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = "SELECT * FROM table1 WHERE txt=?"
crsr.execute(sql, (thing, ))
because this will also correctly handle parameter values that would cause errors if injected directly, e.g.,
thing = "it's good to avoid SQL injection"
I'm querying a json on a website for data, then saving that data into a variable so I can put it into a sqlite table. I'm 2 out of 3 for what I'm trying to do, but the sqlite side is just mystifying. I'm able to request the data, from there I can verify that the variable has data when I test it with a print, but all of my sqlite stuff is failing. It's not even creating a table, much less updating the table (but it is printing all the results to the buffer for some reason) Any idea what I'm doing wrong here? Disclaimer: Bit of a python noob. I've successfully created test tables just copying the stuff off of the python sqlite doc
# this is requesting the data and seems to work
for ticket in zenpy.search("bananas"):
id = ticket.id
subj = ticket.subject
created = ticket.created_at
for comment in zenpy.tickets.comments(ticket.id):
body = comment.body
# connecting to sqlite db that exists. things seem to go awry here
conn = sqlite3.connect('example.db')
c = conn.cursor()
# Creating the table table (for some reason table is not being created at all)
c.execute('''CREATE TABLE tickets_test
(ticket id, ticket subject, creation date, body text)''')
# Inserting the variables into the sqlite table
c.execute("INSERT INTO ticketstest VALUES (id, subj, created, body)")
# committing changes the changes and closing
c.commit()
c.close()
I'm on Windows 64bit and using pycharm to do this.
Your table likely isn't created because you haven't committed yet, and your sql fails before it commits. It should work when you fix your 2nd sql statement.
You're not inserting the variables you've created into the table. You need to use parameters. There are two ways of parameterizing your sql statement. I'll show the named placeholders one:
c.execute("INSERT INTO ticketstest VALUES (:id, :subj, :created, :body)",
{'id':id, 'subj':subj, 'created':created, 'body':body}
)
I have done the following:
import MySQLdb as mdb
con = mdb.connect(hostname, username, password, dbname)
cur = con.cursor()
count = cur.execute(query)
cur.close()
con.close()
I have two queries, I execute them in the mysql console I can view the results.
But when I give the same through python one query works and the other one does not.
I am sure it is not problem with mysql or query or python code. I suspect cur.execute(query) function.
Have anyone come through similar situation? Any solutions?
Use conn.commit() after execution, to commit/finish insertion and deletion based changes.
I have two queries, I execute them in the mysql console I can view the results.
But I only see one query:
import MySQLdb as mdb
con = mdb.connect(hostname, username, password, dbname)
cur = con.cursor()
count = cur.execute(query)
cur.close()
con.close()
My guess is query contains the both queries separated by a semin-colon and is an INSERT statement? You probably need to use executemany().
See Executing several SQL queries with MySQLdb
On the other hand, if both of your queries are SELECT statements (you say "I see the result"), I'm not sure you can fetch both results from only one call to execute(). I would consider that as bad style, anyway.
This is a function and the query is passed to this function. When I
execute one query after the other. I dont get the result for few
queries, there is no problem with the queries because I have crossed
checked them with the mysql console.
As you clarified your question in a comment, I post an other answer -- completely different approach.
Are you connected to your DB in autocommit mode? If no, for changes to be permanently applied, you have to COMMIT them. In normal circumstances, you shouldn't create a new connection for each request. That put excessive load on the DB server for almost nothing:
# Open a connection once
con = mdb.connect(hostname, username, password, dbname)
# Do that *for each query*:
cur = con.cursor()
try:
count = cur.execute(query)
conn.commit() # don't forget to commit the transaction
else:
print "DONE:", query # for "debug" -- in real app you migth have an "except:" clause instead
finally:
cur.close() # close anyway
# Do that *for each query*:
cur = con.cursor()
try:
count = cur.execute(query)
conn.commit() # don't forget to commit the transaction
else:
print "DONE:", query # for "debug" -- in real app you migth have an "except:" clause instead
finally:
cur.close() # close anyway
# Close *the* connection
con.close()
The above code is directly typed into SO. Please forgive typos and other basic syntax errors. But that's the spirit of it.
A last word, while typing I was wondering how you deal with exceptions? By any chance could the MySQLdb error be silently ignored at some upper level of your program?
Use this query, this will update multiple rows of column in one query
sql=cursor.executemany("UPDATE `table` SET `col1` = %s WHERE `col2` = %s",
[(col1_val1, col2_val1),(col2_val2, col_val2)])
and also commit with database to see the changes.
conn.commit()
I am using python 2.7 and MySQL as database. In my python program have an INSERT query like this:
cursor.execute("insert into login(username,passw)values('"+i.username+"','"+i.password+"')")
result=cursor.execute("select * from login")
print cursor.fetchall()
When I check in the database, there is no entry. But after the select in my python code, when I print the results it is showing the inserted data. I am not using any transaction statement either.
You need to commit your transaction for the database to make your insert permanent, and you need to use SQL parameters to prevent SQL injection attacks and general quoting bugs:
cursor.execute("insert into login (username, passw) values (%s, %s)", (i.username, i.password))
connection.commit()
Until you commit, the data you inserted will only be visible to your python program; if you do not commit at all, then the changes will be discarded again by the database.
Alternatively, you could switch on auto-commit mode:
connection.autocommit()
After switching on auto-commit, your insertions will be committed instantly. Be careful with this as this could lead to inconsistent data if you need to insert data into multiple rows and / or tables that is interdependent.
You also need to commit the data after your execution statement. It is important to call this method after you are done inserting, or updating data, as the Python connector does not auto commit by default.
# Execute & Commit
cursor.execute("insert into login(username,passw) values('%s','%s')",
i.username, i.password)
# Commit the insert query!
conn.commit()
# Fetch Result
result=cursor.execute("select * from login")
print cursor.fetchall()
If you use mysql-python, you can set connection options to enable autocommit feature.
conn = mysql.connection(host, port, autocommit=True)
# or
conn = mysql.connection(host, port)
conn.autocommit(True)
You can see more details here