I'm querying a json on a website for data, then saving that data into a variable so I can put it into a sqlite table. I'm 2 out of 3 for what I'm trying to do, but the sqlite side is just mystifying. I'm able to request the data, from there I can verify that the variable has data when I test it with a print, but all of my sqlite stuff is failing. It's not even creating a table, much less updating the table (but it is printing all the results to the buffer for some reason) Any idea what I'm doing wrong here? Disclaimer: Bit of a python noob. I've successfully created test tables just copying the stuff off of the python sqlite doc
# this is requesting the data and seems to work
for ticket in zenpy.search("bananas"):
id = ticket.id
subj = ticket.subject
created = ticket.created_at
for comment in zenpy.tickets.comments(ticket.id):
body = comment.body
# connecting to sqlite db that exists. things seem to go awry here
conn = sqlite3.connect('example.db')
c = conn.cursor()
# Creating the table table (for some reason table is not being created at all)
c.execute('''CREATE TABLE tickets_test
(ticket id, ticket subject, creation date, body text)''')
# Inserting the variables into the sqlite table
c.execute("INSERT INTO ticketstest VALUES (id, subj, created, body)")
# committing changes the changes and closing
c.commit()
c.close()
I'm on Windows 64bit and using pycharm to do this.
Your table likely isn't created because you haven't committed yet, and your sql fails before it commits. It should work when you fix your 2nd sql statement.
You're not inserting the variables you've created into the table. You need to use parameters. There are two ways of parameterizing your sql statement. I'll show the named placeholders one:
c.execute("INSERT INTO ticketstest VALUES (:id, :subj, :created, :body)",
{'id':id, 'subj':subj, 'created':created, 'body':body}
)
Related
I use pyodbc in my python flask Project for the SQLite DB connection.
I know and understand SQL Injections but this is my first time dealing with it.
I tried to execute some
I have a function which concatenates the SQL String in my database.py file:
def open_issue(self, data_object):
cursor = self.conn.cursor()
# data_object is the issue i get from the user
name = data_object["name"]
text = data_object["text"]
rating_sum = 0
# if the user provides an issue
if name:
# check if issue is already in db
test = cursor.execute(f'''SELECT name FROM issue WHERE name = "{name}"''')
data = test.fetchall()
# if not in db insert
if len(data) == 0:
# insert the issue
cursor.executescript(f'''INSERT INTO issue (name, text, rating_sum)
VALUES ("{name}", "{text}", {rating_sum})''')
else:
print("nothing inserted!")
In the api.py file the open_issue() function gets called:
#self.app.route('/open_issue')
def insertdata():
# data sent from client
# data_object = flask.request.json
# unit test dictionary
data_object = {"name": "injection-test-table",
"text": "'; CREATE TABLE 'injected_table-1337';--"}
DB().open_issue(data_object)
The "'; CREATE TABLE 'injected_table-1337';--" sql injection has not created the injected_table-1337, instead it got inserted normally like a string into the text column of the injection-test-table.
So i don't really know if i am safe for the standard ways of SQL injection (this project will only be hosted locally but good security is always welcome)
And secondary: are there ways with pyodbc to check if a string contains sql syntax or symbols, so that nothing will get inserted in my example or do i need to check the strings manually?
Thanks a lot
As it turns out, with SQLite you are at much less risk of SQL injection issues because by default neither Python's built-in sqlite3 module nor the SQLite ODBC driver allow multiple statements to be executed in a single .execute call (commonly known as an "anonymous code block"). This code:
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = f"SELECT * FROM table1 WHERE txt='{thing}'"
crsr.execute(sql)
throws this for sqlite3
sqlite3.Warning: You can only execute one statement at a time.
and this for SQLite ODBC
pyodbc.Error: ('HY000', '[HY000] only one SQL statement allowed (-1) (SQLExecDirectW)')
Still, you should follow best practices and use a proper parameterized query
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = "SELECT * FROM table1 WHERE txt=?"
crsr.execute(sql, (thing, ))
because this will also correctly handle parameter values that would cause errors if injected directly, e.g.,
thing = "it's good to avoid SQL injection"
I'm new to python and I was trying to insert data from one table to another and the code is executing but it is not reflecting any changes in new table in which I want to insert data.
import mysql.connector
conn=mysql.connector.connect(user='root',password='',host='localhost',database='xyz')
mycursor=conn.cursor()
mycursor.execute("insert into newTable select * from oldTable group by mac,date,time order by mac")
I suppose it is because you are not committing the changes. Use the below code and see if it works.
conn.commit()
You need to use conn because that is the variable name for your connection to the database. A commit must be sent to the server, which in turn will commit your changes.
You need to add commit() to make the changes.
conn.commit()
So, I have the following code that inserts the data of an old database to a new one:
...
cur_old.execute("""SELECT DISTINCT module FROM all_students_users_log_course266""")
module_rows = cur_old.fetchall()
for row in module_rows:
cur_new.execute("""INSERT INTO modules(label) SELECT %s WHERE NOT EXISTS (SELECT 1 FROM modules WHERE label=%s)""", (row[0], row[0]))
...
The last line executes a query where labels are inserted into the new database table. I tested this query on pgAdmin and it works as I want.
However, when execute the script, nothing is inserted on the modules table. (Actually the sequences are updated, but none data is stored on the table).
Do I need to do anything else after I call the execute method from the cursor?
(Ps. The script is running till the end without any errors)
You forgot to do connection.commit(). Any alteration in the database has to be followed by a commit on the connection. For example, the sqlite3 documentation states it clearly in the first example:
# Save (commit) the changes.
conn.commit()
And the first example in the psycopg2 documentation does the same:
# Make the changes to the database persistent
>>> conn.commit()
As Evert said, the commit() was missing. An alternative to always specifying it in your code is using the autocommit feature.
http://initd.org/psycopg/docs/connection.html#connection.autocommit
For example like this:
with psycopg2.connect("...") as dbconn:
dbconn.autocommit=True
I'm writing a python script that would reset the database to an initial state (some hardcoded entries in every table). The db consists of multiple tables with primary and foreign keys.
Every time I would run the script, it should remove all the old entries in all of the tables, reset the primary key counter and insert the sample entries.
Currently I am trying to achieve this like this:
# Delete all the entries from the tables
cursor.execute("DELETE FROM table1")
cursor.execute("DELETE FROM table2")
cursor.execute("DELETE FROM table3")
# Reset the primary key counter and insert sample entries
cursor.execute("ALTER TABLE table1 AUTO_INCREMENT = 1")
cursor.execute("INSERT INTO table1(username, password) VALUES('user01', '123')")
cursor.execute("ALTER TABLE table2 AUTO_INCREMENT = 1")
cursor.execute("INSERT INTO table2(column1, column2) VALUES('column1_data', 'column2_data')")
This isn't working due to the presence of foreign keys in some tables (it won't let me delete them).
I generate the tables using a models.py script (I also use Django), so I thought I could solve this the following way:
remove the database programatically and create a new one with the same name
call the models.py script to generate empty tables in the db
insert sample data using the script I wrote
Is this a good solution or am I overlooking something?
I use scripts monthly to purge a transaction table, after archiving the contents.
Try using the 'truncate' command, ie.
truncate table [tablename];
It resets the counter (auto-increment) for primary key, automatically.
Then use your insert statements to populate base info.
Also, this preserves all of the table base settings (keys,indexes,.).
I’m looking for a way to take a result set and use it to find records in a table that resides in SQL Server 2008 – without spinning through the records one at a time. The result sets that will be used to find the records could number in the hundreds of thousands. So far I am pursuing creating a table in memory using sqlite3 and then trying to feed that table to a stored procedure that takes a table valued parameter. The work on the SQL Server side is done, the user defined type is created, the test procedure accepting a table valued parameter exists and I’ve tested it through TSQL and it appears to work just fine. In Python a simple in memory table was created through sqlite3. Now the catch, the only documentation I have found for accessing a stored procedure with a table valued parameter uses ADO.Net and VB, nothing in Python. Unfortunately, I’m not enough of a programmer to translate. Has anyone used a SQL Server stored procedure with a table valued parameter? Is there another approach I should look into?
Here are some links:
Decent explanation of table valued parameters and how to set them up in SQL and using in .Net
http://www.sqlteam.com/article/sql-server-2008-table-valued-parameters
http://msdn.microsoft.com/en-us/library/bb675163.aspx#Y2142
Explanation of using ADO in Python – almost what I need, just need the structured parameter type.
http://www.mayukhbose.com/python/ado/ado-command-3.php
My simple code
--TSQL to create type on SQL database
create Type PropIDList as Table
(Prop_Id BigInt primary key)
--TSQL to create stored procedure on SQL database. Note reference to
create procedure PropIDListTest #PIDList PropIDList READONLY
as
SET NOCOUNT ON
select * from
#PIDList p
SET NOCOUNT OFF
--TSQL to test objects.
--Declare variable as user defined type (table that has prop_id)
declare #pidlist as propidlist
--Populate variable
insert into #pidlist(prop_id)
values(1000)
insert into #pidlist(prop_id)
values(2000)
--Pass table variable to stored procedure
exec PropIDListTest #pidlist
Now the tough part – Python.
Here is the code creating the in memory table
import getopt, sys, string, os, tempfile, shutil
import _winreg,win32api, win32con
from win32com.client import Dispatch
from adoconstants import *
import sqlite3
conn1 = sqlite3.connect(':memory:')
c = conn1.cursor()
# Create table
c.execute('''create table PropList
(PropID bigint)''')
# Insert a row of data
c.execute("""insert into PropList
values (37921019)""")
# Save (commit) the changes
conn1.commit()
c.execute('select * from PropList order by propID')
# lets print out what we have to make sure it works
for row in c:
print row
Ok, my attempt at connecting through Python
conn = Dispatch('ADODB.Connection')
conn.ConnectionString = "Provider=sqloledb.1; Data Source=nt38; Integrated Security = SSPI;database=pubs"
conn.Open()
cmd = Dispatch('ADODB.Command')
cmd.ActiveConnection = conn
cmd.CommandType = adCmdStoredProc
cmd.CommandText = "PropIDListTest #pidlist = ?"
param1 = cmd.CreateParameter('#PIDList', adUserDefined) # I “think” the parameter type is the key and yes it is most likely wrong here.
cmd.Parameters.Append(param1)
cmd.Parameters.Value = conn1 # Yeah, this is probably wrong as well
(rs, status) = cmd.Execute()
while not rs.EOF:
OutputName = rs.Fields("Prop_ID").Value.strip().upper()
print OutputName
rs.MoveNext()
rs.Close()
rs = None
conn.Close()
conn = None
# We can also close the cursor if we are done with it
c.close()
conn1.close()
I have coded TVPs from ADO.NET before.
Here is a question on TVPs in classic ADO that I am interested in, sql server - Classic ADO and Table-Valued Parameters in Stored Procedure - Stack Overflow. It does not give a direct answer but alternatives.
The option of XML is easier, you have probably already considered it; it would require more server side processing.
Here is the msdn link for low level ODBC programming of TVPs.
Table-Valued Parameters (ODBC). This one is the closest answer if you can switch to ODBC.
You could pass a csv string to nvarchar(max) and then pass it to a CLR SplitString function, that one is fast but has default behaviour I disagree with.
Please post back what works or does not here.