I want to use sqlite3 to deal with data in Ubuntu with python. But I always failed and get errors. Codes related to database are as follows:
sqlite = "%s.db" % name
#connnect to the database
conn = sqlite3.connect(sqlite)
print "Opened database successfully"
c = conn.cursor()
#set default separator to "\t" in database
c.execute(".separator "\t"")
print "Set separator of database successfully"
#create table data_node
c.execute('''create table data_node(Time int,Node Text,CurSize int,SizeVar int,VarRate real,Evil int);''')
print "Table data_node created successfully"
node_info = "%s%s.txt" % (name,'-PIT-node')
c.execute(".import %\"s\" data_node") % node_info
print "Import to data_node successfully"
#create table data_face
data_info = "%s%s.txt" % (name,'-PIT-face')
c.execute('''create table data_face(Time int,Node Text,TotalEntry real,FaceId int,FaceEntry real,Evil int);''')
c.execute(".import \"%s\" data_face") % face_info
#get the final table : PIT_node
c.execute('''create table node_temp as select FIRST.Time,FIRST.Node,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio from tracer_temp FIRST,tracer_temp SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.Type='InData' AND SECOND.Type='OutInterests';''')
c.execute('''create table PIT_node as select A.Time,A.Node,B.SatisRatio,A.CurSize,A.SizeVar,A.VarRate,A.Evil from data_node A,node_temp B WHERE A.Time=B.Time AND A.Node=B.Node;''')
#get the final table : PIT_face
c.execute('''create table face_temp as select FIRST.Time,FIRST.Node,FIRST.FaceId,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio,SECOND.Packets from data_tracer FIRST,data_tracer SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.FaceId=SECOND.FaceId AND FIRST.Type='OutData' AND SECOND.Type='InInterests';''')
c.execute('''create table PIT_face as select A.Time,A.Node,A.FaceId,B.SatisRatio,B.Packets,ROUND(A.FaceEntry/A.TotalEntry,4),A.Evil from data_face as A,face_temp as B WHERE A.Time=B.Time AND A.Node=B.Node AND A.FaceId = B.FaceId;''')
conn.commit()
conn.close()
These sql-commands are right. When I run the code, it always shows sqlite3.OperationalError: near ".": syntax error. So how to change my code and are there other errors in other commands such as create table?
You have many problems in your code as posted, but the one you're asking about is:
c.execute(".separator "\t"")
This isn't valid Python syntax. But, even if you fix that, it's not valid SQL.
The "dot-commands" are special commands to the sqlite3 command line shell. It intercepts them and uses them to configure itself. They mean nothing to the actual database, and cannot be used from Python.
And most of them don't make any sense outside that shell anyway. For example, you're trying to set the column separator here. But the database doesn't return strings, it returns row objects—similar to lists. There is nowhere for a separator to be used. If you want to print the rows out with tab separators, you have to do that in your own print statements.
So, the simple fix is to remove all of those dot-commands.
However, there is a problem—at least one of those dot-commands actually does something:
c.execute(".import %\"s\" data_node") % node_info
You will have to replace that will valid calls to the library that do the same thing as the .import dot-command. Read what it does, and it should be easy to understand. (You basically want to open the file, parse the columns for each row, and do an executemany on an INSERT with the rows.)
Related
I am attempting to write an SQL query to insert rows into a table when the script is run. I have been able to get it working whilst duplicating data, but am unable to set up an insert that can determine whether or not the record is unique.
I have tried following these threads:
Most efficient way to do a SQL 'INSERT IF NOT EXISTS'
Python MySQLdb / MySQL INSERT IGNORE & Checking if Ignored
Currently, I can get the system to add all records, but due to the frequency the script is ran at the database is being filled with the same records. This is my current code:
for post in r.get_subreddit("listentothis").get_new(limit=5):
if 'youtube.com' in post.url or 'youtu.be' in post.url:
print(str(post.title))
print(str(post.url))
sql = """INSERT INTO listentothis2(title,url) VALUES ('%s', '%s') WHERE NOT EXISTS (SELECT 1 FROM listentothis2 WHERE url = '%s') """ % (post.title, post.url, post.url)
try:
cursor.execute(sql)
db.commit()
except:
db.rollback()
I have also tried:
sql = """INSERT IGNORE INTO listentothis2(title,url) VALUES ('%s', '%s') """ % (post.title, post.url)
However, this adds the 3 most recent records whether or not they are commited to the database already. Any help is greatly appreciated!
I have written a small test application using SQLite with Python 3.3:
import sqlite3
MDB = sqlite3.connect('D:\MDB.db') # create the db object
cursor = MDB.cursor() # assign a cursor
cursor.execute('''CREATE TABLE IF NOT EXISTS section (
Code INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
Description TEXT )
''')
cursor.execute('''DELETE FROM section''') # delete contents for reruns
cursor.execute('''INSERT INTO section
(Description)
VALUES (?)
''', ('Abdul, Paula',))
cursor.execute('''INSERT INTO section
(Description)
VALUES (?)
''', ('ABWH',))
print('Results:\n')
cursor.execute('''SELECT * FROM section''')
selection = cursor.fetchall()
for row in selection:
print('\t', row)
The SELECT statement shows the results expected (seeming to indicate that the row exists), but if I connect to the database with SQLite-Manager, the table exists but is empty, and if I try the same query with another script connected to the database, nothing is returned. Can anyone please explain what I am doing wrong?
You're not saving changes (calling MDB.commit).
I've written my first 'update' query in python, while it seems correct, I'm not sure how to receive back the output to confirm it worked..
This is supposed to load a CSV file and replace the values in the first column with those in the second:
def main():
try:
conn=psycopg2.connect("dbname='subs' user='subs' host='localhost' password=''")
except:
print "I am unable to connect to the database."
sys.exit()
with open("dne.txt", "r+") as f:
for line in f:
old = line.split(',')[0].strip()
new = line.split(',')[1].strip()
cur = conn.cursor()
cur.execute("UPDATE master_list SET subs = '{0}' WHERE subs = '{1}';".format(new, old))
conn.commit()
results = cur.fetchall()
for each in results:
print str(each)
if __name__=="__main__":
main()
I thought the results (UPDATE 1 for each change?) would come back as a tuple, but I got an error instead:
psycopg2.ProgrammingError: no results to fetch
I'm not sure if this means my query just didn't work and there were no updates, or if I can't use fetchall() like I'm trying to.
Any feedback or suggestions welcome!
The UPDATE statement won't return any values as you are asking the database to update its data not to retrieve any data.
By far the best way to get the number of rows updated is to use cur.rowcount. This works with other drivers too, like with Psycopg2 for Postgresql it's the same syntax.
cur.execute("UPDATE master SET sub = ('xyz') WHERE sub = 'abc'")
print(cur.rowcount)
A more roundabout way of checking the update is by running a SELECT against the table after updating it; you should get the data returned. In my example below the first SELECT will return the row(s) where the update will happen. The second SELECT after the update should then return no rows as you have already updated all fields. The third SELECT should return the rows you have updated, plus any that already existed with the 'xyz' value.
import sqlite3
def main():
try:
conn=sqlite3.connect(":memory:")
cur = conn.cursor()
cur.execute("create table master(id text, sub text)")
cur.execute("insert into master(id, sub) values ('1', 'abc')")
cur.execute("insert into master(id, sub) values ('2', 'def')")
cur.execute("insert into master(id, sub) values ('3', 'ghi')")
conn.commit()
except:
print("I am unable to connect to the database.")
sys.exit()
cur.execute("select id, sub from master where sub='abc'")
print(cur.fetchall())
cur.execute("UPDATE master SET sub = ('xyz') WHERE sub = 'abc'")
conn.commit()
cur.execute("select id, sub from master where sub='abc'")
print(cur.fetchall())
cur.execute("select id, sub from master where sub='xyz'")
print(cur.fetchall())
if __name__=="__main__":
main()
In PostgreSQL 9.5 or later you can add RETURNING * to end your query that then returns the modified rows.
PostgreSQL docs: https://www.postgresql.org/docs/9.5/dml-returning.html
Sometimes it is useful to obtain data from modified rows while they
are being manipulated. The INSERT, UPDATE, and DELETE commands all
have an optional RETURNING clause that supports this. Use of RETURNING
avoids performing an extra database query to collect the data, and is
especially valuable when it would otherwise be difficult to identify
the modified rows reliably.
I have created table using this create command as:
CREATE TABLE test_table(id INT PRIMARY KEY,name
VARCHAR(50),price INT)
i want to insert into this table wherein values are stored already in variable
bookdb=# name = 'algorithms'
bookdb-# price = 500
bookdb-# INSERT INTO test_table VALUES(1,'name',price);
I get the following error:
ERROR: syntax error at or near "name"
LINE 1: name = 'algorithms'
Can anyone point out the mistake and propose solution for the above?
Thanks in advance
Edit:
import psycopg2
import file_content
try:
conn = psycopg2.connect(database='bookdb',user='v22')
cur = conn.cursor()
cur.execute("DROP TABLE IF EXISTS book_details")
cur.execute("CREATE TABLE book_details(id INT PRIMARY KEY,name VARCHAR(50),price INT)")
cur.execute("INSERT INTO book_details VALUES(1,'name',price)")
conn.commit()
except:
print "unable to connect to db"
I have used the above code to insert values into table,variables name and price containing the values to be inserted into table are available in file_content python file and i have imported that file.The normal INSERT statement takes values manually but i want my code to take values which are stored in variables.
SQL does not support the concept of variables.
To use variables, you must use a programming language, such as Java, C, Xojo. One such language is PL/pgSQL, which you can think of as a superset of SQL. PL/PgSQL is often bundled as a part of Postgres installers, but not always.
I suggest you read some basic tutorials on SQL.
See this similar question: How do you use script variables in PostgreSQL?
don't have postgres installed here, but you can try this
import psycopg2
import file_content
try:
conn = psycopg2.connect(database='bookdb',user='v22')
cur = conn.cursor()
cur.execute("DROP TABLE IF EXISTS book_details")
cur.execute("CREATE TABLE book_details(id INT PRIMARY KEY,name VARCHAR(50),price INT)")
cur.execute("INSERT INTO book_details VALUES(1, '%s', %s)" % (name, price))
conn.commit()
except:
print "unable to connect to db"
If you are using PSQL console:
\set name 'algo'
\set price 10
insert into test_table values (1,':name',:price)
\g
I have created a database with MySQLdb.
In database I have a table with name student with columns:
id(is int),
id_user(is int),
f_name(is str),
l_name(is str)
I want to update a row.
My code is below:
db=mdb.connect(host="localhost", use_unicode="True", charset="utf8",
user="", passwd="", db="test")
# prepare a cursor object using cursor() method
cursor = db.cursor()
sql="""SELECT id_user FROM student"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
rows = cursor.fetchall()
the=int(7)
se=str('ok')
for row in rows:
r=int(row[0])
if r==the:
sql2 = """UPDATE student
SET f_name=%s
WHERE id_user = %s"""% (se,the)
# Execute the SQL command
cursor.execute(sql2)
# Commit your changes in the database
db.commit()
db.rollback()
# disconnect from server
db.close()
When I run it I take the error there is column with name ok why?
Can anyone help me find what I am doing wrong please?
str doesn't wrap its argument in quotation marks, so your statement is this:
UPDATE student SET f_name=ok WHERE id_user = 7
when it needs to be this:
UPDATE student SET f_name='ok' WHERE id_user = 7
So, either change this line:
SET f_name=%s
to this:
SET f_name='%s'
or else change this line:
se=str('ok')
to this:
se="'" + str('ok') + "'"
Though I recommend reading about SQL injection, which will become a concern as soon as you start using user-supplied data instead of hard-coded values.
You should run the query like this:
sql2 = """UPDATE student
SET f_name = %s
WHERE id_user = %s"""
cursor.execute(sql2, (se, the))
Don't use string interpolation, let the database driver handle passing the parameters for you. Otherwise you have to deal with syntax errors like this, or worse, SQL injection.
More details here.
You should always enclose your data with quotes.
Instead of %s solely use '%s' the only types you dont need it are numeric ones, but even there i would enclose %d with '%d' cos it is more save.
And you should use at least db.escape_string(your_data) before inserting or updating same values into your database.
Or have a look at the pdo-using style of mysqldb:
http://mysql-python.sourceforge.net/MySQLdb.html#some-examples
c=db.cursor()
max_price=5
c.execute("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < %s""", (max_price,))