Develop Reference

SQLite query throws a syntax error in code but works fine when copied into sqlite db browser

Solution: I had an out of date version of sqlite installed which didn't support RANK(). I fixed this by installing Python3.7.4
I am trying to find the position of a user when they are sorted in descending order based on the sum of two columns. I tested this query in SQLite DB Browser and it returned the correct result, but when I copied it into my Python script it throws a syntax error at one of the "("
I have tried running the code and stopping it just before it executes the query so that I can copy the query verbatim into the db browser. When I do that it works perfectly, but when I execute the query in my script it fails
Here is the plain text query that the string concatenation returns:
SELECT top_rank FROM (
SELECT username, RANK() OVER(ORDER BY summed DESC) AS 'top_rank'
FROM (
SELECT SUM(positive_qc) - SUM(negative_qc) AS 'summed', username
FROM sub_activity
WHERE sub_name IN('cryptocurrency', 'cryptomarkets', 'cryptotechnology', 'blockchain')
GROUP BY username)
) WHERE username = 'someuser'
Here is the code I use to get that query:
select_str = "SELECT top_rank FROM(" + \
"SELECT " + self.KEY2_USERNAME + ", RANK() OVER(ORDER BY summed DESC) AS 'top_rank' " \
"FROM(" \
"SELECT SUM(" + self.KEY2_POSITIVE_QC + ") - SUM(" + self.KEY2_NEGATIVE_QC + ") " \
"AS 'summed', " + self.KEY2_USERNAME + " " \
"FROM " + self.TABLE_SUB_ACTIVITY + " " \
"WHERE " + self.KEY2_SUB_NAME + " IN('" + "', '".join(sub_list) + "') " \
"GROUP BY " + self.KEY2_USERNAME + ")" \
") WHERE " + self.KEY2_USERNAME + " = ?"
The exact error I get when executing it in the script is
sqlite3.OperationalError: near "(": syntax error
Update: I have narrowed down the error to the RANK () OVER(ORDER BY summed DESC) portion of the query

Give this a try... I aliased each subquery. ui and uo.
SELECT top_rank FROM (
SELECT username, RANK() OVER(ORDER BY summed DESC) AS 'top_rank'
FROM (
SELECT SUM(positive_qc) - SUM(negative_qc) AS 'summed', username
FROM sub_activity
WHERE sub_name IN ('cryptocurrency', 'cryptomarkets', 'cryptotechnology', 'blockchain')
GROUP BY username) ui
) uo WHERE username = 'someuser'

MySQL crashes during data transfer from large csv using LOAD DATA from python

I have a large csv file of 30 million rows(1.6 gb) and I am using pymysql to load the data from csv to mysql tables.
I have removed all constraints in table schema to make load faster and have also set timeout values to large values.
def setTimeOutLimit(connection):
try:
with connection.cursor() as cursor:
query = "SET GLOBAL innodb_lock_wait_timeout = 28800"
cursor.execute(query)
query2 = "SET innodb_lock_wait_timeout = 28800"
cursor.execute(query2)
query3 = "SET GLOBAL connect_timeout = 28800"
cursor.execute(query3)
query4 = "SET GLOBAL wait_timeout = 28800"
cursor.execute(query4)
query5 = "SET GLOBAL interactive_timeout = 28800"
cursor.execute(query5)
query6 = "SET GLOBAL max_allowed_packet = 1073741824"
cursor.execute(query6)
except:
conn.close()
sys.exit(" Could not set timeout limit ")
The data gets inserted into the table but I need to make one of the column as Primary Key and so I am creating another table that makes that column primary index by ignoring duplicate values. (tableName_1 is old table tableName is new table)
def createNewTableFromOld(connection, tableName):
try:
pprint( " Creating new table from old table with constraints" )
with connection.cursor() as cursor:
query = (" CREATE TABLE " + tableName +
" Like " + tableName + "_1")
cursor.execute(query)
query2 = (" ALTER TABLE " + tableName +
" ADD PRIMARY KEY(TimeStamp) ")
cursor.execute(query2)
query3 = (" INSERT IGNORE INTO " + tableName +
" SELECT * FROM " + tableName + "_1")
cursor.execute(query3)
query4 = ("DROP TABLE " + tableName + "_1")
cursor.execute(query4)
connection.commit()
except:
conn.close()
sys.exit(" Could not create table with Primary Key ")
During this method execution, somewhere after 5-6 minutes I get this error,
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([WinError 10054] An existing connection was forcibly closed by the remote host)')
And when I check services, MYSQL80 automatically crashed and stopped. I have also set max_allowed_packet_size to 1 gb in my.ini file and all timeouts are manually set to 8 hours. What could be the issue?
The original table schema is:
query = ("CREATE TABLE IF NOT EXISTS " + table + " ("
" TimeStamp DECIMAL(15, 3), " +
" Value DECIMAL(30, 11), " +
" Quality INT, " +
" TagName varchar(30) )"
)

I finally solved the issue by setting the innodb_buffer_pool_size in my.ini file to 2GB which was earlier only 4M.

Python MySQLdb Insert fails on MariaDB: Doesn't persist to new connection

The assert statement in the following code fails in an environment running mariadb, while it runs fine in an (older) environment running mysql. The mariadb environment seems to not persist the result of the insert statement. Connecting with a command line client confirms that the insert was not persisted.
import MySQLdb
def newCon():
return MySQLdb.connect('localhost', 'root', '', 'test')
def testStatement(con, aStatement):
# print "---"
# print aStatement
cur = con.cursor()
cur.execute(aStatement)
result = cur.fetchall()
cur.close()
# print result
return result
def test():
con = newCon()
testStatement(con, "DROP TABLE IF EXISTS TestTable")
testStatement(con, """CREATE TABLE TestTable (
id INT NOT NULL AUTO_INCREMENT,
AccountId VARCHAR(50),
PRIMARY KEY(id))""")
testStatement(con, "INSERT INTO TestTable (AccountId) VALUES ('XYZ')")
myCount1 = testStatement(con, "SELECT count(*) from TestTable")[0][0]
con.close()
con = newCon()
myCount2 = testStatement(con, "SELECT count(*) from TestTable")[0][0]
con.close()
assert myCount1 == myCount2, \
"count1 = " + str(myCount1) + " != count2 = " + str(myCount2)
test()
The environment in which it runs fine is:
CentOS 6.7
python 2.6.6
mysql-python 1.2.3
mysql server 5.1.73
The environment in which it fails is:
Fedora 21
python 2.7.8
mysql-python 1.2.3
mariadb server 10.0.21
The failed output reads:
Test2.py:10: Warning: Table 'mysql.table_stats' doesn't exist
cur.execute(aStatement)
Traceback (most recent call last):
File "Test2.py", line 37, in <module>
test()
File "Test2.py", line 35, in test
"count1 = " + str(myCount1) + " != count2 = " + str(myCount2)
AssertionError: count1 = 1 != count2 = 0
I don't know what the table_stats warning is about.
Question: Am I using the api correctly? Can anybody make this test pass with a mariadb environment?

You haven't committed the transaction.

Creating mysql table from dynamic list python

Having an issue creating table using dynamic list. I keep getting error next to Exit as shown:
Traceback (most recent call last): File "pp.py", line 54, in
c.execute(createsqltable) File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in
execute
self.errorhandler(self, exc, value) File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in
defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server
version for the right syntax to use near 'Exit VARCHAR(250),caseid
VARCHAR(250))' at line 1")
here is the code:
lst =['Offset', 'Name', 'PID', 'PPID', 'Thds', 'Hnds', 'Sess', 'Wow64', 'Start', 'Exit', 'caseid']
table_name = "test"
createsqltable = """CREATE TABLE IF NOT EXISTS """ + table_name + " (" + " VARCHAR(250),".join(lst) + " VARCHAR(250))"
#print createsqltable
c.execute(createsqltable)
conn.commit()

Since Exit is the keyword of sql, you must use `` to escape first.
lst =['Offset', 'Name', 'PID', 'PPID', 'Thds', 'Hnds', 'Sess', 'Wow64', 'Start', '`Exit`', 'caseid']
table_name = "test"
createsqltable = """CREATE TABLE IF NOT EXISTS """ + table_name + " (" + " VARCHAR(250),".join(lst) + " VARCHAR(250))"
#print createsqltable
c.execute(createsqltable)
conn.commit()

cant resolve cryptic error in Python/sqlite program

Im running into this error that I can't work out
Im writing some code in Python using tkinter interface to transfer data from a text file to sqlite.
first, here is the relevant code:
def submit_data(self):
self.new_filename = self.file_entry.get()
self.new_tablename = self.table_entry.get()
self.new_fieldname = self.field_entry.get().split(',')
#print(self.new_fieldname)
self.create_new.destroy()
from sqlite3 import dbapi2 as sqlite
con = sqlite.connect(self.new_filename)
cur = con.cursor()
cur.execute('CREATE TABLE ' + self.new_tablename + '(id INTEGER PRIMARY KEY)')
for field in self.new_fieldname:
cur.execute('ALTER TABLE ' + self.new_tablename + ' ADD ' + field)
with open(self.filename, 'r', encoding='latin-1') as self.the_file:
status = True
#_keynumber=1
while status:
_row = self._next_line()
if _row:
_entry_list = _row.split(',')
# add space after text line comma for formatting
_entry_list = ', '.join(_entry_list)
#print(_entry_list)
#entries = {'row': _keynumber, 'entry': _entry_list}
#row_entry = "INSERT INTO " + self.new_tablename + " VALUES(" + _entry_list + ")"
cur.execute("INSERT INTO " + self.new_tablename + " VALUES(" + _entry_list + ")")
#_colrange = range(_colamount)
#_keynumber+=1
else:
status = False
con.commit()
At the cur.execute("INSERT INTO " ... line (about 6 lines up) I get this error:
** cur.execute("INSERT INTO " + self.new_tablename + " VALUES(" + _entry_list + ")")
sqlite3.OperationalError: near ".": syntax error**
I have changed this around in many different ways. At one time I had the whole "INSERT INTO ... VALUES ...." string as a variable and used
cur.execute(*variable*)
when I did it this way the error was the same except "OperationalError: near "." was "OperationalError: near "of" ... and there was no 'of' anywhere.
Im really confused and frustrated. Someone break this down for my please??
Thanks
F
the text file lines its reading are set up like this:
A Big Star In Hollywood,Sandra Dickinson
so I had figured that if I use .join() to put a space after the comma then the string would be the equivalent of two VALUES for the INSERT INTO statement.

Remove
_entry_list = ', '.join(_entry_list)
and use
cur.execute("INSERT INTO " + self.new_tablename + "(" + ",".join(self.new_fieldname) +") VALUES(" + ",".join(("?" for i in xrange(len(_entry_list)))) + ")", _entry_list)
This will parameterize your query and automatically quote all value in _entry_list.
You still have to manually quote self.new_tablename and self.new_fieldname. This should be before you use them in any sql statements.

You need to quote your strings.
As written, your SQL statement is:
INSERT INTO foo VALUES(Hello there, world, I am, unquoted, string, not good)
You should use:
INSERT INTO foo VALUES("Hello there","world","I am","quoted","string","hooray")

I suggest you do the following:
a) Instead of executing the statement, print it to the console. Change the line:
cur.execute("INSERT INTO " + self.new_tablename + ...)
to:
print "INSERT INTO " + self.new_tablename + ...
b) Run your program after making this change. Take a look at the SQL statements that you print to the console. Are they valid SQL statements? Start a SQLite command-line, and copy/paste the statements produced by your program. Does SQLite give any errors when you try to execute the pasted statements?