I'm new to using sqlite3 with python. I'm having trouble with making the insert_row function in my database class. From the manual on the execute function it appears that I'm doing everything correctly https://docs.python.org/2/library/sqlite3.html. This function does return a number which appears to be correct for lastrowid. However, when I access sqlite3 via command line the table is empty.
def insert_row(self, table, input_dict):
'''
(str, dict) -> int
Add a row to a table in the database
'''
count = 0
col_names = input_dict.keys()
insert_sql = 'INSERT INTO ' + self.sanitize(table) + ' (' + utils.list_to_char_seperated(col_names, ', ', False) + ') VALUES ('
for col in col_names:
count += 1
insert_sql += "'" + self.sanitize(input_dict[col]) + "'"
if count < len(col_names):
insert_sql += ', '
else:
insert_sql += ');'
cur = self.conn.cursor()
cur.execute(insert_sql)
return cur.lastrowid
Accessing via command line.
$ sqlite3
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open blood_cultures.db
sqlite> .tables
vein_draws
sqlite> SELECT * FROM vein_draws;
sqlite>
Also if I take the result of the insert_sql variable and insert it directly into the database via a command line connection this works. So I do not believe it is my insert statement.
Any ideas on what I am doing wrong here?
Related
I'm currently trying to speed up the process of piping our data through from 3rd party tools. I've written a function to push through insert statements to our MySQL database in bulk, but am getting the following error:
_mysql_exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now")
The function I've written is as follows, taking in record_array (an array of dicts containing the data to be pushed through to the MySQL table) and tablename (a string containing the name of the table):
def insert_update_bulk(self, record_array, tablename):
session = self.Session()
self.merge_query = ""
for r in record_array:
query = '''INSERT INTO {} ('''.format(tablename)
for k in r.keys():
query += '{}, '.format(k)
query = query[:-2] + ''') VALUES ('''
for v in r.values():
if type(v) == int:
query += '{},'.format(v)
else:
query += '"{}",'.format(str(v).replace('"', "'").replace('\n', ' ').replace('\r', ' '))
query = query[:-1] + ''') ON DUPLICATE KEY UPDATE '''
for k in r.keys():
if type(r[k]) == int:
query += '{}={},'.format(k, str(r[k]).replace('"', "'")).replace("'None'", 'NULL')
else:
query += '{}="{}",'.format(k, str(r[k]).replace('"', "'")).replace("'None'", 'NULL')
query = query[:-1] + '; '
self.merge_query += query
session.execute(self.merge_query.replace('%', '%%'))
self.Session.close_all()
I have a large csv file of 30 million rows(1.6 gb) and I am using pymysql to load the data from csv to mysql tables.
I have removed all constraints in table schema to make load faster and have also set timeout values to large values.
def setTimeOutLimit(connection):
try:
with connection.cursor() as cursor:
query = "SET GLOBAL innodb_lock_wait_timeout = 28800"
cursor.execute(query)
query2 = "SET innodb_lock_wait_timeout = 28800"
cursor.execute(query2)
query3 = "SET GLOBAL connect_timeout = 28800"
cursor.execute(query3)
query4 = "SET GLOBAL wait_timeout = 28800"
cursor.execute(query4)
query5 = "SET GLOBAL interactive_timeout = 28800"
cursor.execute(query5)
query6 = "SET GLOBAL max_allowed_packet = 1073741824"
cursor.execute(query6)
except:
conn.close()
sys.exit(" Could not set timeout limit ")
The data gets inserted into the table but I need to make one of the column as Primary Key and so I am creating another table that makes that column primary index by ignoring duplicate values. (tableName_1 is old table tableName is new table)
def createNewTableFromOld(connection, tableName):
try:
pprint( " Creating new table from old table with constraints" )
with connection.cursor() as cursor:
query = (" CREATE TABLE " + tableName +
" Like " + tableName + "_1")
cursor.execute(query)
query2 = (" ALTER TABLE " + tableName +
" ADD PRIMARY KEY(TimeStamp) ")
cursor.execute(query2)
query3 = (" INSERT IGNORE INTO " + tableName +
" SELECT * FROM " + tableName + "_1")
cursor.execute(query3)
query4 = ("DROP TABLE " + tableName + "_1")
cursor.execute(query4)
connection.commit()
except:
conn.close()
sys.exit(" Could not create table with Primary Key ")
During this method execution, somewhere after 5-6 minutes I get this error,
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([WinError 10054] An existing connection was forcibly closed by the remote host)')
And when I check services, MYSQL80 automatically crashed and stopped. I have also set max_allowed_packet_size to 1 gb in my.ini file and all timeouts are manually set to 8 hours. What could be the issue?
The original table schema is:
query = ("CREATE TABLE IF NOT EXISTS " + table + " ("
" TimeStamp DECIMAL(15, 3), " +
" Value DECIMAL(30, 11), " +
" Quality INT, " +
" TagName varchar(30) )"
)
I finally solved the issue by setting the innodb_buffer_pool_size in my.ini file to 2GB which was earlier only 4M.
I scripted in Python SQL calls to my MonetDB server (which I verify is running, of course). When I print the calls instead of calling them, the commands look OK, but if I run the original script, it does not crash, it does use the CPU and memory, but nothing is changed in the database, not even the first line is executed. Why?
The Python script looks like this:
# script to merge tables in MonetDB
import re
from monetdb import mapi
server = mapi.Server()
server.connect(hostname="localhost", port=50000, username="monetdb", password="monetdb", database="dbfarm", language="sql")
def tablemerge(stub,yearlist):
for year in yearlist:
# server.cmd('ALTER TABLE %s_%d ADD COLUMN "year" INTEGER DEFAULT %d;' % (stub,year,year))
print 'ALTER TABLE %s_%d ADD COLUMN "year" INTEGER DEFAULT %d;' % (stub,year,year)
newstub = re.sub(r'sys.ds_chocker_lev_', r'', stub)
if year == yearlist[0]:
unioncall = 'CREATE TABLE %s AS SELECT * FROM %s_%d ' % (newstub,stub,year)
else:
unioncall += 'UNION ALL SELECT * FROM %s_%d ' % (stub,year)
unioncall += ';'
server.cmd(unioncall)
# print unioncall
for year in yearlist:
server.cmd('DROP TABLE %s_%d;' % (stub,year))
# print 'DROP TABLE %s_%d;' % (stub,year)
print '%s done.' % stub
for stub in ['civandr']:
tablemerge('sys.ds_chocker_lev_%s' % stub,xrange(1998,2013))
E.g. the first call would be:
ALTER TABLE sys.ds_chocker_lev_civandr_1998 ADD COLUMN "year" INTEGER DEFAULT 1998;
But not even this happens. There is no year column in the table.
Or could I run the script in the console with more output than what I print myself?
Do commit! By default, the autocommit parameter is set to False. You can either do:
server.connect(hostname="localhost", port=50000, username="monetdb", password="monetdb", database="dbfarm", language="sql", autocommit=True)
or simply run:
connection.commit()
connection = monetdb.sql.connect(username=username,password=password,hostname=hostname,port=port,database=database)
cursor = connection.cursor()
cursor.execute('create table test (id int, name varchar(50));')
connection.commit()
This is my first post here and I found so many answers here I am really confident, that some can help me. Because I work with Python for only about half a year now, it maybe is a very stupid beginners question. Forgive me so far...
In my project I have several different functions. Two of them are listed below. mysqlLogbookIndex is a thread witch show refresh a list with the names of some table from a database. The connection to the DB is existing at the moment, the function is called and so far it works fine. At the end I can see a tuple containing all the table names.
But now the second function, named create_flight comes in. It is a callback for a Tkinter button and creates some new tables in my database. It uses the same preopened connection than mysqlLogbookIndex.
I expected to see the new tables is my tuple in the next cycle of mysqlLogbookIndex but what happened is that result turned into a None.
Do you know why?
Widget function:
def create_flight(self):
# *********************************************************************************************
# * Create new flight table *
# *********************************************************************************************
if self.sql_write.get():
# ********************************************
# * Try to connect message *
# ********************************************
self.printLog(self.lbl_sql_write, self.LANG['tryCreateTable'], 'normal')
logging.info('Creating table for flight tracking...')
# ********************************************
# * Create name tables *
# ********************************************
event = self.oprSett['mysql']['event']
now = str(int(time.time()))
mainName = "tbl_%s_%s_" % (event, now)
trackingTable = mainName + "flighttrack"
logging.debug('Name of tracking table: %s', trackingTable)
unitsTable = mainName + "units"
logging.debug('Name of units table: %s', unitsTable)
headerTable = mainName + "header"
logging.debug('Name of header table: %s', headerTable)
# ********************************************
# * Read SQL parameter *
# ********************************************
logging.debug('Reading CSV file for table structure...')
csvFile = "config/newFlight.csv"
try:
sqlCsv = csv.reader(open(csvFile, 'rb'),
delimiter = ',',
quotechar = '"',
quoting = csv.QUOTE_ALL
)
except:
msg = csvFile
msg += "\n\n"
msg += self.LANG['e13']
tkMessageBox.showerror("Error 13", msg)
self.printLog(self.lbl_sql_write, self.LANG['e13'], 'error')
logging.error('File not found!')
#print "[Error 13] " + self.LANG['e13']
return 0
# Transfer data from CSV file into own list
sqlVars = []
for row in sqlCsv:
if len(row) == 4 and row[0][0] != "#": # No comment
sqlVars.append(row)
# *************************************************
# * Create SQL statement to create tracking table *
# *************************************************
# Head for creating new table
sql = "CREATE TABLE IF NOT EXISTS `%s` (\n" % trackingTable
sql += " `ID` int(11) NOT NULL AUTO_INCREMENT,\n" # Becomes primary key
# Parse SQL variables from CSV file
for row in sqlVars:
if len(row[2]) > 0: # Data type requires length
sql += " `%s` %s(%s) NOT NULL COMMENT '%s',\n" % (row[0], row[1], row[2], row[3])
else: # Data type not requires length
sql += " `%s` %s NOT NULL COMMENT '%s',\n" % (row[0], row[1], row[3])
# Footer of SQL statement for creating new table
sql += " PRIMARY KEY (`ID`)\n"
sql += ") ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci AUTO_INCREMENT=0;\n"
sql += "\n"
# In debug mode print SQL statement to console
#logging.debug('SQL statement to create tracking table:\n%s', sql)
# **********************************************
# * Create SQL statement to create units table *
# **********************************************
# Head for creating new table
sql += "CREATE TABLE IF NOT EXISTS `%s` (\n" % unitsTable
sql += " `ID` int(11) NOT NULL AUTO_INCREMENT,\n" # Becomes primary key
sql += " `Dataref` varchar(10) COLLATE latin1_german1_ci NOT NULL,\n"
sql += " `Unit` varchar(10) COLLATE latin1_german1_ci NOT NULL,\n"
sql += " PRIMARY KEY (`ID`)\n"
sql += ") ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci AUTO_INCREMENT=1 ;\n"
sql += "\n"
# Parse SQL variables from CSV file
for row in sqlVars:
# Insert units in tbl_units
sql += "INSERT INTO %s VALUES ('', '%s', '%s');\n" % (unitsTable, row[0], row[3])
sql += "\n"
# In debug mode print SQL statement to console
#logging.debug('SQL statement to create units table:\n%s', sql)
# ***********************************************
# * Create SQL statement to create header table *
# ***********************************************
# Head for creating new table
sql += "CREATE TABLE IF NOT EXISTS `%s` (\n" % headerTable
#sql += " `ID` int(11) NOT NULL AUTO_INCREMENT,\n" # Becomes primary key
sql += " `Parameter` char(21) COLLATE latin1_german1_ci NOT NULL,\n"
sql += " `Value` varchar(100) COLLATE latin1_german1_ci NOT NULL,\n"
sql += " PRIMARY KEY (`Parameter`)\n"
sql += ") ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci AUTO_INCREMENT=1 ;\n"
sql += "\n"
# IGC syntax from: http://carrier.csi.cam.ac.uk/forsterlewis/soaring/igc_file_format/igc_format_2008.html
# Adding header parameters, some values are coming later
sql += "INSERT INTO %s VALUES ('AXXX001', '');\n" % (headerTable) # Manufacturer code
sql += "INSERT INTO %s VALUES ('HFFXA', '035');\n" % (headerTable) # Fix accuracy
sql += "INSERT INTO %s VALUES ('HFDTE', '');\n" % (headerTable) # UTC date of flight
sql += "INSERT INTO %s VALUES ('HFPLTPILOT', '');\n" % (headerTable) # Pilots name
sql += "INSERT INTO %s VALUES ('HFGTYGLIDERTYPE', 'KA8B');\n" % (headerTable) # Glider type
sql += "INSERT INTO %s VALUES ('HFGIDGLIDERID', 'D1389');\n" % (headerTable) # Glider callsign
sql += "INSERT INTO %s VALUES ('HFDTM100DATUM', 'WGS-1984');\n" % (headerTable) # GPS datum
sql += "INSERT INTO %s VALUES ('HFGPSGPS', 'X-PLANE 10');\n" % (headerTable) # Manufacturer of GPS module
sql += "INSERT INTO %s VALUES ('HFFTYFRTYPE', 'FLORIANMEISSNER,HCM');\n" % (headerTable) # Logger type
sql += "INSERT INTO %s VALUES ('HFRFWFIRMWAREVERSION', '%s');\n" % (headerTable, self.VERSION) # Firmware version
sql += "INSERT INTO %s VALUES ('HFRHWHARDWAREVERSION', '%s');\n" % (headerTable, self.XPLANEVERSION) # Hardware version
sql += "INSERT INTO %s VALUES ('HFCCLCOMPETITIONCLASS', 'CLUB');\n" % (headerTable) # Competition class
# ********************************************
# * Handover SQL statement to create table *
# * to DB. *
# ********************************************
logging.debug('SQL statement to create all tables:\n%s', sql)
try:
cur = self.conn.cursor()
cur.execute(sql)
cur.close()
except pymysql.Error, e:
tkMessageBox.showerror("Error 9", self.LANG['e9'])
self.printLog(self.lbl_sql_write, self.LANG['e9'], 'error')
#print "[Error 9] " + self.LANG['e9']
logging.error('Could not create tracking table!')
logging.debug(e)
return 0
# ********************************************
# * Print success message *
# ********************************************
logging.info('Tracking table created.')
self.printLog(self.lbl_sql_write, self.LANG['doneCreateTable'], 'success')
text = self.LANG['flightId'] + " " + trackingTable
self.printLog(self.lbl_sql_write, text, 'normal', timestamp=False)
# Make trackingTable public
self.tableName = trackingTable
# Enable tambour register to write in database
self.tambourInMysql = True
Refresh Loop:
def mysqlLogbookIndex(self):
delay = float(self.oprSett['logbook']['refresh_delay']) / 1000
# ********************************************************************
# * Run only if MySQL connection from writeMysql.py is active *
# ********************************************************************
while self.sql_write.get():
# ********************************************
# * Query to get list with tables *
# ********************************************
logging.info('Querying list with tables from database')
dbName = self.oprSett['mysql']['db']
sql = "SELECT TABLE_NAME\n"
sql += "FROM `information_schema`.`TABLES`\n"
sql += "WHERE `TABLE_SCHEMA` LIKE '%s'" % dbName
try:
cur = self.conn.cursor()
cur.execute(sql)
result = cur.fetchall()
cur.close()
except pymysql.Error, e:
logging.warning('Could not take flights from database!')
logging.debug(e)
self.printLog(self.lbl_sql_write, self.LANG['e20'], 'error')
#continue
else:
logging.info('Logbook refreshed.')
print result
Thanks in advance for all helpful posts...
Found the error more or less. It seems to be a MySQL issue.
In my original idea I connected to DB "hcm" and tried to get some table informations from "information_schema" DB. If the "hcm" was changed the query returned None.
Solution:
I established a second independent MySQL connection in my function for the refresh loop and connected to "information_schema" directly.
I can't show the data from database sqlite in python.
connection = sqlite3.connect('db')
connection.cursor().execute('CREATE TABLE IF NOT EXISTS users ( \
id TEXT, \
name TEXT, \
avatar TEXT \
)')
# In cycle:
query = 'INSERT INTO users VALUES ("' + str(friend.id) + '", "' + friend.name + '", "' + friend.avatar +'" )'
print query
connection.cursor().execute(query)
connection.commit()
# After cycle
print connection.cursor().fetchall()
Sample output of query variable:
INSERT INTO users VALUES ("111", "Some Name", "http://avatar/path" )
In result, fetchall returns empty tuple. Why?
UPD
Forgotten code:
connection.cursor().execute('SELECT * FROM users')
connection.cursor().fetchall()
→
[]
INSERT does not return data. To get the data back out, you'll have to issue a SELECT statement.
import sqlite3
con = sqlite3.connect("db")
con.execute("create table users(id, name, avatar)")
con.execute("insert into users(id, name, avatar) values (?, ?, ?)", (friend.id, friend.name, friend.avatar))
con.commit()
for row in con.execute("select * from users")
print row
con.close()
Because the create table string as displayed is syntactically invalid Python, as is the insert into string.
Actually, the answer to your first question is: because you use different cursors.
connection.cursor() creates a new cursor in the connection you created before. fetchall() gives you the results of the query you executed before in that same cursor. I.e. what you did was this:
# After cycle
cursor1 = connection.cursor()
cursor1.execute('SELECT * FROM users')
cursor2 = connection.cursor()
cursor2.execute("")
cursor2.fetchall()
What you should have done was this:
# After cycle
cursor = connection.cursor()
cursor.execute('SELECT * FROM users')
print cursor.fetchall()