MySQL SELECT from field in CSV in Python - python

I am currently having issues with performing a MySQL SELECT Query based on rows in a .csv file inside a python script;
#!/usr/bin/env python
import MySQLdb, csv, sys
db = MySQLdb.connect(host="hostname",
user="user",
passwd="password",
db="database")
cur = db.cursor()
customers=csv.reader(file("customers.csv"))
for row in customers(row):
cur.execute("select field from database.table where customernumber = %s;" % row)
cur.commit()
cur.close()
I am getting this;
Traceback (most recent call last):
File "script.py", line 17, in <module>
cur.execute("select field from database.table where field = %s;" % row)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '['1598']' at line 1")
The first row in the .csv file is 1598.
For some reason it is encasing this entry with '['1598']' hence the mySQL query failing,
Any ideas why it is encasing it and how to stop it?
Thanks in advance!
I have renamed all mysql details to defaults for DPA Reasons
Ashley


First, you need to fix your csv reader logic. Then you need to pass the value as an argument to execute, commit() is done on the connection not the cursor and finally - you are closing your connection in the loop itself.
import csv
with open('customers.csv') as f:
reader = csv.reader(f)
rows = list(reader)
for row in rows:
cur.execute("select field from database.table where customernumber = %s;", (row[0],))
con.commit()
con.close()


try this,
for row in customers:
print row , row[0]
row[0] should be first column value.


Change executing the SELECT to something like this:
cur.execute("SELECT field FROM database.table WHERE customernumber = %s", (row[0],))
If row[0] has the number, otherwise which ever index has it, like row[4].


cur.execute("select field from database.table where customernumber = %s;" % row[0])
You need to pass row[0] instead of row. row is a list and row[0] is the first element of that list.

Related

Python MySQL Update from Excel file doesn't work

I am able to insert and replace the data in my MySQL table.
But I want to update the column dLieferdatum if the column sku equals cArtNr.
The Excel file looks as follows:
My Python Code:
cursor = connection.cursor()
query = """UPDATE d032683a.jll99_deliverytime_import SET dLieferdatum %s WHERE sku = %s"""
for r in range(1, sheet.nrows):
cArtNr = sheet.cell(r,1).value
dLieferdatum = sheet.cell(r,2).value
values = (dLieferdatum, cArtNr)
cursor.execute(query, values)
connection.commit()
cursor.close()
connection.close()
My Compiler (cygwin) error message:
Traceback (most recent call last):
File "sql_update.py", line 63, in
cursor.execute(query, values)
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''2020-05-29' WHERE sku = '11330342'' at line 1")
Many thanks and best regards.

I think you are missing an equal sign in the update statement and that is why you are getting the error. I think if you add equal sign then you should be fine.
Also,if I were you, I would print the query in the console to see what is the exact query that is being sent to database and also for testing might run that query directly in database.
UPDATE d032683a.jll99_deliverytime_import SET dLieferdatum = %s WHERE sku = %s
You print(query) should look like below:
UPDATE d032683a.jll99_deliverytime_import SET dLieferdatum = '2020-05-29' WHERE sku = '11330342'

SQL concatenating with python

I am trying to update my mysql database field with a concatenation. I have to read my file line by line, and i need to append the existing string with the loaded line. I have to do it like this because my goal is to insert a 3gb long whitespace separated text file into one longtext field, and mysql only capable of handling 1gb text to insert.
The problem with my code is that if i add the field name to the concat function like seq=concat(seq, %s) I get a SQL syntax error, but when I add the field name as a variable, python acts like it's a string.
So short story long with this input file:
aaa
bbb
ccc
I want to have an updated mysql field like this:
aaabbbccc
But I get this: seqccc
Any idea how should i work with the fieldname to get this work?
import mysql.connector
connection = mysql.connector.connect(host='localhost',
database='sys',
user='Pannka',
password='???')
cursor = connection.cursor()
with open('test.txt', 'r') as f:
for line in f:
sql = "update linedna set seq=concat(%s, %s) where id=1"
val=('seq', line.rstrip())
print(line.rstrip())
cursor.execute(sql, val)
connection.commit()
cursor.close()
connection.close()
f.close()
print(0)

I think that you want:
sql = "update linedna set seq = concat(seq, %s) where id=1"
val=(line.rstrip())
cursor.execute(sql, val)
connection.commit()
This will append each new line at the end of the already existing database value in column seq.

Error loading log file data into mysql using cvs format and python

I am trying to take a data from a log file in cvs format, open the log file and inserting row by row into mysql. I am getting an error like this:
ERROR Traceback (most recent call last): File "/Users/alex/PycharmProjects/PA_REPORTING/padb_populate.py", line 26, in VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)', row) File "/Users/alex/anaconda/lib/python2.7/site-packages/MySQLdb/cursors.py", line 187, in execute query = query % tuple([db.literal(item) for item in args]) TypeError: not all arguments converted during string formatting.
import csv
import MySQLdb
mydb = MySQLdb.connect(host='192.168.56.103',
user='user',
passwd='pass',
db='palogdb')
cursor = mydb.cursor()
csv_data = csv.reader(file('/tmp/PALOG_DEMODATA-100.csv'))
for row in csv_data:
cursor.execute('INSERT INTO palogdb(RECEIVE_TIME,SERIAL,TYPE,SUBTYPE,COL1,TIME_GENERATED,SRC,DST,NATSRC,NATDST,RULE,\
SRCUSR,DSTUSR,APP,VSYS1,FROM,TO,INBOUND_IF,OUTBOUND_IF,LOGSET,COL2,SESSIONID,COL3,REPEATCNT,SOURCEPORT,NATSPORT,NATDPORT, \
FLAGS,PROTO,ACTION,BYTES,BYTES_SENT,BYTES_RECEIVED,PACKETS,START,ELAPSED,CATEGORY,COL4,SEQNO,ACTIONFLAGS,SRCLOC,DSTLOC,NONE, \
PKTS_SENT,PKTS_RECEIVED,SESSION_END_REASON) \
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)', row)
#close the connection to the database.
mydb.commit()
cursor.close()

Is it possible, that you don't have enough data in row for all your %s's? Maybe your row is interpreted as one value, and thus only the first %s is expanded? Try *row to expand the vector to values.
To debug, you could try to build the string passed to execute by some other method, e.g.
sql_string = 'INSERT ... VALUES ({}, {}, {})'.format(*row)
and print it. If you get such an error, you can check, whether the generated string looks reasonable...

python inserting single quotes (') around MySQL table name

I have a database called project1 with the following tables:
_systbl1
_systbl2
_systbl3
dataset1
dataset2
dataset3
MySQL user odbc will need to be granted SELECT permissions on dataset% tables whenever a new one is added.
To accomplish this, I'm using a simple python script, like so:
#!/usr/bin/python
import MySQLdb
db = MySQLdb.connect(
host="localhost",
user="user",
passwd="pass",
db="project1"
)
# Create Cursor Object
cur = db.cursor()
# get list of tables beginning with dataset
cur.execute("SHOW TABLES FROM project1 LIKE 'dataset%';")
# run GRANT statement for each table
for row in cur.fetchall() :
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;", row)
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`%`;", row)
Unfortunately, it gives me the following error:
Traceback (most recent call last):
File "mysql_query.py", line 20, in <module>
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;", row)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1146, "Table 'project1.'dataset1'' doesn't exist")
As you can see in the last line of the error, the problem is that python is putting a single quote around the table names when generating the query.
What am I missing here?

Do not use SQL parameters for table names. SQL parameters are escaped by the database adapter to not be interpreted as anything but literal values.
You'll have to interpolate those yourself instead, but be absolutely certain that your table name does not hold untrusted data (prevent SQL injection attacks):
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;" % row)
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`%%`;" % row)
(where the % character in the grant has been escaped by doubling it to %%).

Instead use:
cur.execute("GRANT SELECT ON `project`.`%s` TO `odbc`#`localhost`" % row)
This will not use the normal escaping of the input. Beware of a backtick in any of your table names, though.

sql = """CREATE TABLE IF NOT EXISTS `""" + project + """` ( `id` INT(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`))"""

With Python and SQLite, how do i read an item from the database into a variable?

I am a programming beginner, and would like to get a little help with learning SQLite and Python.
Currently, I have a database called "Status.db" which contains two columns. These columns are "stamp", type INT, and "messages", type TEXT, consecutively.
The code I am using to try to read one instance of messages, with a particular stamp, ID, into a variable "output" is as follows:
#cherrypy.expose
def comment(self, ID = None):
con = lite.connect('static/database/Status.db')
with con:
cur = con.cursor()
cur.execute("SELECT messages FROM Status WHERE stamp is ?", (ID,))
temp = cur.fetchone()
output = temp[0]
However, when i execute this code, I get an error message that reads:
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/cherrypy/_cprequest.py", line 606, in respond
cherrypy.response.body = self.handler()
File "/usr/lib/pymodules/python2.7/cherrypy/_cpdispatch.py", line 25, in __call__
return self.callable(*self.args, **self.kwargs)
File "proj1base.py", line 176, in comment
output = temp[0]
TypeError: 'NoneType' object is not subscriptable
I am wondering if anyone could clarify to me what this error means, and what i could do to make it work.
Thanks in advance!
EDIT:
Here is the code that created and writes into the database
#cherrypy.expose
def writeStatus(self, newStatus=None, post=None):
"""Update the status file with the most recent status update
by inserting a timestamp and new status into a database
"""
con = lite.connect('static/database/Status.db')
stamp = time()
with con:
cur = con.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS Status(stamp INT, messages TEXT)")
cur.execute("INSERT INTO Status VALUES(?, ?)", (stamp, newStatus))

That means that fetchone() is returning None, which means that your SQL query isn't returning any result rows.
This is probably because you should use = instead of is in your SQL:
SELECT messages FROM Status WHERE stamp = ?

Categories

Resources