New to SO and fairly new to coding, so doing my best to follow the appropriate protocols.
In my python script, I'm creating a new table and populating column names from a list, named 'dups'.
dups = ['Id', 'Name', 'Price', 'Rating']
I'm inputting this list as columns for the new table, called "SuperTable", via a for loop. See code below:
with new_db:
cur = new_db.cursor()
cur.execute("DROP TABLE IF EXISTS SuperTable")
for i in dups:
if i == dups[0]:
new_col = i.replace("'","")
cur.execute("CREATE TABLE SuperTable(%s)" % (new_col))
else:
cur.execute("ALTER TABLE SuperTable ADD COLUMN %s" % i)
I've looked around a lot and can't seem to identify what I'm doing wrong. This approach worked with Sqlite but I keep getting this same error for MySQLdb:
Traceback (most recent call last):
File "MySQL_SuperTable.py", line 125, in <module>
cur.execute("CREATE TABLE Super(%s)" % (new_col))
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')' at line 1")
Thanks to eggyal! He pointed out that MySQL columns require a datatype. This is what the code looks like now (I created a list of tuples to input the datatypes + column names via a for loop):
with new_db:
cur = new_db.cursor()
cur.execute("DROP TABLE IF EXISTS SuperTable")
for i in col_namestypes:
if i == col_namestypes[0]:
cur.execute("CREATE TABLE SuperTable(%s %s)" % (i))
else:
cur.execute("ALTER TABLE SuperTable ADD COLUMN %s %s" % i)
for i in new_table:
count = len(i)
question_marks = []
while a < count:
question_marks.append('%s')
a += 1
quests = ','.join(question_marks)
cur.executemany("INSERT INTO SuperTable VALUES(%s)" % quests, new_table)
Related
I have a database, and I wish to add multiple values to the same row. I am somewhat new with sqlite and databases, but I am learning. I know I can do this:
conn = sqlite3.connect('sqlitedb.db')
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS threadTable(threadName varchar(30)')
c.execute('INSERT INTO threadTable (threadName) Values(?), x.Name')
This works for me, but I want to pass Multiple variables into the table, like so:
conn = sqlite3.connect('sqlitedb.db')
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS threadTable(threadName varchar(30),threadKey varchar(10),threadID varchar(1000))')
c.execute('INSERT INTO threadTable (threadName),(threadKey),(threadID) VALUES(?,?,?)', (x.Name, x.Key, x.ID))
When I try this, I get this error:
Traceback (most recent call last):
File "Files/main.py", line 39, in <module>
c.execute('INSERT INTO threadTable (threadName),(threadKey),(threadID) VALUES(?,?,?)', (x.Name, x.Key, x.ID))
sqlite3.OperationalError: near ",": syntax error
you don't need parantheses around each column :
c.execute('INSERT INTO threadTable (threadName,threadKey,threadID) VALUES(?,?,?)', (x.Name, x.Key, x.ID))
I am new to coding and databases, I can not get the query to work if I write it long hand but I have a lot to carry out and want it in a function but cannot get it to work, it returns a parameters error
import mysql.connector
def connection_check_1(query, value):
mydb = mysql.connector.connect(
host="******",
user="*****",
passwd="*****",
database="****"
)
mycursor = mydb.cursor()
mycursor.execute(query, (value))
myresult = mycursor.fetchall()
mydb.close()
return myresult
value = "sheep"
query = 'select inlicence from licence where animal = %s'
myresult = connection_check_1(query, value)
print(myresult)
Here is the SQL table I have
create table licence
(
animal varchar (20) primary key,
inlicence int (1)
);
This is the error I get
Traceback (most recent call last):
File "*******************", line 20, in
myresult = connection_check_1(query, value)
File "********************", line 13, in connection_check_1
mycursor.execute(query, (value))
File "********************************************88", line 246, in execute
prepared = self._cnx.prepare_for_mysql(params)
File "/home/kev/PycharmProjects/test bed/venv/lib/python3.5/site-packages/mysql/connector/connection_cext.py", line 535, in prepare_for_mysql
raise ValueError("Could not process parameters")
ValueError: Could not process parameters
I have tried changing the way the query is written, changing it to fetchall().
Wrapping a value with () doesn't turn it in to a tuple. You probably meant to add a comma there:
mycursor.execute(query, (value,))
# Creates a one-element tuple-^
I'm using dynamic method to create table and then insert values into it. There are two lists that dynamically filled from different files and these lists could be increased or decreased in count as well as the values are changing every time. My problem here I managed to create the table with its fields dynamically but I still have some issues inserting values into it. here is my code:
f = open(filepath,"r")
pluginoutput= f.read()
pluginoptojson = json.loads(pluginoutput)
columnsnames = (pluginoptojson["columns"])
countcolumns = len(pluginoptojson["columns"])
count = 0
lst = []
for name in columnsnames:
if count < countcolumns:
lst.append(str(name))
count +=1
lst.append("caseid")
table_name = "test2"
createsqltable = """CREATE TABLE IF NOT EXISTS """ + table_name + " (" + " VARCHAR(50),".join(lst) + " VARCHAR(50))"
print createsqltable
c.execute(createsqltable)
conn.commit()
#c.close()
#conn.close()
#gc.collect()
rowsvalue = (pluginoptojson["rows"])
for row in rowsvalue:
lst2 =[]
rowelementscount = len(row)
for value in row:
lst2.append(str(value))
lst2.append("test")
insert_intotable = "INSERT INTO " + table_name + "(" + ",".join(lst) + ") VALUES (" + ",".join(lst2) +")"
print insert_intotable
c.execute(insert_intotable)
conn.commit()
c.close()
conn.close()
gc.collect()
the file content as an example is :
{"lines": [[2, 12121, 33, 44, "ff"], [2, 786, 33, 66, "ww"]], "columns": ["id", "testnumber", "size", "gnumber", "filenname"]}
the print out of interesting values int table is :
INSERT INTO test8(id,testnumber,size,gnumber,filenname) VALUES (2,786,33, 66,ww,test)
the error I'm getting is:
Traceback (most recent call last):
File "pp.py", line 50, in
c.execute(insert_intotable)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near 'ww,test)' at line 1")
Any help will be appreciated :)
I have created a webcrawler in python 2.7 and i am using mysqldb to insert data into database.
I have executed each function as a different script for different webpages, but after i put them into a single file as functions, the program shows error;
(After entering seed page and depth)
Traceback (most recent call last):
File "C:\Users\Chetan\Desktop\webCrawler.py", line 207, in
mainFunc(depth,url)
File "C:\Users\Chetan\Desktop\webCrawler.py", line 194, in mainFunc
lst=perPage(url)
File "C:\Users\Chetan\Desktop\webCrawler.py", line 186, in perPage
filterContent(url,page)
File "C:\Users\Chetan\Desktop\webCrawler.py", line 149, in filterContent
cursor.execute(sql)
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 202, in execute
self.errorhandler(self, exc, value)
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'s and specials." />\n
I can't seem to find any problem. Here is the code;
def metaContent(page,url):#EXTRACTS META TAG CONTENT
lst=[]
while page.find("<meta")!=-1:
start_link=page.find("<meta")
page=page[start_link:]
start_link=page.find("content=")
start_quote=page.find('"',start_link)
end_quote=page.find('"',start_quote+1)
metaTag=page[start_quote+1:end_quote]
page=page[end_quote:]
lst.append(metaTag)
#ENTER DATA INTO DB
i,j=0,0
while i<len(lst):
sql = "INSERT INTO META(URL, \
KEYWORD) \
VALUES ('%s','%s')" % \
(url,lst[i])
cursor.execute(sql)
db.commit()
def filterContent(page,url):#FILTERS THE CONTENT OF THE REMAINING PORTION
phrase = ['to','a','an','the',"i'm",\
'for','from','that','their',\
'i','my','your','you','mine',\
'we','okay','yes','no','as',\
'if','but','why','can','now',\
'are','is','also']
#CALLS FUNC TO REMOVE HTML TAGS
page = strip_tags(page)
#CONVERT TO LOWERCASE
page = page.lower()
#REMOVES WHITESPACES
page = page.split()
page = " ".join(page)
#REMOVES IDENTICAL WORDS AND COMMON WORDS
page = set(page.split())
page.difference_update(phrase)
#CONVERTS FROM SET TO LIST
lst = list(page)
#ENTER DATA INTO DB
i,j=0,0
while i<len(lst):
sql = "INSERT INTO WORDS(URL, \
KEYWORD) \
VALUES ('%s','%s')" % \
(url,lst[i])
cursor.execute(sql)
db.commit()
#<6>
def perPage(url):#CALLS ALL THE FUNCTIONS
page=pageContent(url)
#REMOVES CONTENT BETWEEN SCRIPT TAGS
flg=0
while page.find("<script",flg)!=-1:
start=page.find("<script",flg)
end=page.find("</script>",flg)
end=end+9
i,k=0,end-start
page=list(page)
while i<k:
page.pop(start)
i=i+1
page=''.join(page)
flg=start
#REMOVES CONTENT BETWEEN STYLE TAGS
flg=0
while page.find("<script",flg)!=-1:
start=page.find("<style",flg)
end=page.find("</style>",flg)
end=end+9
i,k=0,end-start
page=list(page)
while i<k:
page.pop(start)
i=i+1
page=''.join(page)
flg=start
metaContent(url,page)
lst=linksExt(url,page)
filterContent(url,page)
return lst#CHECK WEATHER NEEDED OR NOT
#<7>
crawled=[]
def mainFunc(depth,url):#FOR THE DEPTH MANIPULATION
if (depth):
lst=perPage(url)
crawled.append(url)
i=0
if (depth-1):
while i<len(lst):
if url[i] not in crawled:
mainFunc(depth-1,url[i])
i+=1
#CALLING MAIN FUNCTION
mainFunc(depth,url)
Please mention any error, especially in depth manipulation function( mainFunc()). Anything regarding improving the crawler would be helpful.
It is definitely sql error, your quotes are not being escaped.
Instead of this
sql = "INSERT INTO META(URL, \
KEYWORD) \
VALUES ('%s','%s')" % \
(url,lst[i])
cursor.execute(sql)
and this
sql = "INSERT INTO WORDS(URL, \
KEYWORD) \
VALUES ('%s','%s')" % \
(url,lst[i])
cursor.execute(sql)
Try this
sql = "INSERT INTO WORDS(URL, \
KEYWORD) \
VALUES (%s, %s)"
cursor.execute(sql, (url, lst[i]))
and this
sql = "INSERT INTO META(URL, \
KEYWORD) \
VALUES (%s, %s)"
cursor.execute(sql, (url, lst[i]))
Also you are using while but not incrementing i, instead you can use this
for keyword in lst:
sql = "INSERT INTO META(URL, \
KEYWORD) \
VALUES (%s, %s)"
cursor.execute(sql, (url, keyword))
In the recursive call of mainFunc you are calling main function,
main(depth-1,url[i])
There is no main function in your code.
change it to,
mainFunc(depth-1,url[i])
I am importing the specific data from excel sheet and dumping that data to mysql database.
But while doing that i am getting the error
$ python dtz_db.py
dtz_db.py:43: Warning: Field 'datetime' doesn't have a default value
cursor.execute(query2, values2)
dtz_db.py:43: Warning: Data truncated for column 'lease_start_date' at row 1
cursor.execute(query2, values2)
dtz_db.py:43: Warning: Data truncated for column 'lease_end_date' at row 1
cursor.execute(query2, values2)
dtz_db.py:43: Warning: Incorrect integer value: '' for column 'lease' at row 1
cursor.execute(query2, values2)
dtz_db.py:43: Warning: Incorrect integer value: '' for column 'leased' at row 1
cursor.execute(query2, values2)
Traceback (most recent call last):
File "dtz_db.py", line 44, in <module>
cursor.execute(query1, values1)
File "c:\Python27\lib\site-packages\MySQLdb\cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "c:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defau
lterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.IntegrityError: (1452, 'Cannot add or update a child row: a fo
reign key constraint fails (`dtz_new`.`property_property`, CONSTRAINT `lease_id_
refs_id_816819bc` FOREIGN KEY (`lease_id`) REFERENCES `property_propertylease` (
`id`))')
my python file is this cod is for selecting the specific data from the excel file and dump that data to mysql database
import xlrd
import MySQLdb
book = xlrd.open_workbook("dtz11.xls")
sheet = book.sheet_by_name("Sheet1")
database = MySQLdb.connect (host="localhost", user="root", passwd="", db="dtz_new")
cursor = database.cursor()
query1 = """INSERT INTO property_property( name, inpection_date) VALUES(%s, %s )"""
query2 = """INSERT INTO property_propertylease( lease_start_date, lease_end_date, lease, leased) VALUES(%s, %s, %s, %s)"""
for r in range(1, sheet.nrows):
gaurav2 = sheet.cell(r,1).value
gaurav3 = sheet.cell(r,2).value
gaurav8 = sheet.cell(r,18).value
gaurav9 = sheet.cell(r,19).value
gaurav10 = sheet.cell(r,20).value
gaurav11 = sheet.cell(r,21).value
values1 = (gaurav2, gaurav3)
values2 = (gaurav8, gaurav9, gaurav10, gaurav11)
cursor.execute(query2, values2)
cursor.execute(query1, values1)
cursor.close()
database.commit()
database.close()
print "dumped successfully"
columns = str(sheet.ncols)
rows = str(sheet.nrows)
print "I just imported "+ columns+ " columns and "+ rows+" rows to MySQL!"
and my db table schema is
Please help me to resolve this problem,,,, Thanks a lot
Your insert statement asserts that there will be 8 variables provided to the DB, but you only give it 7. Start there.
(I know neither python nor excel interaction so I haven't posted any code. I suspect the problem is entirely in that INSERT statement though.)
Edit: So the foreign key constraint error means that according to your schema, property_property's lease_id points to data in another table (property_propertylease). Since you haven't given this second table any data, the insert fails.
Put another way, your insert statement populates a parent table. The parent table is attempting to reference data in a child table that does not exist.