I am newbie for sqlite3 (python user)
Already stored data in database based on below method, but not working this time. INSERT INTO execuated, no error report, but nothing stored in db.
I searched this topic a lot on this website, focused on "no commit". But I am pretty sure I finished "commit" and closed file correctly. After INSERT action, I could found table f65 in my db with header (column name),but no more data. (10 columns x 4k rows).
Key codes snippet as below.
df = pd.DataFrame(pd.read_excel(r'/Users/' + username + r'/Documents/working/addf6-5.xlsx', header=0))
df = df.replace('#NUM!', '')
value = chain(reversed(grouplist2), df.values.tolist())
for x in value:
x[4] = str(x[4])
x[7] = str(x[7])
conn = sqlite3.connect('veradb.db')
c = conn.cursor()
c.execute("DROP TABLE IF EXISTS f65")
conn.commit()
c.close()
conn.close()
conn = sqlite3.connect('veradb.db')
c = conn.cursor()
c.execute("CREATE TABLE IF NOT EXISTS f65 ('offer', 'code', 'desc', 'engine', 'cost', 'supplier', 'remark', 'podate', 'empty', 'border')")
c.executemany("INSERT INTO f65 VALUES (?,?,?,?,?,?,?,?,?,?)", value)
conn.commit()
c.close()
conn.close()
More details for explanation:
Cache data is in addf6-5, I fetched new data in grouplist2 (it's a list) and used chain(reversed(grouplist2), df.values.tolist())to combine data.
Why used
for x in value:
x[4] = str(x[4])
x[7] = str(x[7])
Because my pycharm report "Error binding parameter 4 - probably unsupported type."&"Error binding parameter 7 - probably unsupported type." So I do STR again though they should be TEXT according to SQLite by default.
I tried to test "value" via
for x in value:
x[4] = str(x[4])
x[7] = str(x[7])
print(x)
And found they could be printed correctly.Maybe this proved value (lists) was correct?
Some people may doubt if attribute missed in "CREATE TABLE" sentence, I can say Sqlite provided them TEXT by default is no typing and I execute these codes on other files and worked.
Looking forward to your help. Thank you in advance.
Related
I have a GUI interacting with my database, and MySQL database has around 50 tables. I need to search each table for a value and return the field and key of the item in each table if it is found. I would like to search for partial matches. ex.( Search Value = "test", "Protest", "Test123" would be matches. Here is my attempt.
def searchdatabase(self, event):
print('Searching...')
self.connect_mysql() #Function to connect to database
d_tables = []
results_list = [] # I will store results here
s_string = "test" #Value I am searching
self.cursor.execute("USE db") # select the database
self.cursor.execute("SHOW TABLES")
for (table_name,) in self.cursor:
d_tables.append(table_name)
#Loop through tables list, get column name, and check if value is in the column
for table in d_tables:
#Get the columns
self.cursor.execute(f"SELECT * FROM `{table}` WHERE 1=0")
field_names = [i[0] for i in self.cursor.description]
#Find Value
for f_name in field_names:
print("RESULTS:", self.cursor.execute(f"SELECT * FROM `{table}` WHERE {f_name} LIKE {s_string}"))
print(table)
I get an error on print("RESULTS:", self.cursor.execute(f"SELECT * FROM `{table}` WHERE {f_name} LIKE {s_string}"))
Exception: (1054, "Unknown column 'test' in 'where clause'")
I use a similar insert query that works fine so I am not understanding what the issue is.
ex. insert_query = (f"INSERT INTO `{source_tbl}` ({query_columns}) VALUES ({query_placeholders})")
May be because of single quote you have missed while checking for some columns.
TRY :
print("RESULTS:", self.cursor.execute(f"SELECT * FROM `{table}` WHERE '{f_name}' LIKE '{s_string}'"))
Have a look -> here
Don’t insert user-provided data into SQL queries like this. It is begging for SQL injection attacks. Your database library will have a way of sending parameters to queries. Use that.
The whole design is fishy. Normally, there should be no need to look for a string across several columns of 50 different tables. Admittedly, sometimes you end up in these situations because of reasons outside your control.
cHandler = myDB.cursor()
cHandler.execute('select UserId,C1,LogDate from DeviceLogs_12_2019') // data from remote sql server database
curs = connection.cursor()
curs.execute("""select * from biometric""") //data from my database table
lst = []
result= cHandler.fetchall()
for row in result:
lst.append(row)
lst2 = []
result2= curs.fetchall()
for row in result2:
lst2.append(row)
t = []
r = [elem for elem in lst if not elem in lst2]
for i in r:
print(i)
t.append(i)
for i in t:
frappe.db.sql("""Insert into biometric(UserId,C1,LogDate) select '%s','%s','%s' where not exists(select * from biometric where UserID='%s' and LogDate='%s')""",(i[0],i[1],i[2],i[0],i[2]),as_dict=1)
I am trying above code to insert data into my table if record not exists but getting error :
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '1111'',''in'',''2019-12-03 06:37:15'' where not exists(select * from biometric ' at line 1")
Is there anything I am doing wrong or any other way to achieve this?
It appears you have potentially four problems:
There is a from clause missing between select and where not exists.
When using a prepared statement you do not enclose your placeholder arguments, %s, within quotes. Your SQL should be:
Your loop:
Loop:
t = []
r = [elem for elem in lst if not elem in lst2]
for i in r:
print(i)
t.append(i)
If you are trying to only include rows from the remote site that will not be duplicates, then you should explicitly check the two fields that matter, i.e. UserId and LogDate. But what is the point since your SQL is taking care of making sure that you are excluding these duplicate rows? Also, what is the point of copying everything form r to t?
SQL:
Insert into biometric(UserId,C1,LogDate) select %s,%s,%s from DUAL where not exists(select * from biometric where UserID=%s and LogDate=%s
But here is the problem even with the above SQL:
If the not exists clause is false, then the select %s,%s,%s from DUAL ... returns no columns and the column count will not match the number of columns you are trying to insert, namely three.
If your concern is getting an error due to duplicate keys because (UserId, LogDate) is either a UNIQUE or PRIMARY KEY, then add the IGNORE keyword on the INSERT statement and then if a row with the key already exists, the insertion will be ignored. But there is no way of knowing since you have not provided this information:
for i in t:
frappe.db.sql("Insert IGNORE into biometric(UserId,C1,LogDate) values(%s,%s,%s)",(i[0],i[1],i[2]))
If you do not want multiple rows with the same (UserId, LogDate) combination, then you should define a UNIQUE KEY on these two columns and then the above SQL should be sufficient. There is also an ON DUPLICATE KEY SET ... variation of the INSERT statement where if the key exists you can do an update instead (look this up).
If you don't have a UNIQUE KEY defined on these two columns or you need to print out those rows which are being updated, then you do need to test for the presence of the existing keys. But this would be the way to do it:
cHandler = myDB.cursor()
cHandler.execute('select UserId,C1,LogDate from DeviceLogs_12_2019') // data from remote sql server database
rows = cHandler.fetchall()
curs = connection.cursor()
for row in rows:
curs.execute("select UserId from biometric where UserId=%s and LogDate=%s", (ros[0], row[2])) # row already in biometric table?
biometric_row = curs.fetchone()
if biometric_row is None: # no, it is not
print(row)
frappe.db.sql("Insert into biometric(UserId,C1,LogDate) values(%s, %s, %s)", (row[0],row[1],row[2]))
I have done the following code and I would like to ask the user to input how many new records want and after to fill column by column those records.
import MySQLdb
import mysql.connector
mydb = mysql.connector.connect(
host="localhost",
user="root",
passwd="Adam!977",
database="testdb1"
)
cur = mydb.cursor()
get_tables_statement = """SHOW TABLES"""
cur.execute(get_tables_statement)
tables = cur.fetchall()
table = tables(gene)
x=input("How many records you desire: ")
x
print "Please enter the data you would like to insert into table %s" %(table)
columns = []
values = []
for j in xrange(0, len(gene)):
column = gene[j][0]
value = raw_input("Value to insert for column '%s'?"%(gene[j][0]))
columns.append(str(column))
values.append('"' + str(value) + '"')
columns = ','.join(columns)
values = ','.join(values)
print columns
print values
The error that i get is about table gene( The table exist in db of SQL)
Traceback (most recent call last):
File "C:\Users\Admin\Desktop\π.py", line 25, in
table = tables(gene)
NameError: name 'gene' is not defined
Also, even I don't know if working properly the code. Please, I need help. Thank you
The error being returned by python is down to the lack of definition of a variable gene. In the following line you reference gene, without it existing:
table = tables(gene)
In the documentation for the python mysql connector, under cursor.fetchall() you'll notice that this method returns either a list of tuples or an empty list. It is therefore somewhat puzzling why you call tables as a function and attempt to pass a parameter to it - this is not correct syntax for accessing a list, or a tuple.
At the beginning of your code example you fetch a list of all of the tables in your database, despite knowing that you only want to update a specific table. It would make more sense to simply reference the name of the table in your SQL query, rather than querying all of the tables that exist and then in python selecting one. For example, the following query would give you 10 records from the table 'gene':
SELECT * FROM gene LIMIT 10
Below is an attempt to correct your code:
import mysql.connector
mydb = mysql.connector.connect(
host="localhost",
user="root",
passwd="Adam!977",
database="testdb1"
)
x=input("How many records you desire: ")
cur = mydb.cursor()
get_rows_statement = """SELECT * FROM gene"""
cur.execute(get_rows_statement)
results = cur.fetchall()
This should give you all of the rows within the table.
I have the following MySQL + Python code:
data = json.loads(decoded_response)
insert_values = []
cursor = cnx.cursor()
add_data = """INSERT INTO pb_ya_camps (camp_id,camp_name) VALUES (%s,%s) ON DUPLICATE KEY UPDATE VALUES (%s,%s)"""
for jsonData in data["data"]:
if "No" in jsonData["StatusArchive"]:
print("...processing campaign ",jsonData["Name"],"into the database.")
insert_values.append((jsonData["CampaignID"],jsonData["Name"]))
try:
cursor.executemany(add_data,(insert_values,insert_values))
Which at the moment produces the following error:
MYSQL ERROR: Failed processing format-parameters; 'MySQLConverter' object has no attribute '_tuple_to_mysql'
As far as I understand it is not liking the following:
cursor.executemany(add_data,(insert_values,insert_values))
I believe you can't do that with Python...but my problem probably derives from improper MySQL syntax. Could you please take a look at it?
INSERT INTO pb_ya_camps (camp_id,camp_name) VALUES (%s,%s) ON DUPLICATE KEY UPDATE VALUES (%s,%s)
I am not sure how to properly use the ON DUPLICATE KEY UPDATE without having to re-specify all the values... <<<--- that is the main problem.
I have read the following: LINK TO PREVIOUS EXAMPLE however I don't want to rely on KEY UPDATE col1 = VALUES(col1) because in a further part of my script I have too many columns to keep listing as part of col = value for each column...
Thank you!
Following MySQL Reference Manual, MySQL syntax for INSERT ... ON DUPLICATE KEY UPDATE is:
INSERT INTO table (`a`, `b`, `c`)
VALUES (1, 2, 3)
ON DUPLICATE KEY UPDATE `c` = `c` + 1;
So in your case (please note that writing either camp_id = %s or camp_id = VALUES(%s) is the same:
INSERT INTO `pb_ya_camps` (`camp_id`, `camp_name`)
VALUES (%s,%s)
ON DUPLICATE KEY UPDATE `camp_id` = VALUES(%s), `camp_name` = VALUES(%s)
More information about the syntax at the MySQL Reference Manual.
Here below is my code that I am using
con.execute("""
REPLACE INTO T(var1,var2,...,var300)VALUES(?,?,....?)""",(var1,var2,...,var300)
This statement works just fine if I have var1-var255 , once I have more than that it gave me an error...
So far, I was able to split T into 2 different times
con.execute("""
REPLACE INTO T(var1,var2,...,var150)VALUES(?,?,....?)""",(var1,var2,...,var150)
con.execute("""
REPLACE INTO T(var151,var152,...,var300)VALUES(?,?,....?)""",(var151,var152,...,var300)
This gave me no error , but my final value in table "T" would only values in the second execute statement , all of var1, var2, ... var 150 got replace with null
Have you tried using update instead?
MySQL documentation tells the following:
"REPLACE works exactly like INSERT, except that if an old row in the table has the same value as a new row for a PRIMARY KEY or a UNIQUE index, the old row is deleted before the new row is inserted"
There does not seem to be any inherent problem using more than 255 columns in MySQL, interfaced with MySQLdb:
import MySQLdb
import config
connection = MySQLdb.connect(
host = config.HOST, user = config.USER,
passwd = config.PASS, db = 'test')
cursor = connection.cursor()
cols = ['col{i:d}'.format(i =i) for i in range(300)]
types = ['int(11)']*len(cols)
columns = ','.join('{f} {t}'.format(f = f, t = t) for f, t in zip(cols, types))
sql = '''CREATE TABLE IF NOT EXISTS test (
id INT(11) NOT NULL AUTO_INCREMENT,
{c},
PRIMARY KEY (id)
)'''.format(c = columns)
cursor.execute(sql)
sql = '''REPLACE INTO test({c}) VALUES ({v})'''.format(
c = ','.join(cols),
v = ','.join(['%s']*len(cols)))
cursor.execute(sql, range(300))
result = cursor.fetchall()
This adds rows to test.test without a problem.