I am trying to populate an azure sql database using pyodbc.
I have around 14000item to push into the database per months, and running and executing each single upsert on it is really long.
I am so trying to run them in batch and push them all at once. Sadly, when doing this I am not getting anything in the database. Is there any way to make this works ?
i = 0
queryUpdate = ""
for values in valuesList:
if i == 100:
i = 0
queryUpdate = queryUpdate.replace('\'NULL\'', "NULL") #set null to real sql null
queryUpdate = queryUpdate.replace("True", "1") #replace true with 1 to match the bit type
queryUpdate = queryUpdate.replace("False", "0") #replace false with 0 to match the bit type
try:
cursor.execute(queryUpdate)
except pyodbc.Error as ex:
print(f"[x] Error when running the query:\n[x] {queryUpdate}\n[x] Exception: {ex}")
queryUpdate = ""
queryUpdate += f"""\n
if exists (select * from {tableName} with (updlock,serializable) where {colNames[keyIndex]} = {values[keyIndex]})
begin
update {tableName} set """
for i in range(len(colNames)):
queryUpdate += f"{colNames[i]} = '{values[i]}'"
if i != len(colNames) - 1:
queryUpdate += ","
queryUpdate += f"""
where {colNames[keyIndex]} = {values[keyIndex]}
end
else
begin
insert into {tableName} ({','.join(colNames)})
values {tuple(values)}
end;
"""
i+=1
try:
conn.commit()
except pyodbc.Error as ex:
print(f"[x] Error when commiting to the database.\n[x] Exception: {ex}")
else:
print("[+] Commit confirmed !")
cursor.close()
conn.close()
print("[+] Connection closed.")
Am attempting to use SQLAlchemy ORM to update a particular row in the database but because I want to keep history of everything that has ever happened to the row I want to version off the record and keep the same primary key i.e. the primary key isn't unique. So when I try to version off the old record and create a new record I get the following error"
sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'category' expected to update 1 row(s); 2 were matched.
trying to use the following function:
def update_category(db: Session, category: CategoryUpdate, category_code: str) -> CategoryTable:
"""update the category by ID where current_flag = 1"""
cat = db.query(CategoryTable).filter(CategoryTable.current_flag == "1", CategoryTable.category_code == category_code).first()
if cat is None:
raise Exception("Category not found")
cat.effective_to = datetime.now()
cat.current_flag = "0"
db.commit()
db.refresh(cat)
c = CategoryIn(**cat.__dict__)
c.category_name = category.category_name
# c.category_key = cat.category_key
c.category_class = category.category_class if category.category_class else c.category_class
c.effective_from = c.effective_to
c.effective_to = None
c.current_flag = "1"
c_tab = CategoryTable(**c.dict())
c_tab.category_key = cat.category_key
db.add(c_tab)
db.commit()
db.refresh(c_tab)
return c_tab
any thoughts?
Figured it out, you can import the update function from sqlalchemy if you have an update that is more complicated than just using the primary key
stmt = update(CategoryTable).where(CategoryTable.category_key == cat.category_key,
CategoryTable.current_flag == '1', CategoryTable.effective_from == cat.effective_from).values(
effective_to=effective_date, current_flag='0').execution_options(synchronize_session="fetch")
db.execute(stmt)
I need to set some user meta in my wordpress through local python script. Hence I can't use the WP update_user_meta for it - it has to be done manually.
import mysql.connector as mysql
cnx = mysql.connect(host=HOST, database=DATABASE, user=USER, password=PASSWORD)
cursor = cnx.cursor()
get_meta = ("SELECT * FROM `ff_usermeta` WHERE `user_id`= 1 AND (`meta_key`='nickname' OR `meta_key`='info' OR `meta_key`='bg' OR `meta_key`='avatar' OR `meta_key`='profile_updated')")
cursor.execute(get_meta)
meta = cursor.fetchall()
#some processing of the result
cursor.execute(q, (...))
cnx.commit()
cursor.close()
cnx.close()
Now I need to check if the result has meta with each of the keys.
If the key already exists for this user, it needs to run UPDATE for this meta.
If this user still has no meta of this key, it has to INSERT new row.
if(there's no 'nickname' in meta_key on either of 5 or less rows):
q = ("INSERT INTO `ff_usermeta` ...")
else:
q = ("UPDATE `ff_usermeta` ...")
...and 4 more times like that?.. Seems like a good place for a cycle, but I don't really like the idea to make it 5x queues, especially since there might be more fields in the future.
I was thinking along the lines of searching the fetchall result for matches in meta_key, and if found, adding required data to one array, if not - to another. And then just running one update and one insert at the end, assuming both are not empty. If I were to write it in semi-php style, it would look roughly like this:
if(in_array("nickname", meta))
for_update .= "`nickname`='"+data[0]+"', "
else:
fields .= "`nickname`, "
vals .= "'"+data[0]+"', "
if(in_array("bg", meta)):
for_update .= "`bg`='"+data[1]+"', "
else:
fields .= "`bg`, "
vals .= "'"+data[1]+"', "
if(for_update):
update = ("UPDATE `ff_usermeta` SET "+for_update+" WHERE 1")
if(fields):
insert = ("INSERT INTO `ff_usermeta`("+fields+") VALUES ("+vals+")")
But absolutely no clue how to translate it correctly to python. I had to google it up to things like "why dot not working to add one string to another". Any advice? Or perhaps there is a better way? Thanks!
It is not complete, you can not update your rows in that way.
But with this you can start to make your query
The frist select gets exactly 1 row, if the user_id exists.
The user_id doesn't seem the right choice for this, but to get what you can do it is enough.
If the query doesn't have an entry, the it will insert some data you get from anywhere
The update as the insert are in that form wrong as you have to insert 5 new orws or update max 5 rows, but that is more for you to programm
import mysql.connector as mysql
HOST = "localhost"
DATABASE = ""
USER = "root"
PASSWORD = "mypassword"
cnx = mysql.connect(host=HOST, database=DATABASE, user=USER, password=PASSWORD)
cnx = mysql.connect(host=HOST, database=DATABASE, user=USER, password=PASSWORD)
cursor = cnx.cursor()
user_id = 1
get_meta = ("""SELECT umeta_id, user_id , MAX(IF( `meta_key`='nickname', meta_value,'')) AS 'nickname' , MAX(IF( `meta_key`='info', meta_value,'')) AS 'info' , MAX(IF( `meta_key`='bg', meta_value,'')) AS 'bg' , MAX(IF( `meta_key`='avatar', meta_value,''NULL'')) AS 'avatar' , MAX(IF (`meta_key`='profile_updated', meta_value,'')) AS 'profile_updated' FROM `ff_usermeta` WHERE `user_id`= %s GROUP BY umeta_id, user_id:""")
result = cursor.execute(get_meta,(user_id,))
if result > 0:
data = cursor.fetchone()
for_update = "";
#some processing of the result
if not data["nickname"]:
for_update += "`nickname`='"+data["nickname"]+"', "
if not data["bg"]:
for_update += "`bg`='"+data["bg"]+"', "
query = ("UPDATE `ff_usermeta` SET "+for_update+" WHERE user_id = " + user_id)
else:
#here are no data to be gathered as there is no user_id present add new user
nickname = ""
bg= ""
info = ""
avatar = ""
profile_updated = ""
fields= ""
vals = ""
fields += "`nickname`,`info`, `bg`,`avatar`,`profile_updated`"
vals += "'"+nickname+"', "+"'"+info+"', "+"'"+bg+"', "+"'"+avatar+"', "+"'"+profile_updatedfo+"'"
query = ("INSERT INTO `ff_usermeta`("+fields+") VALUES ("+vals+")")
cursor.execute(query)
cnx.commit()
cursor.close()
cnx.close()
I tried my best to adapt the suggestion above, but couldn't figure out how to make it work. Eventually I went another way, and it seems to work somehow, so I'll post the full code in case anyone would find it useful.
What it does: checks the queue in table with validation request, then parses a page (separate function) and updates user profile accodringly.
import mysql.connector as mysql
import time
from datetime import datetime
cnx = mysql.connect(host=HOST, database=DATABASE, user=USER, password=PASSWORD)
while True: #endless loop as a temporary scheduler
cursor = cnx.cursor()
#getting first request in the queue - 0: id, 1: url, 2: parse, 3: status, 4: user, 5: user_page, 6: req_date, 7: action
cursor.execute("SELECT * FROM `ff_qq` WHERE status = 0 LIMIT 1")
row = cursor.fetchone()
if row:
status = 1 #processed
if row[7] == "verify":
get_user = ("SELECT * FROM `ff_users` WHERE ID = %s LIMIT 1")
cursor.execute(get_user, (row[4],))
user = cursor.fetchone() #0 - ID, 5 - user_url, 8 - user_status, 9 - display_name
#separate function that returns data to insert into mysql
udata = verify(row) #0 - nickname, 1 - fb_av, 2 - fb_bg, 3 - fb_info, 4 - owner
ustat = row[1].split("/authors/")
if udata['owned'] or user[8] == ustat[1]:
update_user = ("UPDATE `ff_users` SET user_status = %s, display_name = %s, user_url = %s WHERE ID = %s LIMIT 1")
cursor.execute(update_user, (ustat[1], udata['nickname'], row[1], user[0]))
status = 2 #success
get = ("SELECT `meta_value` FROM `ff_usermeta` WHERE `user_id`= %s AND `meta_key`='ff_capabilities' LIMIT 1")
cursor.execute(get, (row[4],))
rights = cursor.fetchone()
if rights == 'a:1:{s:10:"subscriber";b:1;}':
promote = ("UPDATE `ff_usermeta` SET `meta_value` = 'a:1:{s:6:\"author\";b:1;}' "
"WHERE `user_id` = %s AND `meta_key`='ff_capabilities' LIMIT 1")
cursor.execute(promote, (row[0],))
#list of meta_key values in same order as returned data
ff = ['nickname', 'fb_av', 'fb_bg', 'fb_info']
for x in range(0,3): #goes through each one of the above list
if udata[ff[x]]: #yes this actually works, who would've thought?..
#current meta_key added directly into the string
get = ("SELECT `meta_value` FROM `ff_usermeta` WHERE `user_id`= %s AND `meta_key`='" + ff[x] + "' LIMIT 1")
cursor.execute(get, (row[4],))
meta = cursor.fetchone()
if(meta): #update if it exists, otherwise insert new row
qq = ("UPDATE `ff_usermeta` SET `meta_value` = %s "
"WHERE `user_id` = %s AND `meta_key`='" + ff[x] + "' LIMIT 1")
else:
qq = ("INSERT INTO `ff_usermeta`(`meta_value`, `meta_key`, `user_id`) "
"VALUES ('%s','" + ff[x] + "','%s'")
cursor.execute(qq, (udata[ff[x]], row[0])) #same execute works for both
else:
status = 3 #verification failed
#update queue to reflect its status
update = ("UPDATE `ff_qq` SET status = %s WHERE id = %s LIMIT 1")
cursor.execute(update, (status, row[0]))
cnx.commit()
cursor.close()
now = datetime.now()
print(now.strftime("%d.%m.%Y %H:%M:%S"))
time.sleep(180) #sleep until it's time to re-check the queue
cnx.close()
In an effort to learn SQL I've been using it at work. I keep a list of every account I work on and the information I gather while working on that particular account. My database has two tables that I update. One is the customer's information and it always gets updated with each new account and the other is an institution table that gets updated sometimes but not every time. These two tables are relational.
As I said, I am new to SQL and have been using the command line to update these tables and it's getting annoying. What I thought would be an easy solution is to run a series of Python prompts that queries the user for each column and then executes the INSERT command at the end of the prompt. It would then ask if I wanted to create an entry for the second table (since I don't always add to this one).
These are my tables.
> create table stu_info (
> id Integer,
> name text,
> CEEB integer,
> degree text,
> terms integer,
> comm text
> );
> create table coll_info (
> CEEB integer,
> name text,
> Accred text,
> Hours text,
> comm text
> );
In Python I figure it'd be easy to just use raw_input() and add an int() around it when required. Then use a loop so that after adding each new row to the database it starts over until I'm done for the day. My problem is I cannot figure out how to execute sqlite commands through Python and I can't figure out how to access the database.
Whenever I run
import sqlite3
conn = sqlite3.connect(stu.db)
c = conn.cursor()
enter code here
I get a NameError: name 'stu' is not defined
Any help? I imagine this is easy and my Google-fu is bad and inexperience has led even good search results to being useless...
It looks like it's as simple as needing quotes.
conn = sqlite3.connect("stu.db")
The error you're getting is because there is no variable stu in scope. So long as "stu.db" is in the working directory then the snippet above should work.
Once I found out, thanks to Jamie, that the filename needs to be in quotes I was able to connect to the database. I then was able to work my way through the problem. Here is my solution:
import sqlite3
conn = sqlite3.connect('stu.db')
c = conn.cursor()
def add_coll():
cceeb = int(raw_input("CEEB:> "))
cname = raw_input("Name:> ")
ccred = raw_input("Accred:> ")
ccal = raw_input("Calendar:> ")
ccomm = raw_input("Comments:> ")
c.execute("INSERT INTO coll VALUES (%d, '%s', '%s', '%s', '%s')" %
(cceeb, cname, ccred, ccal, ccomm))
var = 1 #This creates an infinite loop
while var == 1:
input_stu = raw_input("Add Student?:> ")
if input_stu == 'no' or input_stu == 'n':
add_coll()
else:
id = int(raw_input("ID:> "))
name = raw_input("Name:> ")
ceeb = int(raw_input("CEEB:> "))
deg = raw_input("Degree:> ")
terms = int(raw_input("Terms:> "))
comm = raw_input("Comments:> ")
c.execute("INSERT INTO stu VALUES (%d, '%s', %d, '%s', %d, '%s')" %
(id, name, ceeb, deg, terms, comm))
input_coll = raw_input("Add College?:> ")
if input_coll == 'yes' or input_coll == 'y':
#cceeb = int(raw_input("CEEB:> "))
#cname = raw_input("Name:> ")
#ccred = raw_input("Accred:> ")
#ccal = raw_input("Calendar:> ")
#
#c.execute("INSERT INTO coll VALUES (%d, '%s', '%s', '%s')" %
# (cceeb, cname, ccred, ccal))
add_coll()
conn.commit()
The following works when typed directly in the mongodb shell—I receive the correct output:
db.serial_key.find({key: {$ne : "5SNT0V"}})
However, in Python 3, it's not working. Every time, only the if block runs. It does not matter if I use str or not in the query.
for x in keys:
i +=1;
print('key %d = ' %i , keys[i-1]) #out put: key 1 = 3ZOHSH
# place values in dictionary
keyrecord_record = {'key':keys[i-1],'b_p_code':x1}
if(db.serial_key.find({'key':{ '$ne':str(keys[i-1])}})):
db.insert(keyrecord_record)
else:
print('Record in DB')
Please, I expect some expert help. I'm trying to do this within one day. I only want to write values that are not already in the database.
I found this question:
Mongo db not equal to query not working
...but it does not answer my question.
===================full main class code==========================
from pymongo import MongoClient
from algo import algo
#take in put data
x1 = int(input("Enter a number(Brand_name+Pack_size) : "))
y1 = int(input("Enter a number: key Quntity "))
#create connection
client = MongoClient()
#create emty list
alist = []
#make database_table
db = client.product.serial_key
keyrecord_record = {}
#find table existing entry that code
f_cursor = db.find({"b_p_code": x1})
for document in f_cursor:
alist.append(document['key'])
#print(document['key']) //print expected result
if(x1 == "" or y1==""):
print("please enter valid no")
else:
x = algo(x1,y1,alist)
keys =x.id_generator()
keys.sort(reverse=True)
print('\n')
i=0;
for x in keys:
i +=1;
print('key %d = ' %i , keys[i-1])
# place values in dictionary
keyrecord_record = {'key':keys[i-1],'b_p_code':x1}
#not recived expected result. if key in database again print key
if(db.serial_key.find({'key':{ '$ne':str(keys[i-1])}})):
db.insert(keyrecord_record)
else:
print('Record in DB')
print("\n")
print("Batch No: come from db ")
print('Generate Key beetween %d00000 - %d00000' % (x1 ,(x1+1)) )
print("Startin Serial : " , keys[0])
print("Ending Serial : " , keys[len(keys)-1])
print('\n Database Details \n')
#print details
cursor = db.find()
for document in cursor:
print(document['key'])
#print(document)
Image showing console out put.
From mongodb docs:
$ne operator selects the documents where the value of the field is not equal (i.e. !=) to the specified value. This includes documents that do not contain the field.
db.serial_key.find({'key':{ '$ne':str(keys[i-1])}}) will return a mongo cursor. Thus your if condition will always be true resulting in document addition regardless of the presence of the same value.
If you want to verify that the key already exists you can query the value and check the mongo cursor count.
The following code should do the trick.
for x in keys:
i +=1;
print('key %d = ' %i , keys[i-1]) #out put: key 1 = 3ZOHSH
# place values in dictionary
keyrecord_record = {'key':keys[i-1],'b_p_code':x1}
#check if already exists
if(db.find({'key':str(keys[i-1])}).limit(1).count() == 0):
db.insert(keyrecord_record)
else:
print('Record in DB')