I'm trying to automate a large database process that is run at the beginning of every month. This is done, currently, with a bunch of stored procedures that are already in the database. My task is currently to get these stored procedures to run via Python's pyodbc.
The tricky part about these stored procedures is I need to run them one-at-a-time so they don't hog all of the database's resources. So, to check to see if the stored procedures have been run I created a table that is updated, changing a flag called "IsFinished" to False at the beginning of the procedure to True at the end.
The problem occurs when I try to query the database again. The activity monitor in SQL server is showing that my query against the Tracking table as SUSPENDED and blocked by the stored procedure call.
I've even gone so far as to create a new database so that I can call both of the databases independently just to further make sure I'm not competing for resources... but I still am competing for resources.
I'm using Python 3.6.4 and the pyodbc module. Here's my code. It should be noted that connect_db_1() and connect_db_2() point to two separate databases.
conn = connect_db_1()
conn.execute("{call dbo.StoredProcedure}")
conn2 = connect_db_2()
start_time = time.time()
finished = False
while finished is False:
print("Looping")
time.sleep(120)
cursor = conn2.execute(
"SELECT IsFinished from dbo.Tracking where StoredProcedureName='dbo.StoredProcedure'")
results = cursor.fetchall()
if results[0][0] == True:
finished = True
print(results)
print(f"dbo.StoredProcedure finished in {time_diff(start_time)} seconds.")
cursor.close()
EDIT: Added a sanitized version of my code.
You want to close your cursor inside the loop since you are creating new cursors in the loop.
Basically, you were opening a cursor over and over again without closing.
while finished is False:
print("Looping")
time.sleep(120)
cursor = conn2.execute(
"SELECT IsFinished from dbo.Tracking where StoredProcedureName='dbo.StoredProcedure'")
results = cursor.fetchall()
if results[0][0] == True:
finished = True
print(results)
cursor.close()
print(f"dbo.StoredProcedure finished in {time_diff(start_time)} seconds.")
Related
While creating a new program to handle a database I encountered a problem.
The mysql cursor doesn't seem to notice the WHERE statement in the following function:
def get_new_entries(self, delay):
start_time = t.time()
while True:
cursor = self.cnx.cursor()
check_delay = delay * 2
current_time = datetime.datetime.now() - datetime.timedelta(seconds=check_delay)
current_time = current_time.strftime("%Y-%m-%d %H:%M:%S")
data = current_time
print(current_time)
query = """SELECT FILE_NAME, CREATION_TIME FROM `added_files` WHERE CREATION_TIME < %s"""
cursor.execute(query, (data,))
print(query)
for (FILE_NAME, CREATION_TIME) in cursor:
print(FILE_NAME)
print(CREATION_TIME)
cursor.close()
cursor = None
print("###################")
t.sleep(delay - ((t.time() - start_time) % delay))
With this function I wish to achieve that every minute the function checks for new entries in the past minute. In the end I want it to send the filenames as a list to another class, so that class can use logic to handle the filenames.
However, the WHERE CREATION_TIME < '%s' doesnt seem to do anything. It either doesn't return any entry, while trying the same query in the mysql environment itself does what it should. If however the '<' is changed to '>' it suddenly returns all items, even those which should NOT be returned.
With that said, I have also used this part of code with only
cursor.execute(query)
while the query was changed to
query = "SELECT FILE_NAME, CREATION_TIME FROMadded_filesWHERE CREATION_TIME < {}".format(current_time).
This worked the first time, but the second time in the loop it didn't respond anything, even though I did add stuff to the database. I used the same datetime the program used in the mysql environment in the browser, which returned the correct results, but the program didn't.
So why doesn't it work? And what should I do to make it work?
So after a while I solved the problem. It had nothing to do with the code I sent but had to do with the lack of autocommit = True of the MYSQL connection. I'll try and explain.
My application had to check a database that is automatically updated by another (C#) application. For every new file the C# app found it created a new entry in the database. Meanwhile this Python application checks for new filenames every delay (e.g. 60.0 seconds).
The Python application opens a connection via an mysql.connector and seems to keep hold on the state of the database at that exact moment. Anything added to it will not be found by any code you post, because it doesn't search in the actual database, it searches in it's own saved version of the database.
The fix would be to set the connection to autocommit. So you would do this:
self.cnx = mysql.connector.connect(user='root', password='', host='127.0.0.1', database='mydb')
self.cnx.autocommit = True
This will update the saved state of the database in the python app every time you execute an sql query.
So you don't get what you expect from you sql query in python, go try setting autocommit to true for your connection.
Situation: I have a live trading script which computes all sorts of stuff every x minutes in my main thread (Python). the order sending is performed through such thread. the reception and execution of such orders though is a different matter as I cannot allow x minutes to pass but I need them as soon as they come in. I initialized another thread to check for such data (execution) which is in a database table (POSTGRES SQL).
Problem(s): I cannot continuosly perform query every xx ms, get data from DB, compare table length, and then get the difference for a variety of reasons (not only guy to use such DB, perforamnce issues, etc). so I looked up some solutions and came up with this thread (https://dba.stackexchange.com/questions/58214/getting-last-modification-date-of-a-postgresql-database-table) where basically the gist of it was that
"There is no reliable, authorative record of the last modified time of a table".
Question: what can I do about it, that is: getting near instantenuous responses from a postgres sql table without overloading the whole thing using Python?
You can use notifications in postgresql:
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
import select
def dblisten(dsn):
connection = psycopg2.connect(dsn)
connection.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
cur = connection.cursor()
cur.execute("LISTEN new_id;")
while True:
select.select([connection],[],[])
connection.poll()
events = []
while connection.notifies:
notify = connection.notifies.pop().payload
do_something(notify)
and install a trigger for each update:
CREATE OR REPLACE FUNCTION notify_id_trigger() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('new_id', NEW.ID);
RETURN new;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER data_modified AFTER insert or update on data_table for each row execute procedure notify_id_trigger();")
I have a Python script using socket and threads to allow 10 servers to connect to a port. Each server dumps a string of data. Sometimes the data comes in rapidly, and other times it trickles in.
The Python script takes the data blob, does a substring count to get "column" values, then sends it to MSSQL using pymssql. Pretty straight forward.
Here's a snippet of the MSSQL portion of the script:
dbConn = pymssql.connect(server=mssql_server, user=mssql_user, password=mssql_pass, database=mssql_db)
cursor = dbConn.cursor()
date = data[0:6]
time = data[7:11]
duration = data[12:16]
mssql_output_raw = "('%s','%s','%s');" % (date, time, duration)
mssql_output = mssql_output_raw.replace(" ", "") # Remove any whitespace
# Write to MSSQL table
try:
query = "INSERT INTO %s VALUES %s" % (mssql_table, mssql_output)
cursor.execute( query )
dbConn.commit()
except pymssql.OperationalError as e:
logmsg("pymssql.OperationalError exception caught: %s" % str(e).replace("\n", " ") )
except:
pass
Every so often (and maybe when the data is rapidly coming in?) I'll get this exception:
20019,
'DB-Lib error message 20019, severity 7:
Attempt to initiate a new Adaptive Server operation with results pending
The script doesn't crash, and since the script is either a) running in the background; or b) in the foreground but spewing data, I'm not sure if the data ever makes it to MSSQL.
Can anyone share what this error means?
I may have figured this out. In my script I'm using threading. There was 1 SQL connector for all the threads. My guess is that the one SQL connector was getting overwhelmed with all the queries.
I've updated my script so that each thread has its own connector. So far so good.
I have one script running on a server that updates a list of items in a MySQL database to be processed by another script running on my desktop. The script runs in a loop, processing the list every 5 minutes (the server side script also runs on a 5 minute cycle). On the first loop, the script retrieves the current list (basic SELECT operation), on the second cycle, it gets the same version (not updated) list, on the third, it gets the list it should have gotten on the second pass. On every pass after the first, the SELECT operation returns the data from the previous UPDATE operation.
def mainFlow():
activeList=[]
d=()
a=()
b=()
#cycleStart=datetime.datetime.now()
cur = DBSV.cursor(buffered=True)
cur.execute("SELECT list FROM active_list WHERE id=1")
d=cur.fetchone()
DBSV.commit()
a=d[0]
b=a[0]
activeList=ast.literal_eval(a)
print(activeList)
buyList=[]
clearOrders()
sellDecide()
if activeList:
for i in activeList:
a=buyCalculate(i)
if a:
buyList.append(i)
print ('buy list: ',buyList)
if buyList:
buyDecide(buyList)
cur.close()
d=()
a=()
b=()
activeList=[]
print ('+++++++++++++END OF BLOCK+++++++++++++++')
state=True
while state==True:
cycleStart=datetime.datetime.now()
mainFlow()
cycleEnd=datetime.datetime.now()
wait=300-(cycleEnd-cycleStart).total_seconds()
print ('wait=: ' +str(wait))
if wait>0:
time.sleep(wait)
As you can see, I am re initializing all my variables, I am closing my cursor, I am doing a commit() operation that is supposed to solve this sort of problem, I have tried plain cursors, and cursors with the buffer set True and False, always with the same result.
When I run the exact same Select query from MySQL Workbench, the results returned are fine.
Baffled, and stuck on this for 2 days.
You're performing your COMMIT before your UPDATE/INSERT/DELETE transactions
Though a SELECT statement is, theoretically, DML it has certain differences with INSERT, UPDATE and DELETE in that it doesn't modify the data within the database. If you want to see the data that has been changed within another session then you must COMMIT it only after it's been changed. This is partially exacerbated by you closing the cursor after each loop.
You've gone far too far in trying to solve this problem; there's no need to reset everything within the mainFlow() method (and I can't see a need for most of the variables)
def mainFlow():
buyList = []
cur = DBSV.cursor(buffered=True)
cur.execute("SELECT list FROM active_list WHERE id = 1")
activeList = cur.fetchone()[0]
activeList = ast.literal_eval(activeList)
clearOrders()
sellDecide()
for i in activeList:
a = buyCalculate(i)
if a:
buyList.append(i)
if buyList:
buyDecide(buyList)
DBSV.commit()
cur.close()
while True:
cycleStart = datetime.datetime.now()
mainFlow()
cycleEnd = datetime.datetime.now()
wait = 300 - (cycleEnd - cycleStart).total_seconds()
if wait > 0:
time.sleep(wait)
I've removed a fair amount of unnecessary code (and added spaces), I've removed the reuse of variable names for different things and the declaration of variables that are overwritten immediately. This still isn't very OO though...
As we don't have detailed knowledge of exactly what clearOrders(), sellDecide() and buyCalculate() you might want to double check this yourself.
I made a loop in Python that calls itself to repeatedly check for new entries in a database. On first execution, all affected rows are shown fine. Meanwhile, I add more rows into the database. On the next query in my loop, the new rows are not shown.
This is my query-loop:
def loop():
global mysqlconfig # username, passwd...
tbd=[] # this is where I save the result
conn = MySQLdb.connect(**mysqlconfig)
conn.autocommit(True)
c = conn.cursor()
c.execute("SELECT id, message FROM tasks WHERE date <= '%s' AND done = 0;" % now.isoformat(' '))
conn.commit()
tbd = c.fetchall()
print tbd
c.close()
conn.close()
time.sleep(5)
loop()
loop()
This is the SQL part of my Python insertion-script:
conn = MySQLdb.connect(**mysqlconfig)
conn.autocommit(1)
c = conn.cursor()
c.execute("INSERT INTO tasks (date, message) VALUES ('{0}', '{1}');".format("2012-10-28 23:50", "test"))
conn.commit()
id = c.lastrowid
c.close()
conn.close()
I tried SQLite, I tried Oracle MySQL's connector, I tried MySQLdb on a Windows and Linux system and all had the same problem. I looked through many, many threads on Stackoverflow that recommended to turn on autocommit or use commit() after an SQL statement (ex. one, two, three), which I tried and failed.
When I added data with HeidiSQL to my database it showed up in the loop query, but I don't really know why this is. Rows inserted with mysql-client on Linux and my Python insertion script never show up until I restart my loop script.
I don't know if it's the fact that I open 2 connections, each in their own script, but I close every connection and every cursor when I'm done with them.
The problem could be with your variable now. I don't see anywhere in the loop that it is being reset.
I'd probably use the mysql NOW() function:
c.execute("SELECT id, message FROM tasks WHERE date <= NOW() AND done = 0;")
It looks like the time you are inserting into the database is a time in the future. I don't think your issue is with your database connection, I think it's something to do with the queries you are doing.