While creating a new program to handle a database I encountered a problem.
The mysql cursor doesn't seem to notice the WHERE statement in the following function:
def get_new_entries(self, delay):
start_time = t.time()
while True:
cursor = self.cnx.cursor()
check_delay = delay * 2
current_time = datetime.datetime.now() - datetime.timedelta(seconds=check_delay)
current_time = current_time.strftime("%Y-%m-%d %H:%M:%S")
data = current_time
print(current_time)
query = """SELECT FILE_NAME, CREATION_TIME FROM `added_files` WHERE CREATION_TIME < %s"""
cursor.execute(query, (data,))
print(query)
for (FILE_NAME, CREATION_TIME) in cursor:
print(FILE_NAME)
print(CREATION_TIME)
cursor.close()
cursor = None
print("###################")
t.sleep(delay - ((t.time() - start_time) % delay))
With this function I wish to achieve that every minute the function checks for new entries in the past minute. In the end I want it to send the filenames as a list to another class, so that class can use logic to handle the filenames.
However, the WHERE CREATION_TIME < '%s' doesnt seem to do anything. It either doesn't return any entry, while trying the same query in the mysql environment itself does what it should. If however the '<' is changed to '>' it suddenly returns all items, even those which should NOT be returned.
With that said, I have also used this part of code with only
cursor.execute(query)
while the query was changed to
query = "SELECT FILE_NAME, CREATION_TIME FROMadded_filesWHERE CREATION_TIME < {}".format(current_time).
This worked the first time, but the second time in the loop it didn't respond anything, even though I did add stuff to the database. I used the same datetime the program used in the mysql environment in the browser, which returned the correct results, but the program didn't.
So why doesn't it work? And what should I do to make it work?
So after a while I solved the problem. It had nothing to do with the code I sent but had to do with the lack of autocommit = True of the MYSQL connection. I'll try and explain.
My application had to check a database that is automatically updated by another (C#) application. For every new file the C# app found it created a new entry in the database. Meanwhile this Python application checks for new filenames every delay (e.g. 60.0 seconds).
The Python application opens a connection via an mysql.connector and seems to keep hold on the state of the database at that exact moment. Anything added to it will not be found by any code you post, because it doesn't search in the actual database, it searches in it's own saved version of the database.
The fix would be to set the connection to autocommit. So you would do this:
self.cnx = mysql.connector.connect(user='root', password='', host='127.0.0.1', database='mydb')
self.cnx.autocommit = True
This will update the saved state of the database in the python app every time you execute an sql query.
So you don't get what you expect from you sql query in python, go try setting autocommit to true for your connection.
Related
I have a multi-file Python Project, of which many of the files make connections to an Azure SQL Database. The project works fine but, for some reason, one of the files stops being able to connect to the database after a while of the application running, and I can see no reason as to why; especially when other connection attempts work fine.
The connection string, for all the connections (so across all the files), is define as the following:
SQLServer = os.getenv('SQL_SERVER')
SQLDatabase = os.getenv('SQL_DATABASE')
SQLLogin = os.getenv('SQL_LOGIN')
SQLPassword = os.getenv('SQL_PASSWORD')
SQLConnString = 'Driver={ODBC Driver 17 for SQL Server};Server=' + SQLServer + ';Database=' + SQLDatabase + ';UID='+ SQLLogin +';PWD=' + SQLPassword
sqlConn = pyodbc.connect(SQLConnString,timeout=20)
And the function I am calling, when the error happens is below:
def iscaptain(guild,user):
userRoles = user.roles
roleParam = ""
for role in userRoles:
roleParam = roleParam + "," + str(role.id)
cursor = sqlConn.cursor()
roleParam = roleParam[1:]
cursor.execute('EXEC corgi.GetUserAccess ?, ?;',guild.id,roleParam)
for row in cursor:
if row[1] == "Team Captain":
cursor.close()
return True
cursor.close()
return False
The error specifically happens at cursor.execute. I currently get the error
pyodbc.OperationalError: ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x68 (104) (SQLExecDirectW)')
Previously I didn't have the timeout in the connection on the specific file that was having a problem, and I did get a different error:
Communication link failure
Apologies, I don't have the full previous error.
Other connections, in other files in the same project, work fine, so the problem is not a network issue; if it were none of the connections would work. The problem only happens in one file, where all the connection attempts fail.
Googling the latest error really doesn't get me far. For example, there's a Github issue that gets nowhere, and this question isn't related as connecting works fie from other files.
Note, as well, that this happens after a period of time; I don't really know how long that period is but it's certainly hours. Restarting the project fixes the issue as well; the above function will work fine. That isn't really a solution though, I can't keep restarting the application ad-hoc.
The error is immediate as well; it's like Python/PyODBC isn't trying to connect. When stepping into the cursor.execute the error is generated straight after; it's not like when you get a timeout and you'll be waiting a few seconds, or more, for the timeout to occur.
I'm at a loss here. Why is the file (and only that one) unable to connect any more later on? There are no locks on the database either, so It's not like I have a transaction left hanging; though I would expect a timeout error again then as the procedure would be unable to gain a lock on the data.
Note, as well, that if I manually execute the procedure, in sqlcmd/SSMS/ADS, data is returned fine as well, so the Procedure does work fine. And, again, if I restart the application it'll work without issue for many hours.
Edit: I attempted the answer from Sabik below, however, this only broke to application, unfortunately. The solution they provided had the parameter self on the function validate_conn and so calling validate_conn() failed as I don't have a parameter for this "self". The method they said to use, just validate_conn didn't do anything; it doesn't call the function (which I expected). Removing the parameter, and references to self, also broke the application, stating that sqlConn wasn't declared even though it was; see image below where you can clearly see that sqlConn has a value:
Yet immediately after that line I get the error below:
UnboundLocalError: local variable 'sqlConn' referenced before assignment
So something appears to be wrong with their code, but I don't know what.
One possibility being discussed in the comments is that it's a 30-minute idle timeout on the database end, in which case one solution would be to record the time the connection has been opened, then reconnect if it's been more than 25 minutes.
This would be a method like:
def validate_conn(self):
if self.sqlConn is None or datetime.datetime.now() > self.conn_expiry:
try:
self.sqlConn.close()
except: # pylint: disable=broad-except
# suppress all exceptions; we're in any case about to reconnect,
# which will either resolve the situation or raise its own error
pass
self.sqlConn = pyodbc.connect(...)
self.conn_expiry = datetime.datetime.now() + datetime.timedelta(minutes=25)
(Adjust as appropriate if sqlConn is a global.)
At the beginning of each function which uses sqlConn, call validate_conn first, then use the connection freely.
Note: this is one of the rare situations in which suppressing all exceptions is reasonable; we're in any case about to reconnect to the database, which will either resolve the situation satisfactorily, or raise its own error.
Edit: If sqlConn is a global, it will need to be declared as such in the function:
def validate_conn():
global sqlConn, conn_expiry
if sqlConn is None or datetime.datetime.now() > conn_expiry:
try:
sqlConn.close()
except: # pylint: disable=broad-except
# suppress all exceptions; we're in any case about to reconnect,
# which will either resolve the situation or raise its own error
pass
sqlConn = pyodbc.connect(...)
conn_expiry = datetime.datetime.now() + datetime.timedelta(minutes=25)
As an unrelated style note, a shorter way to write the function would be using (a) a with statement and (b) the any operator, like this:
with sqlConn.cursor() as cursor:
roleParam = roleParam[1:]
cursor.execute('EXEC corgi.GetUserAccess ?, ?;', guild.id, roleParam)
return any(row[1] == "Team Captain" for row in cursor)
The with statement has the advantage that the cursor is guaranteed to be closed regardless of how the code is exited, even if there's an unexpected exception or if a later modification adds more branches.
Although the solution for Sabik didn't work for me, the answer did push me in the right direction to find a solution. That was, specifically, with the use of the with clauses.
Instead of having a long lasting connection, as I have been informed I had, I've now changed these to short lived connections with I open with a with, and then also changed the cursor to a with as well. So, for the iscaptain function, I now have code that looks like this:
def iscaptain(guild,user):
userRoles = user.roles
roleParam = ""
for role in userRoles:
roleParam = roleParam + "," + str(role.id)
#sqlConn = pyodbc.connect(SQLConnString,timeout=20)
with pyodbc.connect(SQLConnString,timeout=20) as sqlConn:
with sqlConn.cursor() as cursor:
roleParam = roleParam[1:]
cursor.execute('EXEC corgi.GetUserAccess ?, ?;', guild.id, roleParam)
return any(row[1] == "Team Captain" for row in cursor)
return False
It did appear that Azure was closing the connections after a period of time, and thus when the connection was attempted to be reused it failed to connect. As, however, both hosts are in Azure, but the Server running the Python application and the SQL Database, I am happy to reconnect as needed, as speed should not be a massive issue; certainly it hasn't been during the last 48 hours of testing.
This does mean i have a lot of code to refactor, but for the stability, it's a must.
I'm trying to automate a large database process that is run at the beginning of every month. This is done, currently, with a bunch of stored procedures that are already in the database. My task is currently to get these stored procedures to run via Python's pyodbc.
The tricky part about these stored procedures is I need to run them one-at-a-time so they don't hog all of the database's resources. So, to check to see if the stored procedures have been run I created a table that is updated, changing a flag called "IsFinished" to False at the beginning of the procedure to True at the end.
The problem occurs when I try to query the database again. The activity monitor in SQL server is showing that my query against the Tracking table as SUSPENDED and blocked by the stored procedure call.
I've even gone so far as to create a new database so that I can call both of the databases independently just to further make sure I'm not competing for resources... but I still am competing for resources.
I'm using Python 3.6.4 and the pyodbc module. Here's my code. It should be noted that connect_db_1() and connect_db_2() point to two separate databases.
conn = connect_db_1()
conn.execute("{call dbo.StoredProcedure}")
conn2 = connect_db_2()
start_time = time.time()
finished = False
while finished is False:
print("Looping")
time.sleep(120)
cursor = conn2.execute(
"SELECT IsFinished from dbo.Tracking where StoredProcedureName='dbo.StoredProcedure'")
results = cursor.fetchall()
if results[0][0] == True:
finished = True
print(results)
print(f"dbo.StoredProcedure finished in {time_diff(start_time)} seconds.")
cursor.close()
EDIT: Added a sanitized version of my code.
You want to close your cursor inside the loop since you are creating new cursors in the loop.
Basically, you were opening a cursor over and over again without closing.
while finished is False:
print("Looping")
time.sleep(120)
cursor = conn2.execute(
"SELECT IsFinished from dbo.Tracking where StoredProcedureName='dbo.StoredProcedure'")
results = cursor.fetchall()
if results[0][0] == True:
finished = True
print(results)
cursor.close()
print(f"dbo.StoredProcedure finished in {time_diff(start_time)} seconds.")
I have a Python script using socket and threads to allow 10 servers to connect to a port. Each server dumps a string of data. Sometimes the data comes in rapidly, and other times it trickles in.
The Python script takes the data blob, does a substring count to get "column" values, then sends it to MSSQL using pymssql. Pretty straight forward.
Here's a snippet of the MSSQL portion of the script:
dbConn = pymssql.connect(server=mssql_server, user=mssql_user, password=mssql_pass, database=mssql_db)
cursor = dbConn.cursor()
date = data[0:6]
time = data[7:11]
duration = data[12:16]
mssql_output_raw = "('%s','%s','%s');" % (date, time, duration)
mssql_output = mssql_output_raw.replace(" ", "") # Remove any whitespace
# Write to MSSQL table
try:
query = "INSERT INTO %s VALUES %s" % (mssql_table, mssql_output)
cursor.execute( query )
dbConn.commit()
except pymssql.OperationalError as e:
logmsg("pymssql.OperationalError exception caught: %s" % str(e).replace("\n", " ") )
except:
pass
Every so often (and maybe when the data is rapidly coming in?) I'll get this exception:
20019,
'DB-Lib error message 20019, severity 7:
Attempt to initiate a new Adaptive Server operation with results pending
The script doesn't crash, and since the script is either a) running in the background; or b) in the foreground but spewing data, I'm not sure if the data ever makes it to MSSQL.
Can anyone share what this error means?
I may have figured this out. In my script I'm using threading. There was 1 SQL connector for all the threads. My guess is that the one SQL connector was getting overwhelmed with all the queries.
I've updated my script so that each thread has its own connector. So far so good.
So I'm using psycopg2, I have a simple table:
CREATE TABLE IF NOT EXISTS feed_cache (
feed_id int REFERENCES feeds(id) UNIQUE,
feed_cache text NOT NULL,
expire_date timestamp --without time zone
);
I'm calling the following method and query:
#staticmethod
def get_feed_cache(conn, feed_id):
c = conn.cursor()
try:
sql = 'SELECT feed_cache FROM feed_cache WHERE feed_id=%s AND localtimestamp <= expire_date;'
c.execute(sql, (feed_id,))
result = c.fetchone()
if result:
conn.commit()
return result[0]
else:
print 'DBSELECT.get_feed_cache: %s' % result
print 'sql: %s' % (c.mogrify(sql, (feed_id,)))
except:
conn.rollback()
raise
finally:
c.close()
return None
I've added the else statement to output the exact sql and result that is being executed and returned.
The get_feed_cache() method is called from a database connection thread pool. When the get_feed_cache() method is called "slowishly" (~1/sec or less) the result is returned as expected, however when called concurrently it will occasionally return None. I have tried multiple ways of writing this query & method.
Some observations:
If I remove 'AND localtimestamp <= expire_date' from the query, the query ALWAYS returns a result.
Executing the query rapidly in serial in psql always returns a result.
After reading about the fetch*() methods of psycopg's cursor class they note that the results are cached for the cursor, I'm assuming that the cache is not shared between different cursors. http://initd.org/psycopg/docs/faq.html#best-practices
I have tried using postgresql's now() and current_timestamp functions with the same results. (I am aware of the timezone aspect of now() & current_timestamp)
Conditions to note:
There will NEVER be a case where there is not a feed_cache value for a provided feed_id.
There will NEVER be a case where any value in the feed_cache table is NULL
While testing I have completely disabled any & all writes to this table
I have set the expire_date to be sufficiently far in the future for all values such that the expression 'AND localtimestamp <= expire_date' will always be true.
Here is a copy & pasted output of it returning None:
DBSELECT.get_feed_cache: None
sql: SELECT feed_cache FROM feed_cache WHERE feed_id=5 AND localtimestamp < expire_date;
Well that's pretty much it, I'm not sure what's going on. Maybe I'm making some really dumb mistake and I just don't notice it! My current guess is that it has something to do with psycopg2 and perhaps the way it's caching results between cursors. If the cursors DO share the cache and the queries happen near-simultaneously then it could be possible that the first cursor fetches the result, the second cursor sees there is a cache of the same query, so it does not execute, then the first cursor closes and deletes the cache and the second cursor tries to fetch a now null/None cache.*
That said, psycopg2 states that it's thread-safe for read-only queries, so unless I'm miss-interpreting their implementation of thread-safe, this shouldn't be the case.
Thank you for your time!
*After adding a thread lock for the get_feed_cache, acquiring before creating the cursor and releasing before returning, I still occasionally get a None result
I think this might have to do with the fact that the time stamps returned by localtimestamp or current_timestamp are fixed when the transaction starts, not when you run the statement. And psycopg manages the transactions behind your back to some degree. So you might be getting a slightly older time stamp.
You could debug this by setting log_statement = all in your server and then observing when the BEGIN statements are executed relative to your queries.
You might want to look into using a function such as clock_timestamp(), which updates more often per transaction. See http://www.postgresql.org/docs/current/static/functions-datetime.html.
I made a loop in Python that calls itself to repeatedly check for new entries in a database. On first execution, all affected rows are shown fine. Meanwhile, I add more rows into the database. On the next query in my loop, the new rows are not shown.
This is my query-loop:
def loop():
global mysqlconfig # username, passwd...
tbd=[] # this is where I save the result
conn = MySQLdb.connect(**mysqlconfig)
conn.autocommit(True)
c = conn.cursor()
c.execute("SELECT id, message FROM tasks WHERE date <= '%s' AND done = 0;" % now.isoformat(' '))
conn.commit()
tbd = c.fetchall()
print tbd
c.close()
conn.close()
time.sleep(5)
loop()
loop()
This is the SQL part of my Python insertion-script:
conn = MySQLdb.connect(**mysqlconfig)
conn.autocommit(1)
c = conn.cursor()
c.execute("INSERT INTO tasks (date, message) VALUES ('{0}', '{1}');".format("2012-10-28 23:50", "test"))
conn.commit()
id = c.lastrowid
c.close()
conn.close()
I tried SQLite, I tried Oracle MySQL's connector, I tried MySQLdb on a Windows and Linux system and all had the same problem. I looked through many, many threads on Stackoverflow that recommended to turn on autocommit or use commit() after an SQL statement (ex. one, two, three), which I tried and failed.
When I added data with HeidiSQL to my database it showed up in the loop query, but I don't really know why this is. Rows inserted with mysql-client on Linux and my Python insertion script never show up until I restart my loop script.
I don't know if it's the fact that I open 2 connections, each in their own script, but I close every connection and every cursor when I'm done with them.
The problem could be with your variable now. I don't see anywhere in the loop that it is being reset.
I'd probably use the mysql NOW() function:
c.execute("SELECT id, message FROM tasks WHERE date <= NOW() AND done = 0;")
It looks like the time you are inserting into the database is a time in the future. I don't think your issue is with your database connection, I think it's something to do with the queries you are doing.