I am executing a selfwritten postgresql function in a loop for several times from Python. I am using the psycopg2 framework to do this.
The function I wrote hast the following structure:
CREATE OR REPLACE FUNCTION my_func()
RETURNS void AS
$$
BEGIN
-- create a temporary table that should be deleted after
-- the functions finishes
-- normally a CREATE TABLE ... would be here
CREATE TEMPORARY TABLE temp_t
(
seq integer,
...
) ON COMMIT DROP;
-- now the insert
INSERT INTO temp_t
SELECT
...
END
$$
LANGUAGE 'plpgsql';
Thats basically the python part
import time
import psycopg2
conn = psycopg2.connect(host="localhost", user="user", password="...", dbname="some_db")
cur = conn.cursor()
for i in range(1, 11):
print i
print time.clock()
cur.callproc("my_func")
print time.clock()
cur.close()
conn.close()
The error I get when I run the python script is:
---> relation "temp_t" already exists
Basically I want to measure how long it takes to execute the function. Doing that, the loop shall run several times. Storing the result of the SELECT in a temporary table is supposed to replace the CREATE TABLE ... part which would normally create the output table
Why doesnt postgres drop the function after I executed the function from Python?
All the function calls in the loop are performed in a single transaction, so the temporary table is not dropped each time. Setting autocommit should change this behavior:
...
conn = psycopg2.connect(host="localhost", user="user", password="...", dbname="some_db")
conn.autocommit = True
cur = conn.cursor()
for i in range(1, 11):
...
Temporary tables are dropped when the session ends. Since your session does not end with the function call, the second function call will try to create the table again. You need to alter your store function and to check whether the temporary table already exists and create it if it doesn't. This post can help you in doing so.
Another quick n dirty is to connect and disconnet after each function call.
import time
import psycopg2
for i in range(1, 11):
conn = psycopg2.connect(host="localhost", user="user", password="...", dbname="some_db")
cur = conn.cursor()
print i
print time.clock()
cur.callproc("my_func")
print time.clock()
cur.close()
conn.close()
Not nice, but does the trick.
Related
I am writing code to create a GUI in Python on the Spyder environment of Anaconda. within this code I operate with a PostgreSQL database and I therefore use the psycopg2 database adapter so that I can interact with directly from the GUI.
The code is too long to post here, as it is over 3000 lines, but to summarize, I have no problem interacting with my database except when I try to drop a table.
When I do so, the GUI frames become unresponsive, the drop table query doesn't drop the intended table and no errors or anything else of that kind are thrown.
Within my code, all operations which result in a table being dropped are processed via a function (DeleteTable). When I call this function, there are no problems as I have inserted several print statements previously which confirmed that everything was in order. The problem occurs when I execute the statement with the cur.execute(sql) line of code.
Can anybody figure out why my tables won't drop?
def DeleteTable(table_name):
conn=psycopg2.connect("host='localhost' dbname='trial2' user='postgres' password='postgres'")
cur=conn.cursor()
sql="""DROP TABLE """+table_name+""";"""
cur.execute(sql)
conn.commit()
That must be because a concurrent transaction is holding a lock that blocks the DROP TABLE statement.
Examine the pg_stat_activity view and watch out for sessions with state equal to idle in transaction or active that have an xact_start of more than a few seconds ago.
This is essentially an application bug: you must make sure that all transactions are closed immediately, otherwise Bad Things can happen.
I am having the same issue when using psycopg2 within airflow's postgres hook and I resolved it with with statement. Probably this resolves the issue because the connection becomes local within the with statement.
def drop_table():
with PostgresHook(postgres_conn_id="your_connection").get_conn() as conn:
cur = conn.cursor()
cur.execute("DROP TABLE IF EXISTS your_table")
task_drop_table = PythonOperator(
task_id="drop_table",
python_callable=drop_table
)
And a solution is possible for the original code above like this (I didn't test this one):
def DeleteTable(table_name):
with psycopg2.connect("host='localhost' dbname='trial2' user='postgres' password='postgres'") as conn:
cur=conn.cursor()
sql="""DROP TABLE """+table_name+""";"""
cur.execute(sql)
conn.commit()
Please comment if anyone tries this.
While creating a new program to handle a database I encountered a problem.
The mysql cursor doesn't seem to notice the WHERE statement in the following function:
def get_new_entries(self, delay):
start_time = t.time()
while True:
cursor = self.cnx.cursor()
check_delay = delay * 2
current_time = datetime.datetime.now() - datetime.timedelta(seconds=check_delay)
current_time = current_time.strftime("%Y-%m-%d %H:%M:%S")
data = current_time
print(current_time)
query = """SELECT FILE_NAME, CREATION_TIME FROM `added_files` WHERE CREATION_TIME < %s"""
cursor.execute(query, (data,))
print(query)
for (FILE_NAME, CREATION_TIME) in cursor:
print(FILE_NAME)
print(CREATION_TIME)
cursor.close()
cursor = None
print("###################")
t.sleep(delay - ((t.time() - start_time) % delay))
With this function I wish to achieve that every minute the function checks for new entries in the past minute. In the end I want it to send the filenames as a list to another class, so that class can use logic to handle the filenames.
However, the WHERE CREATION_TIME < '%s' doesnt seem to do anything. It either doesn't return any entry, while trying the same query in the mysql environment itself does what it should. If however the '<' is changed to '>' it suddenly returns all items, even those which should NOT be returned.
With that said, I have also used this part of code with only
cursor.execute(query)
while the query was changed to
query = "SELECT FILE_NAME, CREATION_TIME FROMadded_filesWHERE CREATION_TIME < {}".format(current_time).
This worked the first time, but the second time in the loop it didn't respond anything, even though I did add stuff to the database. I used the same datetime the program used in the mysql environment in the browser, which returned the correct results, but the program didn't.
So why doesn't it work? And what should I do to make it work?
So after a while I solved the problem. It had nothing to do with the code I sent but had to do with the lack of autocommit = True of the MYSQL connection. I'll try and explain.
My application had to check a database that is automatically updated by another (C#) application. For every new file the C# app found it created a new entry in the database. Meanwhile this Python application checks for new filenames every delay (e.g. 60.0 seconds).
The Python application opens a connection via an mysql.connector and seems to keep hold on the state of the database at that exact moment. Anything added to it will not be found by any code you post, because it doesn't search in the actual database, it searches in it's own saved version of the database.
The fix would be to set the connection to autocommit. So you would do this:
self.cnx = mysql.connector.connect(user='root', password='', host='127.0.0.1', database='mydb')
self.cnx.autocommit = True
This will update the saved state of the database in the python app every time you execute an sql query.
So you don't get what you expect from you sql query in python, go try setting autocommit to true for your connection.
I was reading up on the documentation and saw that you only have to commit in once in a transaction?
Does the following count as one transaction or does each function count as a transaction?
def main():
conn=pyodbc.connect(sqlconnectionstring) # Assume this connects to the database
cursor = conn.cursor()
function1()
function2()
conn.commit()
def function1():
# does inserting here
def function2():
# does inserting here and calls function 3
function3()
def function 3():
# does more inserting here
main()
Is that conn.commit() enough to commit all insertions in all the functions or would I have to pass the "conn" variable as an argument and commit inside each function?
Thanks!
yes thats enough to commit all the transactions as the ones like insert and delete will all be performed inside the functions, until one fail then you will find the old row.
but that one commit will change the database status to the recent one
I have a python script which creates a database and then enters an infinite loop which runs once per second querying the database with some selects.
At the same time I connect to the database with a sqlite cli and try to make an update but I get a database is locked error.
Here the (anonymized) code of the script:
import sqlite3
import time
con = sqlite3.connect(r'path\to\database.sqlite')
con.execute('DROP TABLE IF EXISTS blah;')
con.execute('CREATE TABLE blah;')
con.execute('INSERT INTO blah;')
con.commit()
while True:
result = con.execute('SELECT blah')
print(result.fetchone()[0])
time.sleep(1)
Python's sqlite3 module tries to be clever and manages transactions for you.
To ensure that you can access the database from other threads/processes, disable that (set isolation_level to None), and use explicit transactions, when needed.
Alternatively, call con.commit() whenever you are finished.
I made a loop in Python that calls itself to repeatedly check for new entries in a database. On first execution, all affected rows are shown fine. Meanwhile, I add more rows into the database. On the next query in my loop, the new rows are not shown.
This is my query-loop:
def loop():
global mysqlconfig # username, passwd...
tbd=[] # this is where I save the result
conn = MySQLdb.connect(**mysqlconfig)
conn.autocommit(True)
c = conn.cursor()
c.execute("SELECT id, message FROM tasks WHERE date <= '%s' AND done = 0;" % now.isoformat(' '))
conn.commit()
tbd = c.fetchall()
print tbd
c.close()
conn.close()
time.sleep(5)
loop()
loop()
This is the SQL part of my Python insertion-script:
conn = MySQLdb.connect(**mysqlconfig)
conn.autocommit(1)
c = conn.cursor()
c.execute("INSERT INTO tasks (date, message) VALUES ('{0}', '{1}');".format("2012-10-28 23:50", "test"))
conn.commit()
id = c.lastrowid
c.close()
conn.close()
I tried SQLite, I tried Oracle MySQL's connector, I tried MySQLdb on a Windows and Linux system and all had the same problem. I looked through many, many threads on Stackoverflow that recommended to turn on autocommit or use commit() after an SQL statement (ex. one, two, three), which I tried and failed.
When I added data with HeidiSQL to my database it showed up in the loop query, but I don't really know why this is. Rows inserted with mysql-client on Linux and my Python insertion script never show up until I restart my loop script.
I don't know if it's the fact that I open 2 connections, each in their own script, but I close every connection and every cursor when I'm done with them.
The problem could be with your variable now. I don't see anywhere in the loop that it is being reset.
I'd probably use the mysql NOW() function:
c.execute("SELECT id, message FROM tasks WHERE date <= NOW() AND done = 0;")
It looks like the time you are inserting into the database is a time in the future. I don't think your issue is with your database connection, I think it's something to do with the queries you are doing.