I need to prevent a python script from running more than once. So far I have:
import fcntl
def lockFile(lockfile):
fp = open(lockfile, 'w')
try:
fcntl.flock(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError:
return False
return True
if not lockFile("myfile.lock"):
sys.exit(0)
Problem is the sys.exit() never gets called even if the file is there. Maybe this is a platform dependent way of doing things? I just need to write a lockfile, check for its existence and if it's not there or stale, create a new one. Ideas?
Writing a file will create a new file if none exist; you could instead try to read the file first: if there is none, an error is raised and a file a file written; if there is a file, the program exits.
try:
with open('lockfile.txt', 'r') as f:
lock = f.readline().strip().split()
if lock[0] == 'locked':
print('exiting')
sys.exit(0)
except FileNotFoundError:
with open('lockfile.txt', 'w') as f:
f.write('locked')
print('file written')
if __name__ == '__main__':
pass
If you need something more sophisticated, you could look up the atexit module
You can check if you file exists by using os.path.exists (docs here). If it does, then you can use the sys.exit you mentioned before. If you need something more than sys.exit, try the atexit module #Reblochon suggested. The script will then assume the file is ready to lock and the method will report its success back to the user via a boolean.
import os
import sys
import fcntl
FILE_NAME = 'myfile.lock'
def lockFile(lockfile):
fp = open(lockfile, 'w') # create a new one
try:
fcntl.flock(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
success = True # the file has been locked
except IOError:
success = False # an error occurred
fp.close() # make sure to close your file
return success
if os.path.exists(FILE_NAME): # exit the script if it exists
sys.exit(0)
print('success', lockFile(FILE_NAME))
Related
How can I check when a file window is closed? I tried this:
import time
import os
path = input()
os.startfile(request)
try:
file_path = open(path, 'rb')
except Exception:
done = True
while done:
if file_path.closed:
done = False
time.sleep(SLEEP_TIME)
So the file starts before I start checking if the file is closed. Then, I open the file again, and because its open already, an exception should pop up, where I'm looping until the file is closed. I don't know whether the indication I made is right or not.
Please help!!! I'm desperate!!!
You can't use file_path in the except: block, because the variable isn't assigned if open() raises an exception.
You should put the loop around the try/except, not inside the except.
while True:
try:
file_path = open(path, 'rb')
break
except:
time.sleep(SLEEP_TIME)
It will open the file only once, and if the file is already open it will use the already opened fie
import time
import os
path = 'bescom-17-apr.pdf'
try:
with open(path, "r") as file:
print("opened file")
except IOError:
print("file not found")
What is the best practice to try to open the file and retry after n seconds?
Currently, I do:
import os
from os import path
import shutil
dir_path = path.abspath(path.join("foo", "bar"))
destination_path = path.abspath(path.join("foo", "dest_dir"))
for f in dir_path:
try:
# try to open the file if the file isn't used any longer
file_opened = open(f, 'r')
# move the file on successful opening
shutil.move(file_opened, destination_path)
file_opened.close()
except IOError:
return False
So, at the moment I do not handle the exception. I think about creation of extra function to open the file and recall the function on excepth with time.sleep(n)
However, I am sure there must be something else ...
I do not use
with open(f, 'rb') as file_opened:
do whatever`
EDIT:
One process creates the file and I want Python process to move the file once I am sure the file writting / creation is completed. So, I have added shutil.move in the above code to show the whole situation.
EDIT:
Please find below the code I have developed to solve the problem. I ended with writing own custom solution to handle it:
import os
from os import path
import psutil
from retry import retry
import shutil
from subprocess import check_output,Popen, PIPE
import glob
import time
class FileHandler:
def __init__(self, fn_source, dir_source):
self.file_source_name = fn_source
self.file_source_path = path.join(dir_source, self.file_source_name)
self.dir_dest_path = path.join(dir_source, "test")
self.file_dest_path = path.join(self.dir_dest_path, self.file_source_name)
def check_file(self):
if os.path.exists(self.file_source_path):
try:
os.rename(self.file_source_path, self.file_source_path)
print("file renamed")
return True
except:
print("can not rename the file..retrying")
time.sleep(1)
self.check_file()
else:
print("source file does not exist...retrying")
time.sleep(5)
self.check_file()
def check_destination(self):
if os.path.exists(self.file_source_path) and not os.path.exists(self.file_dest_path):
return True
elif os.path.exists(self.file_source_path) and os.path.exists(self.file_dest_path):
try:
print(self.file_dest_path, self.file_source_name)
os.remove(self.file_dest_path)
return True
except Exception as e:
print("can not remove the file..retrying")
time.sleep(5)
self.check_destination()
def move_file(self):
if self.check_destination():
print(self.file_source_path)
shutil.move(self.file_source_path, self.file_dest_path)
print("moved", str(self.file_source_path))
return True
else:
print("can not move the file..retrying")
time.sleep(1)
self.move_file()
def file_ops(self):
if self.check_file():
self.move_file()
else:
print("source does not exist")
time.sleep(1)
self.file_ops()
return True
def each_file_ops(fn, dir_source):
fh = FileHandler(fn, dir_source)
return fh.file_ops()
def main(dir_source):
dir_files = glob.glob(path.join(dir_source, '*.txt'))
if dir_files:
[each_file_ops(f, dir_source) for f in dir_files]
else:
print("source dir is empty")
time.sleep(1)
main(dir_source)
if __name__ == '__main__':
main(path.join(""))
You can use the retry module for these kind of retrying. This makes the code to look much cleaner. pip install retry should install the module
from retry import retry
import shutil
#retry((FileNotFoundError, IOError), delay=1, backoff=2, max_delay=10, tries=100)
def attempt_to_move_file(fname, dest_path):
# Your attempt to move file
# shutil.move(fname, destination_path)
With the above code, when attempt_to_move_file is invoked it would be retried (upto a max of 100 tries) whenever we hit FileNotFoundError or IOError and the retry happens with a sleep 1, 2, 4, 8, 10, 10, 10 ... seconds between attempts
you do not need to open a file to move it. Because once you have opened a file it is in open state hence you can not move it, Take it like this, when you are playing a video in a player and you try to delete it or cut and paste system won't allow you.
import shutil
file_name = 'trytry.csv'
shutil.copyfile(file_name, '/Users/umeshkaushik/PycharmProjects/newtry.csv')
shutil.move(file_name, '/Users/umeshkaushik/PycharmProjects/newtry1.csv')
Above code runs fine. Just make sure you are providing correct input and output path.
I have a python program (say reader.py) which uses file setting.py to read from:
while( True ):
...
execfile( settings.py )
...
But there is other python program (say writer.py) that uses this file to write to:
...
try:
settings = open('settings.py', 'w')
settings.truncate()
settings.write( 'some text')
except IOError:
print('Cannot write to file')
finally:
settings.close()
...
Note1: reader.py and writer.py do not ''know'' about each other.
Note2: reader.py reads settings.py cyclically, though writer.py writes to file when user wants to (not necessarily right after he/she clicked ''write'', it just means that there is no any rule when to write).
Question: What is the best way to cooperate two programs in order to avoid any contradiction? I know this might depend on platform. I am using Linux. Distributions are: Ubuntu, Scientific Linux.
EDIT1: If I choose to use FiFo I encounter the following problem: Once writer has write to settings file it will probably never write again but reader should have access to settings anyway in this case. In other words, reader should have an ability to read from file and not to wait for writer in this case. Otherwise reader has to wait for writer.
Ordinary using of FiFo does not allow reader to read from file if writer does not write (until it has written). How to deal with this problem?
You may be interested in using a named pipe for your interprocess communications. Available in Linux, it is a special type of file designed for client (writer.py), server (reader.py), tasks. After writing to the pipe, the client will wait until the server has received the data. This allows you to sync the two processes somewhat.
Linux Manual for FiFo
Python doc: os.mkfifo(path[, mode])
I found the following solution which seems to be working. I use flock to create locks.
Reader:
import errno
import fcntl
from time import *
path = "testLock.py"
f = open(path, "r")
while True:
try:
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
break
except IOError as e:
if e.errno != errno.EAGAIN:
raise
else:
sleep(1)
print 'Waiting...'
#reader's action
execfile(path)
#drop lock
fcntl.flock(f, fcntl.LOCK_UN)
Writer:
import errno
import fcntl
from time import *
path = "testLock.py"
f = open(path, "w")
while True:
try:
fcntl.flock(f, fcntl.LOCK_SH | fcntl.LOCK_NB)
break
except IOError as e:
if e.errno != errno.EAGAIN:
raise
else:
sleep(1)
print 'Waiting...'
#writer's action
for i in (1,10,2):
f.write('print "%d" % i')
sleep(1)
#drop lock
fcntl.flock(f, fcntl.LOCK_UN)
I have some question here:
Qusetion 1: Is it correct usage of LOCK_EX and LOCK_SH I mean are they in the right place?
Question 2: Is the reader's action i.e execfile correct here? If the file is already opened is execfile try to open it anyway?
I am running the following code very often to write to files from Python using the following:
def function():
file = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../../my_file')
fifo = open(file, 'w')
if (os.path.isfile(file) and os.access(file, os.W_OK)):
fifo.write("stuff")
fifo.close()
else:
time.sleep(1)
function()
Unfortunately, I get the following error (not all the time):
IOError: [Errno 2] No such file or directory
This is self-explanatory, but I do not understand then why all the precautions do not work, like isfile or access..W_OK ? How do I avoid having this issue?
Also, the slower the machine, the less often the error is encountered.
See also "LBYL vs EAFP in Java?":
Instead of checking if the action is ok (LBYL), you should do an EAFP approach. Just do what you want to do and check if it has worked correctly.
Besides, better don't call function() recursively, it might lead to a stack overflow if the error persists.
Better do
def function():
file = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../../my_file')
while True:
try:
with open(file, 'w') as file: # automatically closes...
file.write("some stuff")
except IOError, e:
time.sleep(1)
continue
break # the while loop
I'm trying to read all files from a folder that matches a certain criteria. My program crashes once I have an exception raised. I am trying to continue even if there's an exception but it still stops executing.
This is what I get after a couple of seconds.
error <type 'exceptions.IOError'>
Here's my code
import os
path = 'Y:\\Files\\'
listing = os.listdir(path)
try:
for infile in listing:
if infile.startswith("ABC"):
fo = open(infile,"r")
for line in fo:
if line.startswith("REVIEW"):
print infile
fo.close()
except:
print "error "+str(IOError)
pass
Put your try/except structure more in-wards. Otherwise when you get an error, it will break all the loops.
Perhaps after the first for-loop, add the try/except. Then if an error is raised, it will continue with the next file.
for infile in listing:
try:
if infile.startswith("ABC"):
fo = open(infile,"r")
for line in fo:
if line.startswith("REVIEW"):
print infile
fo.close()
except:
pass
This is a perfect example of why you should use a with statement here to open files. When you open the file using open(), but an error is catched, the file will remain open forever. Now is better than never.
for infile in listing:
try:
if infile.startswith("ABC"):
with open(infile,"r") as fo
for line in fo:
if line.startswith("REVIEW"):
print infile
except:
pass
Now if an error is caught, the file will be closed, as that is what the with statement does.
Move the try/except inside the for loop.
Like in:
import os
path = 'C:\\'
listing = os.listdir(path)
for infile in listing:
try:
if infile.startswith("ABC"):
fo = open(infile,"r")
for line in fo:
if line.startswith("REVIEW"):
print infile
fo.close()
except:
print "error "+str(IOError)
You're code is doing exactly what you're telling it to do. When you get an exception, it jumps down to this section:
except:
print "error "+str(IOError)
pass
Since there's nothing after that, the program ends.
Also, that pass is superfluous.
As per strictest interpretation of the question "continue even if there's an exception". Python gives us a keyword "finally" which executes a block of code no matter what precedes it. The only issue with this method will run a block of code regardless of the type of error, which might not be desirable for all cases.
try:
unreal = 3/0 # raises divide by zero exception
print(unreal)
# handles zerodivision exception
except :
print("Can't divide by zero, 0 has no multiplicative inverse")
finally:
# this block is always executed
print("Brahmagupta claimed that “zero divided by a zero is zero.”)