I have a Python app running on Linux. It is called every minute from cron. It checks a directory for files and if it finds one it processes it - this can take several minutes. I don't want the next cron job to pick up the file currently being processed so I lock it using the code below which calls portalocker. The problem is it doesn't seem to work. The next cron job manages to get a file handle returned for the file all ready being processed.
def open_and_lock(full_filename):
file_handle = open(full_filename, 'r')
try:
portalocker.lock(file_handle, portalocker.LOCK_EX
| portalocker.LOCK_NB)
return file_handle
except IOError:
sys.exit(-1)
Any ideas what I can do to lock the file so no other process can get it?
UPDATE
Thanks to #Winston Ewert I checked through the code and found the file handle was being closed way before the processing had finished. It seems to be working now except the second process blocks on portalocker.lock rather than throwing an exception.
After fumbling with many schemes, this works in my case. I have a script that may be executed multiple times simultaneously. I need these instances to wait their turn to read/write to some files. The lockfile does not need to be deleted, so you avoid blocking all access if one script fails before deleting it.
import fcntl
def acquireLock():
''' acquire exclusive lock file access '''
locked_file_descriptor = open('lockfile.LOCK', 'w+')
fcntl.lockf(locked_file_descriptor, fcntl.LOCK_EX)
return locked_file_descriptor
def releaseLock(locked_file_descriptor):
''' release exclusive lock file access '''
locked_file_descriptor.close()
lock_fd = acquireLock()
# ... do stuff with exclusive access to your file(s)
releaseLock(lock_fd)
You're using the LOCK_NB flag which means that the call is non-blocking and will just return immediately on failure. That is presumably happening in the second process. The reason why it is still able to read the file is that portalocker ultimately uses flock(2) locks, and, as mentioned in the flock(2) man page:
flock(2) places advisory locks only;
given suitable permissions on a file,
a process is free to ignore the use of
flock(2) and perform I/O on the file.
To fix it you could use the fcntl.flock function directly (portalocker is just a thin wrapper around it on Linux) and check the returned value to see if the lock succeeded.
Don't use cron for this. Linux has inotify, which can notify applications when a filesystem event occurs. There is a Python binding for inotify called pyinotify.
Thus, you don't need to lock the file -- you just need to react to IN_CLOSE_WRITE events (i.e. when a file opened for writing was closed). (You also won't need to spawn a new process every minute.)
An alternative to using pyinotify is incron which allows you to write an incrontab (very much in the same style as a crontab), to interact with the inotify system.
what about manually creating an old-fashioned .lock-file next to the file you want to lock?
just check if it’s there; if not, create it, if it is, exit prematurely. after finishing, delete it.
I think fcntl.lockf is what you are looking for.
Related
I have to wait till file copy/upload finishes completely using python (preferred approach), bash/shell also fine(I will call from python)
I have shared nfs directory /data/files_in/, if somebody copies/uploads a file to /data/files_in/ directory, I should notify to other application, only after complete file copy/upload is done
My current code to check file is completed copied or not
while True:
current_size = Path(file_path).stat().st_size
time.sleep(5)
result_size = Path(file_path).stat().st_size
if result_size == current_size:
break
# Notify your application
It is working only with small size files, for large files like 100G files it is not working properly.
I have increased a timer, but still sometimes it is failing and timer based approach seems not good idea to rely on.
Is there any other way, I can implement code to fix this issue?
OS: Linux, Cent os
Python Version: 3.9
I can't comment so I will ask here. Shouldn't the resulting size be larger (or at least different) from the current one in order for a file to be done uploading and therefore stop the loop?
I assume you cannot establish any kind of direct communications with the other process, i.e. the one which is copying/uploading the file.
One common approach in these cases is to have the other process to write/erase a "semaphore" file. It may be that it creates the semaphore just before beginning copying and erases it just after finishing, so the semaphore means "don't do anything, I'm still running", or the other way round, it creates the semaphore just after finishing and erases it just before starting next time, so the semaphore means "your data are ready to use".
That said, I'm amazed your approach doesn't work if you allow enough time, and 5 secs should be more than enough on any networks
I create an executable using py2exe in python.
I was looking at this post but unfortunately the answers were superficial.
The first solution using Tendo but this works to limit 1 instance of application per user, and my app is being used in a Windows Server Environment where there are 20+ users loged in at a time.
And the second solution offered Listening to a defined port doesn't have examples of how it could be accomplished.
So I decided to go for mutexes to prevent my app from running multiple times.
So I currently use this code for using mutexes, but it doesn't have mutex detection between applications and Services.
This post shows how to accomplish mutexes, but doesn't show how its done in python.
How could I use Mutexes to have single instance of program on Windows where the mutexes don't limit single instance of program on Windows, and has detection between applications and Services.
I'm not sure why you'd have to use mutexes to this purpose on Windows? There's much simpler option: a bad old lockfile.
If all you want to achieve is making sure that only a single instance of the app runs, you could do something like this:
Windows supports you here since you can't delete a file if it's opened by another process. So (code untested):
tempdir = tempfile.gettempdir()
lockfile = os.sep.join([tempdir, 'myapp.lock'])
try:
if os.path.isfile(lockfile):
os.unlink(lockfile)
except WindowsError as e: # Should give you smth like 'WindowsError: [Error 32] The process cannot access the file because it is being used by another process..'
# there's instance already running
sys.exit(0)
with open(lockfile, 'wb') as lockfileobj:
# run your app's main here
main()
os.unlink(lockfile)
with section ensures the file is opened when your main runs and it is closed when your main finishes running. Then os.unlink removes the lockfile.
If another instance tries to start up, it exits on WindowsError exception (it would be good to check its numeric code though to be sure it's precisely the case of the file already opened).
Above is a rough solution, a niftier one would be to use entry/exit functionality to delete lockfile if main exits for any reason. Explanation here: http://effbot.org/zone/python-with-statement.htm
I read How do you create a daemon in Python? and also this topic, and tried to write a very simple daemon :
import daemon
import time
with daemon.DaemonContext():
while True:
with open('a.txt', 'a') as f:
f.write('Hi')
time.sleep(2)
Doing python script.py works and returns immediately to terminal (that's the expected behaviour). But a.txt is never written and I don't get any error message. What's wrong with this simple daemon?
daemon.DaemonContext() has option working_directory that has default fault value / i.e. your program probably doesn't have permission to create a new file there.
The problem described here is solved by J.J. Hakala's answer.
Two additional (important) things :
Sander's code (mentioned here) is better than python-daemon. It is more reliable. Just one example: try to start two times the same daemon with python-daemon : big ugly error. With Sander's code : a nice notice "Daemon already running."
For those who want to use python-daemon anyway: DaemonContext() only makes a daemon. DaemonRunner() makes a daemon + control tool, allowing to do python script.py start or stop, etc.
One thing that's wrong with it, is it has no way to tell you what's wrong with it :-)
A daemon process is, by definition, detached from the parent process and from any controlling terminal. So if it's got something to say – such as error messages – it will need to arrange that before becoming a daemon.
From the python-daemon FAQ document:
Why does the output stop after opening the daemon context?
The specified behaviour in PEP 3143_ includes the requirement to
detach the process from the controlling terminal (to allow the process
to continue to run as a daemon), and to close all file descriptors not
known to be safe once detached (to ensure any files that continue to
be used are under the control of the daemon process).
If you want the process to generate output via the system streams
‘sys.stdout’ and ‘sys.stderr’, set the ‘DaemonContext’'s ‘stdout’
and/or ‘stderr’ options to a file-like object (e.g. the ‘stream’
attribute of a ‘logging.Handler’ instance). If these objects have file
descriptors, they will be preserved when the daemon context opens.
Set up a working channel of communication, such as a log file. Ensure the files you open aren't closed along with everything else, using the files_preserve option. Then log any errors to that channel.
I have a network drive (Z:\) which is shared by multiple Windows computers. Is it possible to implement a cross-machine lock by simply creating/deleting files on this network drive?
For example, two computers, A and B, want to write to a shared resource with an ID of 123 at the same time.
One of the computers, say A, locks the resource first by creating an empty file Z:\locks\123. When B sees there is the lock file with the name of "123", B knows the resource 123 is being used by someone else, so it has to wait for Z:\locks\123 to be deleted by A before it can access the resource.
It's like critical section in multithreading, but I want to do it on multiple machines.
I'm trying to implement in Python. Here's what I came up with:
import os
import time
def lock_it(lock_id):
lock_path = "Z:\\locks\\" + lock_id
while os.path.exists(lock_path):
time.sleep(5) # wait for 5 seconds
# create the lock file
lock_file = open(lock_path, "w")
lock_file.close()
def unlock_it(lock_id):
# delete the lock file
lock_path = "Z:\\locks\\" + lock_id
if os.path.exists(lock_path):
os.remove(lock_path)
This won't work because there could be more than one processes exit the waiting status and create the lock file at the same time.
So again, the question is: Is it possible to implement a cross-machine locking mechanism on a shared storage?
... sort of.
First, you should create a lock directory instead of a lock file. Creating a directory (see os.mkdir) will fail if the directory already exists, so you can acquire the lock like this:
while True:
try:
os.mkdir(r"z:\my_lock")
return
except OSError as e:
if e.errno != 21: # Double check that errno will be the same on Windows
raise
time.sleep(5)
Second (and this is where the "sort of" comes in) you'll want some way to notice when the person holding the lock has died. One simple way to do this might be to have them occasionally update a file inside the lock directory. Then if clients notice that file hasn't been updated in a while, they can remove the directory and try to acquire the lock themselves.
This will not work nearly as well you as might hope. You'll have other issues such as the network drive going away, in which case all your processes will either be stuck or think that no one is holding a lock.
I suggest you look into something like ZooKeeper. You will be able to create synchronous locks and recover in the event of network failures. The framework behind distributed locks is much more complex then creating a file on a network drive.
I have a possibly long running program that currently has 4 processes, but could be configured to have more. I have researched logging from multiple processes using python's logging and am using the SocketHandler approach discussed here. I never had any problems having a single logger (no sockets), but from what I read I was told it would fail eventually and unexpectedly. As far as I understand its unknown what will happen when you try to write to the same file at the same time. My code essentially does the following:
import logging
log = logging.getLogger(__name__)
def monitor(...):
# Spawn child processes with os.fork()
# os.wait() and act accordingly
def main():
log_server_pid = os.fork()
if log_server_pid == 0:
# Create a LogRecordSocketServer (daemon)
...
sys.exit(0)
# Add SocketHandler to root logger
...
monitor(<configuration stuff>)
if __name__ == "__main__":
main()
So my questions are: Do I need to create a new log object after each os.fork()? What happens to the existing global log object?
With doing things the way I am, am I even getting around the problem that I'm trying to avoid (multiple open files/sockets)? Will this fail and why will it fail (I'd like to be able to tell if future similar implementations will fail)?
Also, in what way does the "normal" (one log= expression) method of logging to one file from multiple processes fail? Does it raise an IOError/OSError? Or does it just not completely write data to the file?
If someone could provide an answer or links to help me out, that would be great. Thanks.
FYI:
I am testing on Mac OS X Lion and the code will probably end up running on a CentOS 6 VM on a Windows machine (if that matters). Whatever solution I use does not need to work on Windows, but should work on a Unix based system.
UPDATE: This question has started to move away from logging specific behavior and is more in the realm of what does linux do with file descriptors during forks. I pulled out one of my college textbooks and it seems that if you open a file in append mode from two processes (not before a fork) that they will both be able to write to the file properly as long as your write doesn't exceed the actual kernel buffer (although line buffering might need to be used, still not sure on that one). This creates 2 file table entries and one v-node table entry. Opening a file then forking isn't supposed to work, but it seems to as long as you don't exceed the kernel buffer as before (I've done it in a previous program).
So I guess, if you want platform independent multi-processing logging you use sockets and create a new SocketHandler after each fork to be safe as Vinay suggested below (that should work everywhere). For me, since I have strong control over what OS my software is being run on, I think I'm going to go with one global log object with a FileHandler (opens in append mode by default and line buffered on most OSs). The documentation for open says "A negative buffering means to use the system default, which is usually line buffered for tty devices and fully buffered for other files. If omitted, the system default is used." or I could just create my own logging stream to be sure of line buffering. And just to be clear, I'm ok with:
# Process A
a_file.write("A\n")
a_file.write("A\n")
# Process B
a_file.write("B\n")
producing...
A\n
B\n
A\n
as long as it doesn't produce...
AB\n
\n
A\n
Vinay(or anyone else), how wrong am I? Let me know. Thanks for any more clarity/sureness you can provide.
Do I need to create a new log object after each os.fork()? What happens to the existing global log object?
AFAIK the global log object remains pointing to the same logger in parent and child processes. So you shouldn't need to create a new one. However, I think you should create and add the SocketHandler after the fork() in monitor(), so that the socket server has four distinct connections, one to each child process. If you don't do this, then the child processes forked off in monitor() will inherit the SocketHandler and its socket handle from their parent, but I don't know for sure that it will misbehave. The behaviour is likely to be OS-dependent and you may be lucky on OSX.
With doing things the way I am, am I even getting around the problem that I'm trying to avoid (multiple open files/sockets)? Will this fail and why will it fail (I'd like to be able to tell if future similar implementations will fail)?
I wouldn't expect failure if you create the socket connection to the socket server after the last fork() as I suggested above, but I am not sure the behaviour is well-defined in any other case. You refer to multiple open files but I see no reference to opening files in your pseudocode snippet, just opening sockets.
Also, in what way does the "normal" (one log= expression) method of logging to one file from multiple processes fail? Does it raise an IOError/OSError? Or does it just not completely write data to the file?
I think the behaviour is not well-defined, but one would expect failure modes to present as interspersed log messages from different processes in the file, e.g.
Process A writes first part of its message
Process B writes its message
Process A writes second part of its message
Update: If you use a FileHandler in the way you described in your comment, things will not be so good, due to the scenario I've described above: process A and B both begin pointing at the end of file (because of the append mode), but thereafter things can get out of sync because (e.g. on a multiprocessor, but even potentially on a uniprocessor), one process can (preempt another and) write to the shared file handle before another process has finished doing so.