Basically I am writing a script that can be stopped and resumed at any time. So if the user uses, say PyCharm console to execute the program, he can just click on the stop button whenever he wants.
Now, I need to save some variables and let an ongoing function finish before terminating. What functions do I use for this?
I have already tried atexit.register() to no avail.
Also, how do I make sure that an ongoing function is completed before the program can exit?
Solved it using a really bad workaround. I used all functions that are related to exit in Python, including SIG* functions, but uniquely, I did not find a way to catch the exit signal when Python program is being stopped by pressing the "Stop" button in PyCharm application. Finally got a workaround by using tkinter to open an empty window, with my program running in a background thread, and used that to close/stop program execution. Works wonderfully, and catches the SIG* signal as well as executing atexit . Anyways massive thanks to #scrineym as the link really gave a lot of useful information that did help me in development of the final version.
It looks like you might want to catch a signal.
When a program is told to stop a signal is sent to the process from the OS, you can then catch them and do cleanup before exit. There are many diffferent signals , for xample when you press CTRL+C a SIGINT signal is sent by the OS to stop your process, but there are many others.
See here : How do I capture SIGINT in Python?
and here for the signal library: https://docs.python.org/2/library/signal.html
Related
I am running some Python code using a SLURM script on a remote server accessed through SSH. At some point, issues related to licenses on the SLURM platform may happen, generating errors in Python and ending the subprocess. I want to use try-except to let the Python subprocess wait until the issue is fixed, after that it can keep running from where it stopped.
What are some smart implementations for that?
My most obvious solution is just keeping Python inside a loop if the error occurs and letting it read a file every X seconds, when I finally fix the error and want it to keep running from where it stopped, I would write something on the file and break the loop. I wonder if there is a smarter way to provide input to the Python subprocess while it is running through the SLURM script.
One idea might be to add a signal handler for signal USR1 to your Python script like this.
In the signal handler function, you can set a global variable or send a message or set a threading.Event that the main process is waiting on.
Then you can signal the process with:
kill -USR1 <PID>
or with the Python os.kill() equivalent.
Though I do have to agree there is something to be said for the simplicity of your process doing:
touch /tmp/blocked.$$
and your program waiting in a loop with a 1s sleep for that file to be removed. This way you can tell which process id is blocked.
I have been having trouble with exiting Mulitprocessing pool after keyboard interrupt, and after a long time of trying a gave up, but if i just exited CMD after im done/ have done what i needed with my script, what would be the downsides? I know this isn't a good practice, but it dosen't really need to be good. I'm assuming everything gets killed after the Command line is exited, but i'm not sure.
I think it depends. If it is a Python console the ressources might get cleaned up. But I know from software crashes in Python that sometimes the child processes stay alive even when the main application closes. Usually when started directly via the file explorer.
To make sure that your sub threads/process get closed when the main application closes, you shoud set them as daemon. Then killing the main script kills the childs as well.
see this Link
Quick Question: What signal/Exception does PyCharm's Stop Button send when debugging a python script?
Background: Several posts document that hitting Ctrl-C doesn't send a Keyboard Interrupt/SIGINT signal to a python script when using PyCharm's Debugger. Fine. My question is, what does get sent to the Python script when clicking the Debugger's "Stop Button". I'd like to re-write my code to catch whatever that signal/Exception is. [I'm using OSX w/PyCharm 4.0.4]
When you stop the process after debugging it, it sends a SIGKILL signal to the interpreter.
Process finished with exit code 137
Exit codes above 128 mean that it's a 128 + a signal's number (in this case, 9, which is a SIGKILL).
You could catch SIGTERM using signal.signal(), but SIGKILL can't be caught. There's nothing you can do with it.
Well, you could set up a separate script that would monitor the first one (checking for its PID existance in the running processes, for example) and do something if the given process is terminated.
I am using requests to pull some files. I have noticed that the program seems to hang after some large number of iterations that varies from 5K to 20K. I can tell it is hanging because the folder where the results are stored has not changed in several hours. I have been trying to interrupt the process (I am using IDLE) by hitting CTRL + C to no avail. I would like to interrupt instead of killing the process because restart is easier. I have finally had to kill the process. I restart and it runs fine again until I have the same symptoms. I would like to figure out how to diagnose the problem but since I am having to kill everything I have no idea where to start.
Is there an alternate way to view what is going on or to more robustly interrupt the process?
I have been assuming that if I can interrupt without killing I can look at globals and or do some other mucking around to figure out where my code is hanging.
In case it's not too late: I've just faced the same problems and have some tips
First thing: In python most waiting apis are not interruptible (ie Thread.join(), Lock.acquire()...).
Have a look at theese pages for more informations:
http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html
http://docs.python.org/2/library/thread.html
Then if a thread is waiting on such a call, it cannot be stopped.
There is another thing to know: if a normal thread is running (or hanged) the main program will stay indefinitely untill all threads are stopped or the process is killed.
To avoid that, you can make the thread a daemon thread: Thread.daemon=True before calling Thread.start().
Second thing, to find where your program is hanged, you can launch it with a debugger but I prefer logging because logs are always there in case its to late to debug.
Try logging before and after each waiting call to see how much time your threads have been hanged. To have high quality logs, uses python logging configured with file handler, html handler or even better with a syslog handler.
I have a python script that constantly runs (it has an infinite loop), but I want it to be able to still accept input while running. It will run in the background and then at any time I want to be able to type
scriptname stop
and stop it (or something like that). That way it can call a shutdown method to save information and quit.
Currently it runs in the foreground in the terminal, and can't be stopped by a keyboard interrupt, so the only way to kill it is to close the terminal or kill python.
How can I do something like this?
Use supervisord. It exists to manage processes, and provides a command interface to start and stop them.
When supervisor kills a process, it sends SIGTERM (or any other signal you choose). So, to shutdown cleanly, you need to handle that signal.
See this question on how to handle SIGTERM: Python - Trap all signals
Processes can still listen on their own pipes for input, and send output that way.
If you are in Windows then You are at right point...
Just Rename your file: script.py to script.pyw and Use It Normally.
Your Script will run in background.
To close that script:
Go to Task Manager , click on Process Tab , look out for python , End Task.
If You need more information I am Ready to Provide to you...
I am Not Sure About Linux or Ubuntu.
Thanks.