My application creates subprocesses. Usually, these processeses run and terminate without any problems. However, sometimes, they crash.
I am currently using the python subprocess module to create these subprocesses. I check if a subprocess crashed by invoking the Popen.poll() method. Unfortunately, since my debugger is activated at the time of a crash, polling doesn't return the expected output.
I'd like to be able to see the debugging window(not terminate it) and still be able to detect if a process is crashed in the python code.
Is there a way to do this?
When your debugger opens, the process isn't finished yet - and subprocess only knows if a process is running or finished. So no, there is not a way to do this via subprocess.
I found a workaround for this problem. I used the solution given in another question Can the "Application Error" dialog box be disabled?
Items of consideration:
subprocess.check_output() for your child processes return codes
psutil for process & child analysis (and much more)
threading library, to monitor these child states in your script as well once you've decided how you want to handle the crashing, if desired
import psutil
myprocess = psutil.Process(process_id) # you can find your process id in various ways of your choosing
for child in myprocess.children():
print("Status of child process is: {0}".format(child.status()))
You can also use the threading library to load your subprocess into a separate thread, and then perform the above psutil analyses concurrently with your other process.
If you find more, let me know, it's no coincidence I've found this post.
Related
I am running some Python code using a SLURM script on a remote server accessed through SSH. At some point, issues related to licenses on the SLURM platform may happen, generating errors in Python and ending the subprocess. I want to use try-except to let the Python subprocess wait until the issue is fixed, after that it can keep running from where it stopped.
What are some smart implementations for that?
My most obvious solution is just keeping Python inside a loop if the error occurs and letting it read a file every X seconds, when I finally fix the error and want it to keep running from where it stopped, I would write something on the file and break the loop. I wonder if there is a smarter way to provide input to the Python subprocess while it is running through the SLURM script.
One idea might be to add a signal handler for signal USR1 to your Python script like this.
In the signal handler function, you can set a global variable or send a message or set a threading.Event that the main process is waiting on.
Then you can signal the process with:
kill -USR1 <PID>
or with the Python os.kill() equivalent.
Though I do have to agree there is something to be said for the simplicity of your process doing:
touch /tmp/blocked.$$
and your program waiting in a loop with a 1s sleep for that file to be removed. This way you can tell which process id is blocked.
I am working from a windows platform. In my python script, I can make a call to an external program in the following way:
os.system("C:\mainfolder\menu.exe C:\others\file1.inp C:\others\file2.inp")
os.popen("C:\mainfolder\menu.exe C:\others\file1.inp C:\others\file2.inp")
subprocess.call(["C:\mainfolder\menu.exe","C:\others\file1.inp" "C:\others\file2.inp"])
where:
menu.exe: is my external program.
file1 and file2: are input files to my external program.
All the above works fine. Now that my external program has finished successfully, I need to totally close it along with all the windows that are left opened by it. I have gone through lots of other posts, python documentation, etc and found commands as for example:
os.system("taskkill /im C:\mainfolder\menu.exe")
os.kill(proc.pid,9)
child.kill()
But they did not work. I spent a lot of time trying to find something that worked for me, until I realised that no matter which commands I type after, they will not be read as the computer does not know that my external program has finished. That is the reason why I can easily terminate the program from the command line anytime just by typing taskkill /im menu.exe, but not from python.
Does anybody know how to sort this out?, should I include something else when I make the call to my external program?
Here's some example code, how to detect if a program opens a window. All you need to know is the title of the message box, that menu.exe opens, when it is finished:
import subprocess
import win32gui
import time
def enumHandler(hwnd, lParam):
if win32gui.IsWindowVisible(hwnd):
if 'Menu.exe Finished' in win32gui.GetWindowText(hwnd):
proc.kill()
proc = subprocess.Popen(["C:\mainfolder\menu.exe","C:\others\file1.inp" "C:\others\file2.inp"])
while proc.poll() is None:
win32gui.EnumWindows(enumHandler, None)
time.sleep(1)
If you want to have a process end immediately, i.e., wait for it to end, this is a blocking call, and os.system() normally waits, discussed here as well as .communicate[0] using subprocess there.
If you want to end a process later in your program, an asynchronous, non-blocking process, perhaps get its pid and depending on whether shell=True or not that will either be the pid of the spawned shell or of the child process.
That pid can be used to end it either immediately by using psutil or os, or wait until it ends using little cpu time, though then other tasks can be done while waiting, or threads could be used.
It might be a bit late to post my findings to this question as I asked it some months back but it may still be helpful for other readers.
With the help of the people who tried answering my question, especially Daniel's answer, I found out a solution for my problem. Not sure if it is the best one but I got what I was needing.
Basically instead of looking for the word "finished" on my pop up window, I looked for the results that my external program generates. If these results have been generated, it means that the program has finished so I then kill the process:
proc=subprocess.Popen(["C:\mainfolder\menu.exe","C:\others\file1.inp" "C:\others\file2.inp"])
while proc.poll() is None:
if os.path.exists("D:\\Results_folder\\Solution.txt"):
time.sleep(10)
os.system('taskkill /im menu.exe')
I have a unittest that does a bunch of stuff in several different threads. When I stop everything in the tearDown method, somehow something is still running. And by running I mean sleeping. I ran the top command on the python process (Ubuntu 12.04), which told me that the process was sleeping.
Now I have tried using pdb to figure out what is going on, e.g. by putting set_trace() at the end of tearDown. But that tells me nothing. I suspect this is because some other thread has started sleeping earlier and is therefore not accessed anymore at this point.
Is there any tool or method I can use to track down the cause of my non-stopping process?
EDIT
Using ps -Tp <#Process> -o wchan I now know that 4 threads are still running, of which three waiting on futex_wait_queue_me and one on unix_stream_data_wait. Since I had a subprocess previously, which I killed with os.kill(pid, signal.SIGKILL), I suspect that the Pipe connection is somehow still waiting for that process. Perhaps the fast mutexes are waiting for that as well.
Is there anyway I could further reduce the search space?
If you are working under Linux then you should be able to use 'ps -eLf' to get a list of all active processes and threads. Assuming your have given your threads good names at creation it should be easy to see what is still running.
I believe under windows you can get a tool to do something similar - see http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
N.B. I have not used the windows tool this myself
Also from within Python you can use the psutil package (https://pypi.python.org/pypi/psutil/) to get similar infomration
I want to invoke an external GUI application from a python script which will be triggered upon some file upload to the server.
I would like the process to be launched and kept running whereas the python script should continue and eventually finish its job and quit. I have tried different options but none proved successful.
Right now the script expects the application to be closed before script exits and sends the response.
I tried Subprocess, Popen, os.System, Spawnl, Spawnlp in the main thread as well by calling these API's in a separate thread. There are lot of questions asked in this regard in stackoverflow and other forums. But I couldn't get the exact solution for this yet.
Appreciate any help.
had exactly the same problem and took me friggin ages to find it, but here is your answer:
import win32api
win32api.ShellExecute(0, "open", "python.exe", 'blah.py', '', 1)
This guarantees you an independent process - even after you exit the calling python program, this will continue to work.
Python have been really bumpy for me, because the last time I created a GUI client, the client seems to hang when spawning a process, calling a shell script, and calling outside application.
This have been my major problem with Python since then, and now I'm in a new project, can someone give me pointers, and a word of advice in order for my GUI python application to still be interactive when spawning another process?
Simplest (not necessarily "best" in an abstract sense): spawn the subprocess in a separate thread, communicating results back to the main thread via a Queue.Queue instance -- the main thread must periodically check that queue to see if the results have arrived yet, but periodic polling isn't hard to arrange in any event loop.
Your main GUI thread will freeze if you spawn off a process and wait for it to completely. Often, you can simply use subprocess and poll it now and then for completion rather than waiting for it to finish. This will keep your GUI from freezing.