In my code I am calling a pipeline from a tkinter gui. When the user presses the Run button the entire pipeline starts running. If certain settings are selected a toplevel of the main GUI is called which asks for an aditional file. This all works except when the cancel button or the close window X is pressed. The toplevel closes but the program keeps running. Eventually it will crash because the file is absent. calling sys.exit() isn't the solution because then the entire gui shuts down and I only want the specific toplevel to close and to stop the running file.
How do I kill a file running with exec(open(file).read()) without killing the entire program?
Honestly, you probably shouldn't use exec at all, but assuming you do, exec is still running in the same process and thread as your main program, there's no exiting it without killing the main program.
You should open it in another process, subprocess or thread. Since your exec seem to be running python code, you could simply use:
from subprocess import Popen
p = Popen(['python', filename'])
And then it runs in the background, your normal process continues, and you can kill it at any point with .
p.kill()
It gets more complicated than that if you want to give that process input or read out its output, but that's a matter for a different question. You can start here to see how to read the output: Store output of subprocess.Popen call in a string
A small example to get the output would be something like this:
from subprocess import Popen, PIPE
p = Popen(['python', filename], stdout=PIPE, stderr=PIPE)
output, errors = p.communicate()
However this will wait until the process completes its run, so maybe start all that from another thread, or find another way to get the output (perhaps to a log file)
Notice I used just 'python' in Popen, if the python executable is not in your path or has a different name, you should replace that with the full path to the executable
Related
I have a Flask application where there are links that open Jupyter notebooks. In the function that handles the url, the Jupyter notebooks are opened by a call to subprocess.Popen. Especially on Windows, after some time, the notebooks seems to be dead, i.e. they have lost connection to the kernel, and I can only get them to work again by clicking on the Flask link again. I have not noticed this behavior on a Mac. This makes me think that maybe the subprocess is getting closed. It isn't stored in a variable or anything, so once the function exits, there is no scope for it to be in. Does anyone know if this happens, and if so what happens to the process that should be running?
Here is an example of what one of these functions looks like. When you click on a link it calls the open_lecture function, constructs a cmd and Popens it. Then the function exits.
#app.route("/lecture/<label>")
def open_lecture(label):
fname = 'lectures/{}.ipynb'.format(label)
# Now open the notebook.
cmd = [JUPYTER]
cmd += [fname]
print(cmd)
subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
return redirect(url_for('hello'))
Is there a way to keep this from happening? Or a better way to programmatically open a jupyter notebook?
executed processes should continue after any Popen object has been garbage collected, but in your case you have asked for the output to be returned to Python but then you aren't processing the output. this could do different things depending on how things get cleaned up in Python and how the invoked process handles errors.
as an example, if I run:
proc = subprocess.Popen(["yes"], stdout=subprocess.PIPE)
(yes writes lots of stuff to its stdout) it freezes after writing 64KiB of output, which is what my OS/kernel (Linux 5.3) gives to pipes between processes. you can confirm by waiting a second, calling proc.terminate() then print(len(proc.communicate()[0])).
as you're invoking Jupyter, it'll probably just write status and other informational messages to stdout, so will take a while to fill this buffer which is why you're seeing a sporadic timeout.
Python's garbage collector only works on its own heap (i.e. not other processes, each Python process is independent), so invoked processes will just run as they would according to the semantics of your OS
I'm working on a tool for data entry at my job where it basically takes a report ID number, opens a PDF to that page of that report, allows you to input the information and then saves it.
I'm completely new to instantiating new processes in python; this is the first time that I've really tried to do it. so basically, I have a relevant function:
def get_report(id):
path = report_path(id)
if not path:
raise NameError
page = get_page(path, id)
proc = subprocess.Popen(["C:\Program Files (x86)\Adobe\Reader 11.0\Reader\AcroRd32.exe", "/A", "page={}".format(page),
path])
in order to open the report in Adobe Acrobat and be able to input information while the report is still open, I determined that I had to use multiprocessing. So, as a result, in the main loop of the program, where it iterates through data and gets the report ID, I have this:
for row in rows:
print 'Opening report for {}'.format(ID)
arg = ID
proc = Process(target=get_report, args=(arg,))
proc.start()
row[1] = raw_input('Enter the desired value: ')
rows.updateRow(row)
while proc.is_alive():
pass
This way, one can enter data without the program hanging on the subprocess.Popen() command. However, if it simply continues on to the next record without closing the Acrobat window that pops up, then it won't actually open the next report. Hence the while proc.is_alive():, as it gives one a chance to close the window manually. I'd like to kill the process immediately after 'enter' is hit and the value entered, so it will go on and just open the next report with even less work. I tried several different things, ways to kill processes through the pid using os.kill(); I tried killing the subprocess, killing the process itself, killing both of them, and also tried using subprocess.call() instead of Popen() to see if it made a difference.
It didn't.
What am I missing here? How do I kill the process and close the window that it opened in? Is this even possible? Like I said, I have just about 0 experience with processes in python. If I'm doing something horribly wrong, please let me know!
Thanks in advance.
To kill/terminate a subprocess, call proc.kill()/proc.terminate(). It may leave grandchildren processes running, see subprocess: deleting child processes in Windows
This way, one can enter data without the program hanging on the subprocess.Popen() command.
Popen() starts the command. It does not wait for the command to finish. There are .wait() method and convenience functions such as call()
Even if Popen(command).wait() returns i.e., if the corresponding external process has exited; it does not necessarily mean that the document is closed in the general case (the launcher app is done but the main application may persist).
i.e., the first step is to drop unnecessary multiprocessing.Process and call Popen() in the main process instead.
The second step is to make sure to start an executable that owns the opened document i.e., if it is killed the corresponding document won't stay opened: AcroRd32.exe might be already such program (test it: see whether call([r'..\AcroRd32.exe', ..]) waits for the document to be closed) or it might have a command-line switch that enables such behavior. See How do I launch a file in its default program, and then close it when the script finishes?
I tried killing the subprocess, killing the process itself, killing both of them, and also tried using subprocess.call() instead of Popen() to see if it made a difference.
It didn't.
If kill() and Popen() behave the same in your case then either you've made a mistake (they don't behave the same: you should create a minimal standalone code example with a dummy pdf that demonstrates the problem. Describe using words: what do you expect to happen (step by step) and what happens instead) or AcroRd32.exe is just a launcher app that I've described above (it just opens the document and immediately exits without waiting for the document to be closed).
Small nagging issue:
I have a python script that is working as expected, except when I select a menu option to Popen another python script:
myPath = r"c:\Python27\myScript.py"
cmd = r"c:\Python27\python.exe '{}'".format(myPath)
py_process = Popen(cmd, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
When I run that snippet (in windows), the child process is kicked-off in the background as expected, but when I attempt to exit the primary script, but leave the child process running in the background:
raise SystemExit
...an empty window "c:\python27\python.exe" remains. I've tried other EXIT methods with a similar result. Note: When I exit the primary script without running that snippet, the python window disappears as desired.
My goal is to leave no trace/window once the primary script is exited in all cases, but child process should remain running in background.
Any suggestions to accomplish this goal?
Thanks!
If you want to first communicate to the started process and then leave it alone to run further, you have a few options:
Handle SIGPIPE in your long-running process, do not die on it. Live without stdin after the launcher process exits.
Pass whatever you wanted using arguments, environment, or a temporary file.
If you want bidirectional communication, consider using a named pipe (man mkfifo) or a socket, or writing a proper server.
Make the long-running process fork after the initial bi-direcional communication phase is done.
It does not create "a completely independent process" (that what python-daemon package does). In other cases you should redirect to os.devnull child's stdin/stdout/stderr to avoid waiting for input and/or a spurious output to the terminal
Source
I know there are a bunch of similar questions on SO like this one or this one and maybe a couple more, but none of them seem to apply in my particular situation. My lack of understanding on how subprocess.Popen() works doesn't help either.
What i want to achieve is: launch a subprocess (a command line radio player) that also outputs data to the terminal and can also receive input -- wait for a while -- terminate the subprocess -- exit the shell. I am running python 2.7 on OSX 10.9
Case 1.
This launches the radio player (but audio only!), terminates the process, exits.
import subprocess
import time
p = subprocess.Popen(['/bin/bash', '-c', 'mplayer http://173.239.76.147:8090'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=False,
stderr=subprocess.STDOUT)
time.sleep(5)
p.kill()
Case 2.
This launches the radio player, outputs information like radio name, song, bitrate, etc and also accepts input. It terminates the subprocess but it never exists the shell and the terminal becomes unusable even after using 'Ctrl-C'.
p = subprocess.Popen(['/bin/bash', '-c', 'mplayer http://173.239.76.147:8090'],
shell=False)
time.sleep(5)
p.kill()
Any ideas on how to do it? I was even thinking at the possibility of opening a slave-shell for the subprocess if there is no other choice (of course it is also something that I don't have a clue about). Thanks!
It seems like mplayer uses the curses library and when kill()ing it or terminate()ing it, for some reason, it doesn't clean the library state correctly.
To restore the terminal state you can use the reset command.
Demo:
import subprocess, time
p = subprocess.Popen(['mplayer', 'http://173.239.76.147:8090'])
time.sleep(5)
p.terminate()
p.wait() # important!
subprocess.Popen(['reset']).wait()
print('Hello, World!')
In principle it should be possible to use stty sane too, but it doesn't work well for me.
As Sebastian points out, there was a missing wait() call in the above code (now added). With this wait() call and using terminate() the terminal doesn't get messed up (and so there shouldn't be any need for reset).
Without the wait() I sometimes do have problems of mixed output between the python process and mplayer.
Also, a solution specific to mplayer, as pointed out by Sebastian, is to send a q to the stdin of mplayer to quit it.
I leave the code that uses reset because it works with any program that uses the curses library, whether it correctly tears down the library or not, and thus it might be useful in other situations where a clean exit isn't possible.
What i want to achieve is: launch a subprocess (a command line radio player) that also outputs data to the terminal and can also receive input -- wait for a while -- terminate the subprocess -- exit the shell. I am running python 2.7 on OSX 10.9
On my system, mplayer accepts keyboard commands e.g., q to stop playing and quit:
#!/usr/bin/env python
import shlex
import time
from subprocess import Popen, PIPE
cmd = shlex.split("mplayer http://www.swissradio.ch/streams/6034.m3u")
p = Popen(cmd, stdin=PIPE)
time.sleep(5)
p.communicate(b'q')
It starts mplayer tuned to public domain classical; waits 5 seconds; asks mplayer to quit and waits for it to exit. The output is going to terminal (the same place where the python script's output goes).
I've also tried p.kill(), p.terminate(), p.send_signal(signal.SIGINT) (Ctrl + C). p.kill() creates the impression that the process hangs. Possible explanation: p.kill() leaves some pipes open e.g., if stdout=PIPE then your Python script might hang at p.stdout.read() i.e., it kills the parent mplayer process but there might be a child process that holds the pipes open. Nothing hangs with p.terminate(), p.send_signal(signal.SIGINT) -- mplayer exits in an orderly manner. None of the variants I've tried require reset.
how should I go about having both input from Python and keyboard? Do I need two different subprocesses and how to redirect the keyboard input to PIPE?
It would be much simpler just to drop stdin=PIPE and call p.terminate(); p.wait() instead of p.communicate(b'q').
If you want to keep stdin=PIPE then the general principle is: read from sys.stdin, write to p.stdin until timeout happens. Given that mplayer expects one letter commands, you need to be able to read one character at at time from sys.stdin. The write part is easy: p.stdin.write(c) (set bufsize=0 to avoid buffering on Python side. mplayer doesn't buffer its stdin so you don't need to worry about it).
You don't need two different subprocesses. To implement timeout, you could use threading.Timer(5, p.stdin.write, [b'q']).start() or select.select on sys.stdin with timeout.
I guess something using the good old raw_input has nothing to do with it, or?
raw_input() is not suitable for mplayer because it reads the full lines but mplayer expects one character at a time.
I'd like to execute multiple commands in a standalone application launched from a python script, using pipes. The only way I could reliably pass the commands to the stdin of the program was using Popen.communicate but it closes the program after the command gets executed. If I use Popen.stdin.write than the command executes only 1 time out of 5 or so, it does not work reliable. What am I doing wrong?
To elaborate a bit :
I have an application that listens to stdin for commands and executes them line by line.
I'd like to be able to run the application and pass various commands to it, based on the users interaction with a GUI.
This is a simple test example:
import os, string
from subprocess import Popen, PIPE
command = "anApplication"
process = Popen(command, shell=False, stderr=None, stdin=PIPE)
process.stdin.write("doSomething1\n")
process.stdin.flush()
process.stdin.write("doSomething2\n")
process.stdin.flush()
I'd expect to see the result of both commands but I don't get any response. (If I execute one of the Popen.write lines multiple times it occasionally works.)
And if I execute:
process.communicate("doSomething1")
it works perfectly but the application terminates.
If I understand your problem correctly, you want to interact (i.e. send commands and read the responses) with a console application.
If so, you may want to check an Expect-like library, like pexpect for Python: http://pexpect.sourceforge.net
It will make your life easier, because it will take care of synchronization, the problem that ddaa also describes. See also:
http://www.noah.org/wiki/Pexpect#Q:_Why_not_just_use_a_pipe_.28popen.28.29.29.3F
The real issue here is whether the application is buffering its output, and if it is whether there's anything you can do to stop it. Presumably when the user generates a command and clicks a button on your GUI you want to see the output from that command before you require the user to enter the next.
Unfortunately there's nothing you can do on the client side of subprocess.Popen to ensure that when you have passed the application a command the application is making sure that all output is flushed to the final destination. You can call flush() all you like, but if it doesn't do the same, and you can't make it, then you are doomed to looking for workarounds.
Your code in the question should work as is. If it doesn't then either your actual code is different (e.g., you might use stdout=PIPE that may change the child buffering behavior) or it might indicate a bug in the child application itself such as the read-ahead bug in Python 2 i.e., your input is sent correctly by the parent process but it is stuck in the child's internal input buffer.
The following works on my Ubuntu machine:
#!/usr/bin/env python
import time
from subprocess import Popen, PIPE
LINE_BUFFERED = 1
#NOTE: the first argument is a list
p = Popen(['cat'], bufsize=LINE_BUFFERED, stdin=PIPE,
universal_newlines=True)
with p.stdin:
for cmd in ["doSomething1\n", "doSomethingElse\n"]:
time.sleep(1) # a delay to see that the commands appear one by one
p.stdin.write(cmd)
p.stdin.flush() # use explicit flush() to workaround
# buffering bugs on some Python versions
rc = p.wait()
It sounds like your application is treating input from a pipe in a strange way. This means it won't get all of the commands you send until you close the pipe.
So the approach I would suggest is just to do this:
process.stdin.write("command1\n")
process.stdin.write("command2\n")
process.stdin.write("command3\n")
process.stdin.close()
It doesn't sound like your Python program is reading output from the application, so it shouldn't matter if you send the commands all at once like that.