I'm completely confused between subprocess.call() , subprocess.Popen(), subprocess.check_call().
Which is blocking and which is not ?
What I mean to say is if I use subprocess.Popen() whether the parent process waits for the child process to return/exit before it keep on its execution.
How does shell=True affect these calls?
Popen is nonblocking. call and check_call are blocking.
You can make the Popen instance block by calling its wait or communicate method.
If you look in the source code, you'll see call calls Popen(...).wait(), which is why it is blocking.
check_call calls call, which is why it blocks as well.
Strictly speaking, shell=True is orthogonal to the issue of blocking. However, shell=True causes Python to exec a shell and then run the command in the shell. If you use a blocking call, the call will return when the shell finishes. Since the shell may spawn a subprocess to run the command, the shell may finish before the spawned subprocess. For example,
import subprocess
import time
proc = subprocess.Popen('ls -lRa /', shell=True)
time.sleep(3)
proc.terminate()
proc.wait()
Here two processes are spawned: Popen spawns one subprocess running the shell. The shell in turn spawns a subprocess running ls. proc.terminate() kills the shell, but the subprocess running ls remains. (That is manifested by copious output, even after the python script has ended. Be prepared to kill the ls with pkill ls.)
Related
I have a simple question. I have tried to search for a solution but there are no answers which would explain what I need.
The question is:
How do I start a nohup command from Python? Basically the idea is, that I have a Python script which prepares my environment and I need it to launch multiple scripts with nohup commands. How do I start a nohup command like nohup python3 my_script.py & from within a running Python script to have that nohup command running even after I log out?
Thank you
You do not need nohup -- not even in shell, and even less so in Python. It does the following things:
Configures the HUP signal to be ignored (rarely relevant: if a process has no handles on a TTY it isn't going to be notified when that TTY exits regardless; the shell only propagates signals to children in interactive mode, not when running scripts).
If stdout is a terminal, redirects it to nohup.out
If stderr is a terminal, redirects it to wherever stdout was already redirected.
Redirects stdin to /dev/null
That's it. There's no reason to use nohup to do any of those things; they're all trivial to do without it:
</dev/null redirects stdin from /dev/null in shell; stdin=subprocess.DEVNULL does so in Python.
>nohup.out redirects stdout to nohup.out in shell; stdout=open('nohup.out', 'w') does so in Python.
2>&1 makes stderr go to the same place as stdout in shell; stderr=subprocess.STDOUT does so in Python.
Because your process isn't attached to the terminal by virtue of the above redirections, it won't implicitly get a HUP when that terminal closes. If you're worried about a signal being sent to the parent's entire process group, however, you can avoid that by splitting off the child into a separate one:
The subprocess.Popen argument start_new_session=True splits the child process into a separate group from the parent in Python, so a parent sent to the process group of the parent as a whole will not be received by the child.
Adding a preexec_fn with signal.signal(signal.SIGHUP, signal.SIG_IGN) is even more explicit that the child should by default ignore a SIGHUP even if one is received.
Putting this all together might look like (if you really do want logs to go to a file named nohup.out -- I would suggest picking a better name):
import subprocess, signal
subprocess.Popen(['python3', 'my_script.py'],
stdin=subprocess.DEVNULL,
stdout=open('nohup.out', 'w'),
stderr=subprocess.STDOUT,
start_new_session=True,
preexec_fn=(lambda: signal.signal(signal.SIGHUP, signal.SIG_IGN)))
The code below to kill a process opened using subprocess.Popen() works on Linux.
import os
import signal
import subprocess
# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(os.getpgid(pro.pid), signal.SIGTERM) # Send the signal to all the process groups
Unfortunately, it does not work on Windows. The process refused to be killed. How to modify the code such that the process can be killed on Windows?
I am using Windows 10, Python 3.8 on Anaconda.
USE subprocess.Popen.terminate() TO TERMINATE A SUBPROCESS
I am trying to use Python to send a SIGINT signal to a process created through subprocess.Popen. I need to run test.bash and have it open in a new window, then wait 4 seconds, send SIGINT (the equivalent of using Ctrl-C) so I am using the following code:
command = "xterm -e bash test.bash"
process = subprocess.Popen(command.split())
time.sleep(4)
os.kill(process.pid, signal.SIGINT)
I have "test.bash" set up to trap the SIGINT signal, and run a cleanup command. This works when I run "test.bash" straight from the command line. But when I send it through xterm -e bash test.bash in Python it instantly terminates the process entirely and does not handle the SIGINT signal at all.
I've tried some things I've read online about killing the process group, but that's not doing anything either. I tried replacing os.kill(process.pid, signal.SIGINT) with process.send_signal(signal.SIGINT) to no avail. Any ideas?
You probably need to add the -hold option to xterm, i.e. xterm -hold -e... to prevent it from closing after the bash script completes since you're using sleep in python rather than the shell.
If you're unable to get the right PID, replacing the second line with this might work:
cmd = "pgrep -f 'xterm -e bash test.bash'"
id = subprocess.check_output(cmd, shell=True)
Then use id instead of process.pid
In Python, external processes can be started easily using the subprocess module. For instance on Windows:
command = 'external_app'
my_process = subprocess.Popen(
command,
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,
shell=True,
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE,
universal_newlines=True)
To kill the process, we can run:
os.kill(my_process.pid, signal.CTRL_BREAK_EVENT)
This works fine using the command-line interpreter of Python (python.exe). But if I like to start and stop processes from within a graphical Python application without a command-line window using pythonw.exe the problem occurs that I can’t stop the process with os.kill anymore.
How can I kill an external process on Windows with pythonw.exe?
Killing subprocess created with shell=True passed to subprocess.Popen's constructor under Linux kills only shell and not the process created by shell (see How to terminate a python subprocess launched with shell=True). However python -c "import subprocess; subprocess.Popen(['ping', '-t', 'google.com'], shell=True).terminate()" run under Windows Vista x64 SP3 and Python 2.7.3 64bit kills the ping process. Under Windows subprocess.Popen.terminate() calls TerminateProcess. However documentation of this function states
Terminates the specified process and all of its threads.
There is no mention about terminating whole process tree. What am I missing?
I think this is only with the one-liner you give, and my observations suggest that the ping does not even get started. If you run as a script (Windows 7):
import subprocess
proc = subprocess.Popen(['ping', '-t', 'google.com'], shell=True)
raw_input("<RETURN> to terminate")
proc.terminate()
raw_input("<RETURN> to end")
Then the proc.terminate() only terminates the shell, it does not terminate the ping!
However, if you set shell=False then it behaves as expected - it terminates the ping. Same behaviour on Python 2.7 and 3.2.
Edit: I tried this code as a one-liner as well, and got the same results as the questioner. I hate sleep hacks, but this works:
python -c "import subprocess,time;proc = subprocess.Popen(['ping','-t', 'google.com'], shell=True);time.sleep(1);proc.terminate()"