Develop Reference

Opens a process with Popen cant close it ( need to run Ros command in cmd)

I need to save some image files from my simulation at different times. So my idea was to open a subprocess save some image files and close it .
import subprocess
cmd = "rosrun pcl_ros pointcloud_to_pcd input:=camera/depth/points"
proc = subprocess.Popen(cmd, shell=True)
When it comes to closing I tried different things:
import os
import signal
import subprocess
cmd = "rosrun pcl_ros pointcloud_to_pcd input:=camera/depth/points"
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(os.getpgid(pro.pid), signal.SIGTERM)
command did not execute , so it doesn't work for me. I also tried a solution with psutil and it didn't work neither...

you probably don't need shell=True here, which is the cause of your problems. I suspect that when you kill the process group in your second snippet, the shell process is killed before the process you want to run has a chance to start...
Try to pass the parameters as a list of strings (so you don't need shell=True), wait a bit, and use terminate on the Popen object. You don't need process group, or psutil to kill the process & its children, just plain old terminate() on the process object does the trick.
cmd = ["rosrun","pcl_ros","pointcloud_to_pcd","input:=camera/depth/points"]
proc = subprocess.Popen(cmd)
time.sleep(1) # maybe needed to wait the process to do something useful
proc.terminate()
Note that proc.terminate() tries to exit gracefully where proc.kill() would have just killed the process (there's a difference under Un*x systems, not under Windows)
Another plea for "do not use shell=True unless forced at gunpoint".

subprocess.Popen doesn't work with shell=False

I try to run simple script in windows in the same shell.
When I run
subprocess.call(["python.exe", "a.py"], shell=False)
It works fine.
But when I run
subprocess.Popen(["python.exe", "a.py"], shell=False)
It opens new shell and the shell=false has no affect.
a.py just print message to the screen.

First calling Popen with shell=False doesn't mean that the underlying python won't try to open a window/console. It's just that the current python instance executes python.exe directly and not in a system shell (cmd or sh).
Second, Popen returns a handle on the process, and you have to perform a wait() on this handle for it to end properly or you could generate a defunct process (depending on the platform you're running on). I suggest that you try
p = subprocess.Popen(["python.exe", "a.py"], shell=False)
return_code = p.wait()
to wait for process termination and get return code.
Note that Popen is a very bad way to run processes in background. The best way would be to use a separate thread
import subprocess
import threading
def run_it():
subprocess.call(["python.exe", "a.py"], shell=False)
t = threading.Thread(target=run_it)
t.start()
# do your stuff
# in the end
t.join()

Terminate a gnome-terminal opened with subprocess

Using subprocess and the command 'gnome-terminal -e bash' I can open up a gnome-terminal as desired (and have it stick around). This is done with either
p=subprocess.Popen(['gnome-terminal', '-e', 'bash'])
or
p=subprocess.Popen(['gnome-terminal -e bash'], shell=True)
but I cannot close the terminal using p.terminate() or p.kill(). From what I understand, this is a little trickier when using shell=True but I did not expect to run into problems otherwise.

To terminate a terminal and its children (in the same process group):
#!/usr/bin/env python
import os
import signal
import subprocess
p = subprocess.Popen(['gnome-terminal', '--disable-factory', '-e', 'bash'],
preexec_fn=os.setpgrp)
# do something here...
os.killpg(p.pid, signal.SIGINT)
--disable-factory is used to avoid re-using an active terminal so that we can kill newly created terminal via the subprocess handle
os.setpgrp puts gnome-terminal in its own process group so that os.killpg() could be used to send signal to this group

You should be able to do this workaround:
get the process id
kill the process
Working Solution: Close gnome-terminal-server
As suggested by #j-f-sebastian in the comment, gnome-terminal
just sends the request (to gnome-terminal-server) to start a new terminal and exits immediately -- there is nothing to kill the process is already dead (and newly created processes are not descendants:  the new bash process is a child of gnome-terminal-server, not gnome-terminal).
import subprocess
import os, signal
import time
p=subprocess.Popen(['gnome-terminal -e bash'], stdout=subprocess.PIPE, shell=True, preexec_fn=os.setsid)
print "this is going to be closed in 3 sec"
time.sleep(3)
# this line returns the list of bash instances pid as string
bash_pids = subprocess.check_output(["pidof", "bash"])
# I get the last instance opened
pid_to_kill = bash_pids.split(" ")[0]
os.kill(int(pid_to_kill), signal.SIGTERM)
My solution is following this logic:
run gnome-terminal
get the latest bash instance opened process id
kill this process id
Broken solutions
These solutions might work in simpler cases:
Solution 1
import subprocess
import os, signal
p=subprocess.Popen(['gnome-terminal -e bash'], shell=True)
p_pid = p.pid # get the process id
os.kill(p_pid, signal.SIGKILL)
In order to choose the appropriate method of signal to pass instead of SIGKILL you can refer the signal documentation. E.g.
On Windows, signal() can only be called with SIGABRT, SIGFPE, SIGILL, SIGINT, SIGSEGV, or SIGTERM
For Unix you have a quite extensive list of method to call.
To have a better overview about os.kill, you can refer its documentation.
Solution 2
An alternative method useful for Unix could be:
import subprocess
import os, signal
p=subprocess.Popen(['gnome-terminal -e bash'], stdout=subprocess.PIPE, shell=True, preexec_fn=os.setsid)
os.killpg(os.getpgid(p.pid), signal.SIGTERM)
It seems that your process is opening child process that prevent the parent to be close. Adding a session id to your parent process, you should be able to fix it.
Solution 3
import subprocess, psutil
def kill(p_pid):
process = psutil.Process(p_pid)
for proc in process.get_children(recursive=True):
proc.kill()
process.kill()
p = subprocess.Popen(['gnome-terminal -e bash'], shell=True)
try:
p.wait(timeout=3)
except subprocess.TimeoutExpired:
kill(p.pid)
This solution requires psutil.
Solution 4
According to askubuntu, it seems that the best way to close a gnome terminal instance would be to execute a bash command like:
killall -s {signal} gnome-terminal
where {signal} simulates Alt + F4.
You can try to do it using [pexpect]:
p = pexpect.spawn(your_cmd_here)
p.send('^F4')

I wanted to add this snippet for anyone who is running on Linux Ubuntu and trying to open a subprocess, run a script, and terminate it after a time.wait().
I found a litany of solutions that would open a window, but not close it. Or a solution would open a window, and close it, but wouldn't run the script inside the terminal.
There was no exact answer so I had to hack together several solutions, as I am a novice when it comes t subprocess/shell.
This snippet was able to open a subprocess, run the script, and when 10 seconds had passed the subprocess was terminated. Again, this was built ofn the shoulders of giants. I hope this saves someone time; cheers.
import os
import signal
import subprocess
import time
command = 'python3 Rmonitor.py'
p = subprocess.Popen(['gnome-terminal','--disable-factory', '--', 'bash', '-c', command],preexec_fn=os.setpgrp)
time.sleep(10)
os.killpg(p.pid, signal.SIGINT)

python - subprocess.Popen().pid return the pid of the parent script

I am trying to run a Python script from another Python script, and getting its pid so I can kill it later.
I tried subprocess.Popen() with argument shell=True', but thepidattribute returns thepid` of the parent script, so when I try to kill the subprocess, it kills the parent.
Here is my code:
proc = subprocess.Popen(" python ./script.py", shell=True)
pid_ = proc.pid
.
.
.
# later in my code
os.system('kill -9 %s'%pid_)
#IT KILLS THE PARENT :(

shell=True starts a new shell process. proc.pid is the pid of that shell process. kill -9 kills the shell process making the grandchild python process into an orphan.
If the grandchild python script can spawn its own child processes and you want to kill the whole process tree then see How to terminate a python subprocess launched with shell=True:
#!/usr/bin/env python
import os
import signal
import subprocess
proc = subprocess.Popen("python script.py", shell=True, preexec_fn=os.setsid)
# ...
os.killpg(proc.pid, signal.SIGTERM)
If script.py does not spawn any processes then use #icktoofay suggestion: drop shell=True, use a list argument, and call proc.terminate() or proc.kill() -- the latter always works eventually:
#!/usr/bin/env python
import subprocess
proc = subprocess.Popen(["python", "script.py"])
# ...
proc.terminate()
If you want to run your parent script from a different directory; you might need get_script_dir() function.
Consider importing the python module and running its functions, using its object (perhaps via multiprocessing) instead of running it as a script. Here's code example that demonstrates get_script_dir() and multiprocessing usage.

So run it directly without a shell:
proc = subprocess.Popen(['python', './script.py'])
By the way, you may want to consider changing the hardcoded 'python' to sys.executable. Also, you can use proc.kill() to kill the process rather than extracting the PID and using that; furthermore, even if you did need to kill by PID, you could use os.kill to kill the process rather than spawning another command.

How do I fork a new process with independent stdout, stderr, and stdin?

I have read most of the related questions about subprocess and os.fork(), including all the discussions about the double forking trick. However, none of the those solutions appear to work correctly for my scenario.
I want to fork a new process and allow the parent to terminate (normally) without screwing up the child's stdin, stdout, and stderr and without killing the child.
My first attempt was to use subprocess.Popen().
#!/usr/bin/python
from subprocess import call,Popen
Popen("/bin/bash", shell=True)
call("echo Hello > /tmp/FooBar", shell=True)
This fails because the child process is killed once the parent is exits. I am aware of creationflags but that is Windows-specific and I am running on Linux. Note that the above code works beautifully if we simply keep the parent process alive by adding an infinite loop to the end of it. This is undesirable because the parent is already finished with its job and there's no real reason for it to stick around.
The second attempt was to use os.fork().
#!/usr/bin/python
from subprocess import call
from os import fork
try:
pid = fork()
if pid > 0:
pass
else: # child process will start interactive process
call("/bin/bash", shell=True)
except:
print "Forking failed!"
call("echo Hello > /tmp/FooBar", shell=True)
Here, the child process no longer dies with the parent, but after the parent's death the child can no longer read input and write output.
Thus, I want to know how I fork a new process with utterly independent stdout, stderr, and stdin. Independence means that the parent process can terminate (normally), and the child process (whether it is bash or tmux or any other interactive program) behaves exactly as though the parent program had not terminated. To be even more precise, consider the following variation of the original program.
#!/usr/bin/python
from subprocess import call,Popen
Popen("/bin/bash", shell=True)
call("echo Hello > /tmp/FooBar", shell=True)
while True:
pass
The above code has all the behaviors I seek, but it keeps the Python process alive artificially. I am trying to achieve this exact behavior, without the Python process being alive.
Caveat: I am running these applications over ssh, so spawning a new GUI window is not a viable solution.
Desired Behavior:
I run the python code.
I get a shiny new bash shell that works exactly like the bash shell I started with.
The file /tmp/FooBar is created.
The original Python script finishes.
I continue on with my shiny new bash shell, and the output of ps aux | grep python does not include python script I just ran.

Your first example has an extra unnecessary call to Popen(), as the call convenience function will just execute its commands in a shell and exit, so it would work if you just ran:
from subprocess import call, Popen
call("echo Hello > /tmp/FooBar", shell=True)
However if you want to send a series of commands to a shell then the shell needs to opened with stdin attached to pipe so it doesn't get mixed up with the parents stdin (which is effectively what you're asking for in terms of obtaining independent stdio):
p = Popen("/bin/bash", shell=True, stdin = subprocess.PIPE)
p.stdin.write("echo Hello > /tmp/FooBar\n")
p.stdin.write("date >> /tmp/FooBar\n")
p.terminate()
If you want to also control the output then you redirect stdout and stderr to PIPE (i.e. stdout = subprocess.PIPE, stderr = subprocess.PIPE) and then call p.stdout.read() to obtain output as needed.
To allow the process to continue running after exiting python one can add & operator to the end of the command so it continues running in the background e.g.:
call("nc -l 2000 > /tmp/nc < &",shell=True)
To have a process run so that both stdin and stdout and still connected one can use shell redirects. To maintain access to stdin one can create a named pipe using mkfifo e.g:
call("mkfifo /tmp/pipe",shell=True)
call("tail -f /tmp/pipe > /tmp/out &",shell=True)
To provide input stdin one just send data to the pipe .e.g. from the shell:
$ echo 'test' > /tmp/pipe

I recently encountered this problem, I found a solution that might help,
using creationflags to tell Popen that the child process should not inherit the parent process console, and thus has it's own stdout, stdin and stderr as if it was a parent process.
subprocess.Popen("/bin/bash", creationflags= subprocess.CREATE_NEW_CONSOLE)
Popen supports creationflags keyword argument according to docs:
creationflags, if given, can be one or more of the following flags:
CREATE_NEW_CONSOLE
CREATE_NEW_PROCESS_GROUP
ABOVE_NORMAL_PRIORITY_CLASS
BELOW_NORMAL_PRIORITY_CLASS
HIGH_PRIORITY_CLASS
IDLE_PRIORITY_CLASS
NORMAL_PRIORITY_CLASS
REALTIME_PRIORITY_CLASS
CREATE_NO_WINDOW
DETACHED_PROCESS
CREATE_DEFAULT_ERROR_MODE
CREATE_BREAKAWAY_FROM_JOB
The ones that you are interested in are DETACHED_PROCESS and CREATE_NEW_CONSOLE, I've used CREATE_NEW_CONSOLE, what it does is it spawns a new process as if it was a parent, I used Python3.5 Windows, in Python 3.7 DETACHED_PROCESS was added and it's documented to do the same as CREATE_NEW_CONSOLE.