I'm very new to python. I would like to know, when using subprocess.Popen do we need to close the connection or subprocess automatically closes the connection?
process = subprocess.Popen(["mysql", "-uroot", "-ppassword", "database"],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
process_out, process_err = process.communicate(file("test.sql").read())
print process_out
.communicate() call closes all the pipes (if it is what you mean by "connection") and reaps the child process. You don't need to do anything after that.
#!/usr/bin/env python3
from subprocess import Popen, PIPE
with open('test.sql', 'rb', 0) as input_file, \
Popen([cmd] + args, stdin=input_file, stdout=PIPE, stderr=PIPE) as p:
output, errors = p.communicate()
if p.returncode != 0:
raise Error
process is the object which represents the subprocess. With it, you can do everything you want, but at the end, after having processed all communication, you should .wait() for it in order not to have a zombie process for a too long time.
Only after .wait() the subprocess will be really gone from the view of the OS.
The said is valid if you handle the communication on your own. But if your circumstances allow you to use the .communicate() method and you do so, you don't have to call .wait(), as it does so for you.
Related
I am calling a shell script fom within Python, that spawns multiple child processes. I want to terminate that process and all of its children, if it did not finish after two minutes.
Is there any way I can do that with subprocess.run or do I have to go back to using Popen? Since run is blocking, I am not able to save the pid somewhere to kill the children in an extra command. A short code example:
try:
subprocess.run(["my_shell_script"], stderr=subprocess.STDOUT, timeout=120)
except subprocess.TimeoutExpired:
print("Timeout during execution")
This problem was reported as a bug to the Python developers. It seems to happen specifically when stderr or stdout is redirected.
Here is a more correct version of #Tanu's code.
import subprocess as sp
try:
proc = sp.Popen(['ls', '-l'], stdout=sp.PIPE, stderr=sp.PIPE)
outs, errs = proc.communicate(timeout=120)
except sp.TimeoutExpired:
proc.terminate()
Popen doesn't accept timeout as a parameter. It must be passed to communicate.
On Posix OSs, terminate is more gentle than kill, in that it reduces the risk of creating zombie processes.
Quoting from the docs:
subprocess.run - This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments.
Don't have to use Popen() if you don't want to. The other functions in the module, such as .call(), .Popen().
There are three 'file' streams: stdin for input, and stdout and stderr for output. The application decides what to write where; usually error and diagnostic information to stderr, the rest to stdout. To capture the output for either of these outputs, specify the subprocess.PIPE argument so that the 'stream' is redirected into your program.
To kill the child process after timeout:
import os
import signal
import subprocess
try:
proc = subprocess.Popen(["ls", "-l"], stdout=PIPE, stderr=PIPE, timeout=120)
except subprocess.TimeoutExpired:
os.kill(proc.pid, signal.SIGTERM)
I have a c program (I'm not the author) that reads from stderr. I call it using subprocess.Popen as below. Is there any way to write to stderr of the subprocess.
proc = subprocess.Popen(['./std.bin'],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Yes, maybe, but you should be aware of the irregularity of writing to the standard output or standard error output of a subprocess. The vast majority of processes only writes to these and almost none is actually trying to read (because in almost all cases there's nothing to read).
What you could try is to open a socket and supply that as the stderr argument.
What you most probably want to do is the opposite, to read from the stderr from the subprocess (the subprocesses writes, you read). That can be done by just setting it to subprocess.PIPE and then access the stderr attribute of the subprocess:
proc subprocess(['./std.bin'], stderr=subprocess.PIPE)
for l in proc.stderr:
print(l)
Note that you could specify more than one of stdin, stdout and stderr as being subprocess.PIPE. This will not mean that they will be connected to the same pipe (subprocess.PIPE is no actuall file, but just a placeholder to indicate that a pipe should be created). If you do this however you should take care to avoid deadlocks, this can for example be done by using the communicate method (you can inspect the source of the subprocess module to see what communicate does if you want to do it yourself).
If the child process reads from stderr (note: normally stderr is opened for output):
#!/usr/bin/env python
"""Read from *stderr*, write to *stdout* reversed bytes."""
import os
os.write(1, os.read(2, 512)[::-1])
then you could provide a pseudo-tty (so that all streams point to the same place), to work with the child as if it were a normal subprocess:
#!/usr/bin/env python
import sys
import pexpect # $ pip install pexpect
child = pexpect.spawnu(sys.executable, ['child.py'])
child.sendline('abc') # write to the child
child.expect(pexpect.EOF)
print(repr(child.before))
child.close()
Output
u'abc\r\n\r\ncba'
You could also use subprocess + pty.openpty() instead pexpect.
Or you could write a code specific to the weird stderr behavior:
#!/usr/bin/env python
import os
import sys
from subprocess import Popen, PIPE
r, w = os.pipe()
p = Popen([sys.executable, 'child.py'], stderr=r, stdout=PIPE,
universal_newlines=True)
os.close(r)
os.write(w, b'abc') # write to subprocess' stderr
os.close(w)
print(repr(p.communicate()[0]))
Output
'cba'
for line in proc.stderr:
sys.stdout.write(line)
This is write the stderr of the subprocess. Hope it answers your question.
We are having some problems with the dreaded "too many open files" on our Ubuntu Linux machine rrunning a python Twisted application. In many places in our program, we are using subprocess Popen, something like this:
Popen('ifconfig ' + iface, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = process.stdout.read()
while in other places we use subprocess communicate:
process = subprocess.Popen(['/usr/bin/env', 'python', self._get_script_path(script_name)],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
close_fds=True)
out, err = process.communicate(data)
What exactly do I need to do in both cases in order to close any open file descriptors? Python documentation is not clear on this. From what I gather (which could be wrong) both communicate() and wait() will indeed clean up any open fds on their own. But what about Popen? Do I need to close stdin, stdout, and stderr explicitly after calling Popen if I don't call communicate or wait?
According to this source for the subprocess module (link) if you call communicate you should not need to close the stdout and stderr pipes.
Otherwise I would try:
process.stdout.close()
process.stderr.close()
after you are done using the process object.
For instance, when you call .read() directly:
output = process.stdout.read()
process.stdout.close()
Look in the above module source for how communicate() is defined and you'll see that it closes each pipe after it reads from it, so that is what you should also do.
If you're using Twisted, don't use subprocess. If you were using spawnProcess instead, you wouldn't need to deal with annoying resource-management problems like this.
If I spawn a new subprocess in python with a given command (let's say I start the python interpreter with the python command), how can I send new data to the process (via STDIN)?
Use the standard subprocess module. You use subprocess.Popen() to start the process, and it will run in the background (i.e. at the same time as your Python program). When you call Popen(), you probably want to set the stdin, stdout and stderr parameters to subprocess.PIPE. Then you can use the stdin, stdout and stderr fields on the returned object to write and read data.
Untested example code:
from subprocess import Popen, PIPE
# Run "cat", which is a simple Linux program that prints it's input.
process = Popen(['/bin/cat'], stdin=PIPE, stdout=PIPE)
process.stdin.write(b'Hello\n')
process.stdin.flush()
print(repr(process.stdout.readline())) # Should print 'Hello\n'
process.stdin.write(b'World\n')
process.stdin.flush()
print(repr(process.stdout.readline())) # Should print 'World\n'
# "cat" will exit when you close stdin. (Not all programs do this!)
process.stdin.close()
print('Waiting for cat to exit')
process.wait()
print('cat finished with return code %d' % process.returncode)
Don't.
If you want to send commands to a subprocess, create a pty and then fork the subprocess with one end of the pty attached to its STDIN.
Here is a snippet from some of my code:
RNULL = open('/dev/null', 'r')
WNULL = open('/dev/null', 'w')
master, slave = pty.openpty()
print parsedCmd
self.subp = Popen(parsedCmd, shell=False, stdin=RNULL,
stdout=WNULL, stderr=slave)
In this code, the pty is attached to stderr because it receives error messages rather than sending commands, but the principle is the same.
I'm using the following to execute a process and hide its output from Python. It's in a loop though, and I need a way to block until the sub process has terminated before moving to the next iteration.
subprocess.Popen(["scanx", "--udp", host], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Use subprocess.call(). From the docs:
subprocess.call(*popenargs, **kwargs)
Run command with arguments. Wait for command to complete, then
return the returncode attribute.
The arguments are the same as for the
Popen constructor.
Edit:
subprocess.call() uses wait(), and wait() is vulnerable to deadlocks (as Tommy Herbert pointed out). From the docs:
Warning: This will deadlock if the
child process generates enough output
to a stdout or stderr pipe such that
it blocks waiting for the OS pipe
buffer to accept more data. Use
communicate() to avoid that.
So if your command generates a lot of output, use communicate() instead:
p = subprocess.Popen(
["scanx", "--udp", host],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate()
If you don't need output at all you can pass devnull to stdout and stderr. I don't know if this can make a difference but pass a bufsize. Using devnull now subprocess.call doesn't suffer of deadlock anymore
import os
import subprocess
null = open(os.devnull, 'w')
subprocess.call(['ls', '-lR'], bufsize=4096, stdout=null, stderr=null)