Develop Reference

Python with tcpdump in a subprocess: how to close subprocess properly?

I have a Python script to capture network traffic with tcpdump in a subprocess:
p = subprocess.Popen(['tcpdump', '-I', '-i', 'en1',
'-w', 'cap.pcap'], stdout=subprocess.PIPE)
time.sleep(10)
p.kill()
When this script completes its work, I'm trying to open output .pcap file in Wireshark and getting this error:
"The capture file appears to have been cut short in the middle of a packet."
What solution could be applied for "proper" closing of tcpdump's subprocess?

Instead of p.kill(), you can use p.send_signal(subprocess.signal.SIGTERM) to send a terminate signal rather than a kill (p.terminate() does the same).
The Popen docs describe the send_signal() command. The documentation on signals is a bit weak, but a dir(subprocess.signal) will list all the signals you may send to the process, but terminate should allow it some time to clean up.

Found working solution:
I've changed p.kill() to p.terminate().
After this change the subprocess is "properly" finished (output of tcpdump subprocess with statistics available in console) and output .pcap file not damaged.

I had the same problem about closing subprocesses. This thread really helped, so thanks, especially to https://stackoverflow.com/users/3583715/rkh. My solution:
Before:
output = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
message = output.stdout.read()
output.stdout.close()
After reading the Popen docs:
output = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
message = output.stdout.read()
output.TerminateProcess()
For some reason, calling output.kill(), and/or output.terminate() or sending output.send_signal(subprocess.signal.SIGTERM) didn't work, but output.TerminateProcess() did.

Python: subprocess Popen requires to be joined?

i have a daemon running which gets command and executes it by:
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
I never do anything with it afterwards, no wait(), no communicate()
is that okay to do so?
or are joining of the process required?
is there a similar thing to threading.deamon=True for subprocess.Popen?

Since you set stdout=subprocess.PIPE, stderr=subprocess.PIPE, and if you want to get the stdout, you can do this:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for line in p.stdout:
# do something here
#...
As you say the cmd is executed as a daemon, I think daemon process should not be run in this way. Since once your python process exit with error, the daemon process would exited and your data will lost.
And if you run other process by subprocess.Popen(), such as mysql, linux comands etc, it is ok to do so. But you should know the process status(the return code), you should check the status of the process by call poll() or wait(). For more information, see docs of subproces.Popen.

Subprocess.Popen stdin in new console

I want to execute a python subprocess in a new console. Once started, I want the user to be able to answer questions asked by this new process on stdin.
I tried the following code:
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd, creationflags=subprocess.CREATE_NEW_CONSOLE)
(o, e) = p.communicate()
As soon as the subprocess asks for input on stdin the following error message is displayed:
EOFError: EOF when reading a line
Is it the good way to achieve this ?

As i'm not really interested in the stdout/stderr redirection, i tried this way:
subprocess.Popen(cmd, cwd=cwd, creationflags=subprocess.CREATE_NEW_CONSOLE)
It works fine now. I guess that it's not compatible to redirect standard input/outputs and to create a new console.

Python: spawn subprocess with several requirements

Using python 2.6 or 2.7, I need to spawn a subprocess:
it must be detached
it's output must be redirected
the spawning python process must print the subprocess's PID, and then exit.
I've gone through the various modules (and various Stackoverflow posts), and it seems all of them conflict with one or more of these requirements. E.g. os.system() = no pid; subprocess.* = either no redirect or no detach.

By detached I assume you mean you want your script to continue running after you start the subprocess, correct? If so, I believe you'll have to fork, start the subprocess in the child process and capture it's output there.
import os
import subprocess
cmd = 'ls'
if os.fork() == 0:
process = subprocess.Popen(cmd, shell=True, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
print "subprocess pid: %d" % process.pid
stdout = process.communicate()
print stdout
else:
print 'parent...'

Python - Execute Process -> Block till it exits & Suppress Output

I'm using the following to execute a process and hide its output from Python. It's in a loop though, and I need a way to block until the sub process has terminated before moving to the next iteration.
subprocess.Popen(["scanx", "--udp", host], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Use subprocess.call(). From the docs:
subprocess.call(*popenargs, **kwargs)
Run command with arguments. Wait for command to complete, then
return the returncode attribute.
The arguments are the same as for the
Popen constructor.
Edit:
subprocess.call() uses wait(), and wait() is vulnerable to deadlocks (as Tommy Herbert pointed out). From the docs:
Warning: This will deadlock if the
child process generates enough output
to a stdout or stderr pipe such that
it blocks waiting for the OS pipe
buffer to accept more data. Use
communicate() to avoid that.
So if your command generates a lot of output, use communicate() instead:
p = subprocess.Popen(
["scanx", "--udp", host],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate()

If you don't need output at all you can pass devnull to stdout and stderr. I don't know if this can make a difference but pass a bufsize. Using devnull now subprocess.call doesn't suffer of deadlock anymore
import os
import subprocess
null = open(os.devnull, 'w')
subprocess.call(['ls', '-lR'], bufsize=4096, stdout=null, stderr=null)