Develop Reference

Running shell commands from Python and printing the output in real time

I want to write a function that will execute multiple shell commands one at a time and print what the shell returns in real time.
I currently have the following code which does not print the shell (I am using Windows 10 and python 3.6.2):
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
p.stdin.write((command + "\n").encode("utf-8"))
p.stdin.close()
p.stdout.read()
How can I see what the shell returns in real time ?
Edit : This question is not a duplicate of the two first links in the comments, they do not help printing in real time.

It is possible to handle stdin and stdout in different threads. That way one thread can be handling printing the output from stdout and another one writing new commands on stdin. However, since stdin and stdout are independent streams, I do not think this can guarantee the order between the streams. For the current example it seems to work as intended, though.
import subprocess
import threading
def stdout_printer(p):
for line in p.stdout:
print(line.rstrip())
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True)
t = threading.Thread(target=stdout_printer, args=(p,))
t.start()
for command in commands:
p.stdin.write((command + "\n"))
p.stdin.flush()
p.stdin.close()
t.join()
Also, note that I am writing stdout line by line, which is normally OK, since it tends to be buffered and being generated a line (or more) at a time. I guess it is possible to handle an unbuffered stdout stream (or e.g. stderr) character-by-character instead, if that is preferable.

I believe you need something like this
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
p.stdin.write((command + "\n").encode("utf-8"))
out, err = p.communicate()
print("{}".format(out))
print("{}".format(err))

Assuming you want control of the output in your python code you might need to do something like this
import subprocess
def run_process(exe):
'Define a function for running commands and capturing stdout line by line'
p = subprocess.Popen(exe.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
if __name__ == '__main__':
commands = ["foo", "foofoo"]
for command in commands:
for line in run_process(command):
print(line)

Python2: subprocess.Popen() has another output as it runs direct on console

When i am running a program in the console, i get some text output.
When i am running the same program in Popen(..), with the same parameters, stdout and stderr are empty.
I tried everything i could imagine like shell=False and shell=True, set stdout=subprocess.PIPE, did a os.chdir() to change into the directory of this program, try p.wait() and p.communicate(), set the command as a list and as a string, but nothing works.
example:
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
out, err = p.communicate()
--> out and err are empty strings, but if i ran this command in console i get a real output. Command is with fullpath, so its regardless where the command will be started.
My question is, are there mechanisms for programms to detect they weren't run in a real console? If so, how can i cheat.
Or miss i something?
(Python 2.7.8. x32 in Win7 x64)

from subprocess import Popen, STDOUT, PIPE
p = Popen(command, shell=True, stdout=PIPE, stderr=STDOUT, stdin=PIPE)
while p.poll() is None:
print(p.stdout.read())
p.stdout.close()
p.stdin.close()
Try this and see if it makes any difference. Also make sure command is a string and not a list/touple, shell=True for whatever reason works better or only with strings.
Also note that shell=True will get you hanged because it's insecure etc.
Also skipping .communicate() you'll need to tap off stdout otherwise the buffer will get full and you might hang both yours and the child process.
If this doesn't work, please provide more information. Such as the command used and the expected output (at least first few lines)

Python subprocess passes one argument only

I have a Python script (2.7) which I use to invoke an external process.Till recently it worked fine.
But now when I run it I see it doesn't pass over process arguments.I have also debugged the invoked process and it receives only the single argument (the path of the process executable).
p = subprocess.Popen(["./myapp","-p","s"],shell=True)
p.communicate()
Execution of the above code passes only "myapp" as the command argument.Why could that happen?

When using shell=True, just pass a string (not a list);
p = subprocess.Popen('./myapp -p s', shell=True)
p.communicate()
Update
Always prefer;
shell=False (the default) to shell=True and pass an array of strings; and
an absolute path to the executable, not a relative path.
I.e.;
with subprocess.Popen(['/path/to/binary', '-p', 's']) as proc:
stdout, stderr = proc.communicate()
If you're just interested in the stdout (and not the stderr), prefer this to the above solution (it's safer and shorter):
stdout = subprocess.check_output(['/path/to/binary', '-p', 's'])

Don't use shell=True:
p = subprocess.Popen(["./myapp","-p","s"])
p.communicate()

Python's Subprocess.Popen With Shell=True. Wait till it is completed

Submitting a complex cmd string made of a full file path to an executable, the multiple flags, arguments, parameters, inputs and outputs seems to require me to set shell=True otherwise subprocess.Popen is not able understand anything more complex than just a simple path to executable (with no spaces in a filepath).
In my example I have quite a long cmd:
cmd = " '/Application/MyApp.app/Contents/MacOS/my_executable' '/Path/to/input/files' -some -flags -here -could -be -a -lot '/full/path/to/output/files' "
Submitting this cmd to subprocess.Popen " results to an error that complains on something about the path and not being able to find it.
So instead of using :
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
check_call seems workings quite well:
proc = subprocess.check_call(cmd, shell=True)
Interesting, only after shell is set to True
shell=True
the subprocess.check_call works with a supplied cmd.
The side effect is that the rest of the code seems proceeds running without waiting for subprocess.check_call(cmd, shell=True) to finish first.
The code is designed the way that the rest of the execution is dependent on a result of subprocess.check_call(cmd, shell=True).
I wonder if there is anyway to enforce the code execution to wait till subprocess.check_call(cmd, shell=True) is finished. Thanks in advance!

As #mikkas suggest just use it as a list here is a working example:
mainProcess = subprocess.Popen(['python', pyfile, param1, param2], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# get the return value from the method
communicateRes = mainProcess.communicate()
stdOutValue, stdErrValue = communicateRes
You are calling python.exe pyfile param1 param2
By using communicate() you can get the stdout and stderr as a Tuple
You can use python method split() to split your string to a list for example:
cmd = "python.exe myfile.py arg1 arg2"
cmd.split(" ")
Output:
['python.exe', 'myfile.py', 'arg1', 'arg2']

I think the check_call function should wait for the command to finish.
See the docs here
http://docs.python.org/2/library/subprocess.html

Check call does not wait. You need to do a process.wait() and check the return code explicitly to get the functionaly you want.
Process = subprocess.Popen('%s' %command_string,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
Process.wait()
if Process1.returncode!=0:
print Process1.returncode
sendMail()
return
else:
sendMail()

Python - Execute Process -> Block till it exits & Suppress Output

I'm using the following to execute a process and hide its output from Python. It's in a loop though, and I need a way to block until the sub process has terminated before moving to the next iteration.
subprocess.Popen(["scanx", "--udp", host], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Use subprocess.call(). From the docs:
subprocess.call(*popenargs, **kwargs)
Run command with arguments. Wait for command to complete, then
return the returncode attribute.
The arguments are the same as for the
Popen constructor.
Edit:
subprocess.call() uses wait(), and wait() is vulnerable to deadlocks (as Tommy Herbert pointed out). From the docs:
Warning: This will deadlock if the
child process generates enough output
to a stdout or stderr pipe such that
it blocks waiting for the OS pipe
buffer to accept more data. Use
communicate() to avoid that.
So if your command generates a lot of output, use communicate() instead:
p = subprocess.Popen(
["scanx", "--udp", host],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate()

If you don't need output at all you can pass devnull to stdout and stderr. I don't know if this can make a difference but pass a bufsize. Using devnull now subprocess.call doesn't suffer of deadlock anymore
import os
import subprocess
null = open(os.devnull, 'w')
subprocess.call(['ls', '-lR'], bufsize=4096, stdout=null, stderr=null)