Develop Reference

Python subprocess: get all stdout data after proccess has terminated

I am using subprocess in python to invoke another executable, write some data to it's stdin (close the stream once everything is written - which is how the sub process knows it's recieved everything) and then receive all of it's stdout data after it's terminated - which it will after some period of time.
In pseudo code:
Open subprocess
write to it's stdout
let program finish
retrieve anything it spat out to stdout
I have tried the following:
import subprocess
p = subprocess.Popen([cmd],
stdout=subprocess.PIPE,stdin=subprocess.PIPE)
p.stdin.write(str(data))
p.stdin.close()
p.wait()
result = p.communicate()[0]
However I get the following stack trace:
result = p.communicate()[0]
File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate .
return self._communicate(input)
File "/usr/lib64/python2.7/subprocess.py", line 1396, in _communicate
self.stdin.flush()
ValueError: I/O operation on closed file
Please advise

Use communicate:
import subprocess
p = subprocess.Popen([cmd], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
result = p.communicate(data)[0]

data stream python subprocess.check_output exe from another location

I would like to run an exe from this directory:/home/pi/pi_sensors-master/bin/Release/
This exe is then run by tying mono i2c.exe and it runs fine.
I would like to get this output in python which is in a completely different directory.
I know that I should use subprocess.check_output to take the output as a string.
I tried to implement this in python:
import subprocess
import os
cmd = "/home/pi/pi_sensors-master/bin/Release/"
os.chdir(cmd)
process=subprocess.check_output(['mono i2c.exe'])
print process
However, I received this error:
The output would usually be a data stream with a new number each time, is it possible to capture this output and store it as a constantly changing variable?
Any help would be greatly appreciated.

Your command syntax is incorrect, which is actually generating the exception. You want to call mono i2c.exe, so your command list should look like:
subprocess.check_output(['mono', 'i2c.exe']) # Notice the comma separation.
Try the following:
import subprocess
import os
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
print subprocess.check_output(['mono', executable])
The sudo is not a problem as long as you give the full path to the file and you are sure that running the mono command as sudo works.
I can generate the same error by doing a ls -l:
>>> subprocess.check_output(['ls -l'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
However when you separate the command from the options:
>>> subprocess.check_output(['ls', '-l'])
# outputs my entire folder contents which are quite large.
I strongly advice you to use the subprocess.Popen -object to deal with external processes. Use Popen.communicate() to get the data from both stdout and stderr. This way you should not run into blocking problems.
import os
import subprocess
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
proc = subprocess.Popen(['mono', executable])
try:
outs, errs = proc.communicate(timeout=15) # Times out after 15 seconds.
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
Or you can call the communicate in a loop if you want a 'data-stream' of sort, an answer from this question:
from subprocess import Popen, PIPE
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
p = Popen(["mono", executable], stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.communicate() # close p.stdout, wait for the subprocess to exit

Running powershell script within python script, how to make python print the powershell output while it is running

I am writing a python script which checks various conditions and runs a powershell script accordingly to help me automate migration from windows XP to windows 7. The powershell script gives its own output giving the user updates as to what is happening. I would like to take the output of the powershell script and print it as output of the python script. I have looked around at some questions which seem to want to do the same thing but they don't seem to be working for me. Initially I tried using
import subprocess
subprocess.call(["C:\Users\gu2124\Desktop\helloworld.ps1"])
As was suggested here Run PowerShell function from Python script but I found out that this waits for the program to execute first and does not give output so I found out I need to use subprocess.Popen() as was suggusted here Use Popen to execute a Powershell script in Python, how can I get the Powershell script's output and update it to web page? so I tried this
import subprocess
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=sys.stdout)
and I get this error
Traceback (most recent call last):
File "C:\Users\gu2124\Desktop\pstest.py", line 5, in <module>
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.py1"], stdout=sys.stdout)
File "C:\Python27\lib\subprocess.py", line 701, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\Python27\lib\subprocess.py", line 848, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
File "<string>", line 523, in __getattr__
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\netref.py", line 150, in __getattr__
return syncreq(self, consts.HANDLE_GETATTR, name)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\netref.py", line 71, in syncreq
return conn.sync_request(handler, oid, *args)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 434, in sync_request
raise obj
AttributeError: DebugOutput instance has no attribute 'fileno'
I'm not completely sure what this means but from what I think I understand after reading this AttributeError: StringIO instance has no attribute 'fileno' is that it is because I am messing with the stdout incorrectly. I looked a around more and I found this Why won't my python subprocess code work? where the answers said to use stdout=subprocess.PIPE so I tried this
import subprocess
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
which also does not give me output
Finally I saw this http://www.pythonforbeginners.com/os/subprocess-for-system-administrators and changed my code to this
import subprocess
p = subprocess.Popen(["powershell","C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
print p.communicate
I thought that it may because I am initially trying to run a powershell script from the command line so I have to open powershell first. When I type these commands directly into the command line it works the way it should but when I run it through the python script it gives this
<bound method Popen.communicate of <subprocess.Popen object at 0x00000000026E4A90>>
which is an improvement I guess but not the "Hello world" I was expecting.
I have no idea what I should try to do next to get this to work. Any help would be greatly appreciated
Also if the powershell script I am using is needed here it is
$strString = "Hello World"
write-host $strString
function ftest{
$test = "Test"
write-host $test
}
EDIT: I tried upgrading to python 3.3 like was suggested in the first answer but I still can't get it to work. I used the command p = subprocess.Popen(['powershell.exe', "C:\\Users\\gu2124\\Desktop\\helloworld.ps1"], stdout=sys.stdout) and am sure the file is there but am getting this error:
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
p = subprocess.Popen(['powershell.exe', "C:\\Users\\gu2124\\Desktop\\helloworld.ps1"], stdout=sys.stdout)
File "C:\Python27\lib\subprocess.py", line 701, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\Python27\lib\subprocess.py", line 848, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
UnsupportedOperation: fileno

Make sure you can run powershell scripts (it is disabled by default). Likely you have already done this. http://technet.microsoft.com/en-us/library/ee176949.aspx
Set-ExecutionPolicy RemoteSigned
Run this python script on your powershell script helloworld.py:
# -*- coding: iso-8859-1 -*-
import subprocess, sys
p = subprocess.Popen(["powershell.exe",
"C:\\Users\\USER\\Desktop\\helloworld.ps1"],
stdout=sys.stdout)
p.communicate()
This code is based on python3.4 (or any 3.x series interpreter), though it should work on python2.x series as well.
C:\Users\MacEwin\Desktop>python helloworld.py
Hello World

I don't have Python 2.7 installed, but in Python 3.3 calling Popen with stdout set to sys.stdout worked just fine. Not before I had escaped the backslashes in the path, though.
>>> import subprocess
>>> import sys
>>> p = subprocess.Popen(['powershell.exe', 'C:\\Temp\\test.ps1'], stdout=sys.stdout)
>>> Hello World
_

In addition to the previous answers, I have some suggestions which makes your code more portable.
Instead of setting ExecutionPolicy globally to RemoteSigned (which imposes some security issues) you can use this to set it only for the PowerShell instance created by your Python script:
import subprocess, sys
p = subprocess.Popen('powershell.exe -ExecutionPolicy RemoteSigned -file "hello world.ps1"', stdout=sys.stdout)
p.communicate()
Note the quotes which allows your PowerShell-script's path/filename to contain spaces.
Furthermore, as shown in the above example, you can use a relative path to call your PowerShell script. The path is relative to your Python workspace directory.

This is how I get the output from Popen
p = subprocess.Popen(["powershell","C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
p_out, p_err = p.communicate()
print(p_out)
From the docs on Popen.communicate(). The function returns a tuple (stdout_data, stderr_data).

subprocess piping output on Windows

I've got this piece of code that works fine on Linux but fails on Windows. Process is created fine, but I get an error and nothing is read from pipe:
p = subprocess.Popen(['python', '-u', self.file_to_run,
'-s', '-g', '-i', self.input_file],
universal_newlines=True,
stdout=subprocess.PIPE)
...
out = p.stdout.readline().rstrip()
Error I get is
Traceback (most recent call last):
File "bench.py", line 59, in <module>
multi.add_process()
File "bench.py", line 47, in add_process
stdout=subprocess.PIPE)
File "c:\python\v2.5.1-ast3\...\lib\subprocess.py", line 615, in __init__
self.stdout = os.fdopen(c2pread, 'rU', bufsize)
OSError: [Errno 22] Invalid argument
I actually create multiple such processes and based on their output calculate some values, but that is irrelevant. What I need to do is, run the script with certain arguments multiple times and parse the data piped from stdout of each process.

try using sys.executable instead of 'python' in your subprocess args. I think this is because Python is not in the PATH on Windows.
Also check the value of self.file_to_run and self.input_file which must be strings and not None or strange stuff, but this probably won't cause an OSError.

NameError: name 'buffer' is not defined with Ant Based framework batch file

I'm using a python script to execute an Ant based framework batch file(Helium.bat)
subprocess.Popen('hlm '+commands, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
However the script will always stop and display the following error when it executes the .bat file:
import codecs
File "C:\Python25\lib\codecs.py", line 1007, in <module>
strict_errors = lookup_error("strict")
File "C:\Python25\lib\codecs.py", line 1007, in <module>
strict_errors = lookup_error("strict")
File "C:\Python25\lib\encodings\__init__.py", line 31, in <module>
import codecs, types
File "C:\Python25\lib\types.py", line 36, in <module>
BufferType = buffer
NameError: name 'buffer' is not defined
If I execute the .bat directly on command line, there will not be any issue.

I think at least part of the problem is how you're executing the batch file. Give this a try:
# execute the batch file as a separate process and echo its output
Popen_kwargs = { 'stdout': subprocess.PIPE, 'stderr': subprocess.STDOUT,
'universal_newlines': True }
with subprocess.Popen('hlm '+commands, **Popen_kwargs).stdout as output:
for line in output:
print line,
This pass different arguments to Popen -- the difference are this version removes the shell=True which isn't needed on a batch file, sets stderr=subprocess.STDOUT which redirects stdout to the same place stdout is going to to avoid missing any error messages, and adds a universal_newlines=True to make the output more readable.
Another difference is it reads and prints the output from the Popen process which will effectively make the Python script running the batch file wait until it's finished executing before continuing on -- which I suspect is important.