I'm currently learning how to use subprocesses, for this (and some other) reasons
I've bought a book to learn how to work with subprocesses. It's a good book
and I'm not having troubles understanding it. In my book they start to explain
how to execute shell commands as subprocesses.
I had a programming problem I had for ages, and with the subprocesses, I could
be able to solve it but I need to execute a function callback as subprocess.
I have this code to echo something but it's a shell command:
import subprocess
proc = subprocess.Popen(['echo', 'Hello, this is child process speaking'],
stdout=subprocess.PIPE, shell=True)
out, err = proc.communicate()
print(out.decode('utf-8'))
I want this callback to be executed as a subprocess:
def callb()
import time as t
print('2')
t.sleep(2)
print('1')
t.sleep(2)
print('0')
I just tried out to execute this callback like this (it was a simple naive idea):
proc = subprocess.Popen(callb())
but this gives me the following error:
Traceback (most recent call last):
File "/root/testfile.py", line 6, in <module>
proc = subprocess.Popen(callb())
File "/usr/lib/python3.3/subprocess.py", line 818, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.3/subprocess.py", line 1321, in _execute_child
args = list(args)
TypeError: 'NoneType' object is not iterable
The strange thing is, that it does execute the callback, but then it raises this error!
What did I do wrong? Did I forget something?
the subprocess module is not suitable for executing python callbacks. you want to look at the multiprocessing module instead. the first few examples with Process and Pool seem to do what you want.
Related
I am trying to get realtime output of a subprocess.call by defining my own output stream but it doesn't seem to work.
Reason: I want to run a subprocess and get output of that call to both stdout(in realtime so i can look at the script and see current progress) as well as logging it to a file
subprocess.py:
import time
while True:
print("Things")
time.sleep(1)
mainprocess.py
import subprocess
import io
class CustomIO(io.IOBase):
def write(self, str):
print("CustomIO: %s"%str)
# logging to be implemented here
customio = CustomIO()
subprocess.call(["python3", "print_process.py"], stdout=customio)
But when i run this code i get this error message:
Traceback (most recent call last):
File "call_test.py", line 9, in <module>
subprocess.call(["python3", "print_process.py"], stdout=customio)
File "/usr/lib/python3.4/subprocess.py", line 537, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.4/subprocess.py", line 823, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "/usr/lib/python3.4/subprocess.py", line 1302, in _get_handles
c2pwrite = stdout.fileno()
io.UnsupportedOperation: fileno
So, anyone have any clue if this is possible?
Am i inheriting the wrong baseclass?
Am i not overloading the proper methods?
Or am i completely off the rails and should be going about this in a completely different way?
If you want to process the output of a subprocess, you need to pass stdout=subprocess.PIPE. However, call() and run() will both wait until the process is finished before making it available, so you cannot handle it in real time using these functions.
You need to use subprocess.Popen:
import subprocess as sp
def handle_output(output_line):
...
my_process = sp.Popen(["python3", "print_process.py"],
stdout=sp.PIPE,
universal_newlines=True) # changes stdout from bytes to text
for line in my_process.stdout:
handle_output(line)
my_process.wait()
Update: Make sure to flush the output buffer in your child process:
while True:
print("Things", flush=True)
time.sleep(1)
You need to specify and open stream with a file descriptor. fileno isn't implemented for io.IOBase because this is just an in-memory stream:
Frequently Used Arguments
stdin, stdout and stderr specify the executed program’s standard
input, standard output and standard error file handles, respectively.
Valid values are PIPE, DEVNULL, an existing file descriptor (a
positive integer), an existing file object, and None. PIPE indicates
that a new pipe to the child should be created. DEVNULL indicates that
the special file os.devnull will be used. With the default settings of
None, no redirection will occur;
So you might use sockets, pipes, and open files as stdout, the file descriptor is passed to the child process as it's stdout. I didn't use sockets with subprocess.Popen though, but I expect them to work, I believe what matters here is the file descriptor to the child, not what type of object the file descriptor points to.
My code is as follows, basically this module will run the required command and capture its ouput line by line but in my case when the command runs, it takes just more than a second to return to the command prompt and thats where child.stdout.read(1) hangs, if I run a normal command using this it prints everthing as expected. but in a particular case where, the command prints somthing to STDOUT and then takes some time to return to the prompt, it hangs.. Please help
New code:
def run_command(shell_command):
'''run the required command and print the log'''
child = subprocess.Popen(shell_command, shell=True,stdout=subprocess.PIPE)
(stdoutdata, stderrdata) = child.communicate()
print stdoutdata
print "Exiting.."
Error:
File "upgrade_cloud.py", line 62, in <module>
stop_cloud()
File "upgrade_cloud.py", line 49, in stop_cloud
run_command(shell_command)
File "upgrade_cloud.py", line 33, in run_command
(stdoutdata, stderrdata) = child.communicate()
File "/usr/lib/python2.6/subprocess.py", line 693, in communicate
stdout = self.stdout.read()
KeyboardInterrupt
Here's your problem:
child.wait()
This line causes Python to wait for the child process to exit. If the child process tries to print a lot of data to stdout, it will block waiting for Python to read said data. Since Python is waiting for the child process and the child process is waiting for Python, you get a deadlock.
I would recommend using subprocess.check_output() instead of subprocess.Popen. You could also use the Popen.communicate() method instead of the .wait() method.
I would like to run an exe from this directory:/home/pi/pi_sensors-master/bin/Release/
This exe is then run by tying mono i2c.exe and it runs fine.
I would like to get this output in python which is in a completely different directory.
I know that I should use subprocess.check_output to take the output as a string.
I tried to implement this in python:
import subprocess
import os
cmd = "/home/pi/pi_sensors-master/bin/Release/"
os.chdir(cmd)
process=subprocess.check_output(['mono i2c.exe'])
print process
However, I received this error:
The output would usually be a data stream with a new number each time, is it possible to capture this output and store it as a constantly changing variable?
Any help would be greatly appreciated.
Your command syntax is incorrect, which is actually generating the exception. You want to call mono i2c.exe, so your command list should look like:
subprocess.check_output(['mono', 'i2c.exe']) # Notice the comma separation.
Try the following:
import subprocess
import os
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
print subprocess.check_output(['mono', executable])
The sudo is not a problem as long as you give the full path to the file and you are sure that running the mono command as sudo works.
I can generate the same error by doing a ls -l:
>>> subprocess.check_output(['ls -l'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
However when you separate the command from the options:
>>> subprocess.check_output(['ls', '-l'])
# outputs my entire folder contents which are quite large.
I strongly advice you to use the subprocess.Popen -object to deal with external processes. Use Popen.communicate() to get the data from both stdout and stderr. This way you should not run into blocking problems.
import os
import subprocess
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
proc = subprocess.Popen(['mono', executable])
try:
outs, errs = proc.communicate(timeout=15) # Times out after 15 seconds.
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
Or you can call the communicate in a loop if you want a 'data-stream' of sort, an answer from this question:
from subprocess import Popen, PIPE
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
p = Popen(["mono", executable], stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.communicate() # close p.stdout, wait for the subprocess to exit
I am writing a python script which checks various conditions and runs a powershell script accordingly to help me automate migration from windows XP to windows 7. The powershell script gives its own output giving the user updates as to what is happening. I would like to take the output of the powershell script and print it as output of the python script. I have looked around at some questions which seem to want to do the same thing but they don't seem to be working for me. Initially I tried using
import subprocess
subprocess.call(["C:\Users\gu2124\Desktop\helloworld.ps1"])
As was suggested here Run PowerShell function from Python script but I found out that this waits for the program to execute first and does not give output so I found out I need to use subprocess.Popen() as was suggusted here Use Popen to execute a Powershell script in Python, how can I get the Powershell script's output and update it to web page? so I tried this
import subprocess
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=sys.stdout)
and I get this error
Traceback (most recent call last):
File "C:\Users\gu2124\Desktop\pstest.py", line 5, in <module>
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.py1"], stdout=sys.stdout)
File "C:\Python27\lib\subprocess.py", line 701, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\Python27\lib\subprocess.py", line 848, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
File "<string>", line 523, in __getattr__
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\netref.py", line 150, in __getattr__
return syncreq(self, consts.HANDLE_GETATTR, name)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\netref.py", line 71, in syncreq
return conn.sync_request(handler, oid, *args)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 434, in sync_request
raise obj
AttributeError: DebugOutput instance has no attribute 'fileno'
I'm not completely sure what this means but from what I think I understand after reading this AttributeError: StringIO instance has no attribute 'fileno' is that it is because I am messing with the stdout incorrectly. I looked a around more and I found this Why won't my python subprocess code work? where the answers said to use stdout=subprocess.PIPE so I tried this
import subprocess
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
which also does not give me output
Finally I saw this http://www.pythonforbeginners.com/os/subprocess-for-system-administrators and changed my code to this
import subprocess
p = subprocess.Popen(["powershell","C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
print p.communicate
I thought that it may because I am initially trying to run a powershell script from the command line so I have to open powershell first. When I type these commands directly into the command line it works the way it should but when I run it through the python script it gives this
<bound method Popen.communicate of <subprocess.Popen object at 0x00000000026E4A90>>
which is an improvement I guess but not the "Hello world" I was expecting.
I have no idea what I should try to do next to get this to work. Any help would be greatly appreciated
Also if the powershell script I am using is needed here it is
$strString = "Hello World"
write-host $strString
function ftest{
$test = "Test"
write-host $test
}
EDIT: I tried upgrading to python 3.3 like was suggested in the first answer but I still can't get it to work. I used the command p = subprocess.Popen(['powershell.exe', "C:\\Users\\gu2124\\Desktop\\helloworld.ps1"], stdout=sys.stdout) and am sure the file is there but am getting this error:
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
p = subprocess.Popen(['powershell.exe', "C:\\Users\\gu2124\\Desktop\\helloworld.ps1"], stdout=sys.stdout)
File "C:\Python27\lib\subprocess.py", line 701, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\Python27\lib\subprocess.py", line 848, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
UnsupportedOperation: fileno
Make sure you can run powershell scripts (it is disabled by default). Likely you have already done this. http://technet.microsoft.com/en-us/library/ee176949.aspx
Set-ExecutionPolicy RemoteSigned
Run this python script on your powershell script helloworld.py:
# -*- coding: iso-8859-1 -*-
import subprocess, sys
p = subprocess.Popen(["powershell.exe",
"C:\\Users\\USER\\Desktop\\helloworld.ps1"],
stdout=sys.stdout)
p.communicate()
This code is based on python3.4 (or any 3.x series interpreter), though it should work on python2.x series as well.
C:\Users\MacEwin\Desktop>python helloworld.py
Hello World
I don't have Python 2.7 installed, but in Python 3.3 calling Popen with stdout set to sys.stdout worked just fine. Not before I had escaped the backslashes in the path, though.
>>> import subprocess
>>> import sys
>>> p = subprocess.Popen(['powershell.exe', 'C:\\Temp\\test.ps1'], stdout=sys.stdout)
>>> Hello World
_
In addition to the previous answers, I have some suggestions which makes your code more portable.
Instead of setting ExecutionPolicy globally to RemoteSigned (which imposes some security issues) you can use this to set it only for the PowerShell instance created by your Python script:
import subprocess, sys
p = subprocess.Popen('powershell.exe -ExecutionPolicy RemoteSigned -file "hello world.ps1"', stdout=sys.stdout)
p.communicate()
Note the quotes which allows your PowerShell-script's path/filename to contain spaces.
Furthermore, as shown in the above example, you can use a relative path to call your PowerShell script. The path is relative to your Python workspace directory.
This is how I get the output from Popen
p = subprocess.Popen(["powershell","C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
p_out, p_err = p.communicate()
print(p_out)
From the docs on Popen.communicate(). The function returns a tuple (stdout_data, stderr_data).
Windows version of Python 2.6.4: Is there any way to determine if subprocess.Popen() fails when using shell=True?
Popen() successfully fails when shell=False
>>> import subprocess
>>> p = subprocess.Popen( 'Nonsense.application', shell=False )
Traceback (most recent call last):
File ">>> pyshell#258", line 1, in <module>
p = subprocess.Popen( 'Nonsense.application' )
File "C:\Python26\lib\subprocess.py", line 621, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 830, in
_execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
But when shell=True, there appears to be no way to determine if a Popen() call was successful or not.
>>> p = subprocess.Popen( 'Nonsense.application', shell=True )
>>> p
>>> subprocess.Popen object at 0x0275FF90>>>
>>> p.pid
6620
>>> p.returncode
>>>
Ideas appreciated.
Regards,
Malcolm
returncode will work, although it will be None until you've called p.poll(). poll() itself will return the error code, so you can just do
if a.poll() != 0:
print ":("
In the first case it fails to start, in the second - it successfully starts shell which, in turn, fails to execute the application. So your process has been properly spawned, exited and waits for you to inquire about its exit code. So, the thing is, unless your shell or environment (e.g. no memory) is utterly broken there's no way Popen itself may fail.
So, you can safely .poll() and .wait() on it to get all the sad news.