I am on Mac OS X Yosemite 10.10 and Python 2.7.
If I type the following: du -g -d1 /Users 2> /dev/null in the command line, everything works perfectly.
Now, my goal is to use that command in a python script.
My idea was to use the following:
import subprocess
output = subprocess.check_output(['du', '-g', '-d1', '/Users', '/dev/null'])
But I get this error:
Traceback (most recent call last):
File "./verifications.py", line 1, in <module>
output = subprocess.check_output(['du', '-g', '-d1', '/Users', '/dev/null'])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 537, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['du', '-g', '-d1', '/Users', '/dev/null']' returned non-zero exit status 1
Also, when I run subprocess.check_output(['du', '-g', '-d1', '/Users', '/dev/null']) in local everything works fine, the error happens when I am logged on a shared iMac using Apple's Shared Screen tool. I have a feeling that the problem might be due to permissions, but I cannot find anything.
For 2>/dev/null, the appropriate way to control redirection of file descriptor 2 with the subprocess.Popen family of calls is stderr=:
# Python 2.x, or 3.0-3.2
output = subprocess.check_output(['du', '-g', '-d1', '/Users'],
stderr=open('/dev/null', 'w'))
...or, with a Python supporting subprocess.DEVNULL:
# Python 3.3 or newer
output = subprocess.check_output(['du', '-g', '-d1', '/Users'],
stderr=subprocess.DEVNULL)
By the way, personally, I'd suggest something more like this:
p = subprocess.Popen(['du', '-g', '-d1', '/Users'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
if p.returncode != 0:
raise Exception(stderr)
...which, instead of sending stderr to /dev/null, keeps it around to use in generating a useful exception in the event that the command fails. (Pick an appropriate subclass of Exception, obviously).
Related
I'm trying to run multiple UNIX commands in a python script like this
import subprocess
cmds = ['sleep 3', 'uptime','time ls -l /']
p = subprocess.Popen(cmds,stdout=subprocess.PIPE,shell=True)
while p.poll() is None:
time.sleep(0.5)
tempdata = p.stdout.read()
print(tempdata)
However my output does not contain all output and doesn't seem to run all the commands. Setting shell=False also causes an error.
Traceback (most recent call last):
File "task1.py", line 32, in ?
p = subprocess.Popen(commands,stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
File "/usr/lib64/python36/subprocess.py", line 550, in __init__
errread, errwrite)
File "/usr/lib64/python36/subprocess.py", line 996, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
When you create a new process, you don't pass it a list of commands to run; rather, you pass it a single command -- either as a string (with shell=True) or as a list of args (with shell=False).
import subprocess
cmds = ['sleep 1', 'uptime', 'ls -l /']
for cmd in cmds:
stdout = subprocess.check_output(cmd, shell=True)
print('\n# {}'.format(cmd))
print(stdout)
If you just want to collect stdout, subprocess.check_output() might be simpler than Popen() -- but either approach will work, depending on what you need to do with the process.
Your problem is 'sleep 3' causes the error you get from the traceback, when I removed that it worked.
To run all for these:
cmds = ['sleep 3', 'uptime','time ls -l /']
You have to call popen for each of them:
for cmd in cmds:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
while p.poll() is None:
time.sleep(0.5)
output = p.stdout.read()
Or simpler:
for cmd in cmds:
output = subprocess.check_output(cmd, stdout=subprocess.PIPE, shell=True)
Second question: This captures all output written to stdout. To capture also stderr, redirect that into subprocess.PIPE as well.
I would like to run an exe from this directory:/home/pi/pi_sensors-master/bin/Release/
This exe is then run by tying mono i2c.exe and it runs fine.
I would like to get this output in python which is in a completely different directory.
I know that I should use subprocess.check_output to take the output as a string.
I tried to implement this in python:
import subprocess
import os
cmd = "/home/pi/pi_sensors-master/bin/Release/"
os.chdir(cmd)
process=subprocess.check_output(['mono i2c.exe'])
print process
However, I received this error:
The output would usually be a data stream with a new number each time, is it possible to capture this output and store it as a constantly changing variable?
Any help would be greatly appreciated.
Your command syntax is incorrect, which is actually generating the exception. You want to call mono i2c.exe, so your command list should look like:
subprocess.check_output(['mono', 'i2c.exe']) # Notice the comma separation.
Try the following:
import subprocess
import os
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
print subprocess.check_output(['mono', executable])
The sudo is not a problem as long as you give the full path to the file and you are sure that running the mono command as sudo works.
I can generate the same error by doing a ls -l:
>>> subprocess.check_output(['ls -l'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
However when you separate the command from the options:
>>> subprocess.check_output(['ls', '-l'])
# outputs my entire folder contents which are quite large.
I strongly advice you to use the subprocess.Popen -object to deal with external processes. Use Popen.communicate() to get the data from both stdout and stderr. This way you should not run into blocking problems.
import os
import subprocess
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
proc = subprocess.Popen(['mono', executable])
try:
outs, errs = proc.communicate(timeout=15) # Times out after 15 seconds.
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
Or you can call the communicate in a loop if you want a 'data-stream' of sort, an answer from this question:
from subprocess import Popen, PIPE
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
p = Popen(["mono", executable], stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.communicate() # close p.stdout, wait for the subprocess to exit
I've got this piece of code that works fine on Linux but fails on Windows. Process is created fine, but I get an error and nothing is read from pipe:
p = subprocess.Popen(['python', '-u', self.file_to_run,
'-s', '-g', '-i', self.input_file],
universal_newlines=True,
stdout=subprocess.PIPE)
...
out = p.stdout.readline().rstrip()
Error I get is
Traceback (most recent call last):
File "bench.py", line 59, in <module>
multi.add_process()
File "bench.py", line 47, in add_process
stdout=subprocess.PIPE)
File "c:\python\v2.5.1-ast3\...\lib\subprocess.py", line 615, in __init__
self.stdout = os.fdopen(c2pread, 'rU', bufsize)
OSError: [Errno 22] Invalid argument
I actually create multiple such processes and based on their output calculate some values, but that is irrelevant. What I need to do is, run the script with certain arguments multiple times and parse the data piped from stdout of each process.
try using sys.executable instead of 'python' in your subprocess args. I think this is because Python is not in the PATH on Windows.
Also check the value of self.file_to_run and self.input_file which must be strings and not None or strange stuff, but this probably won't cause an OSError.
Windows version of Python 2.6.4: Is there any way to determine if subprocess.Popen() fails when using shell=True?
Popen() successfully fails when shell=False
>>> import subprocess
>>> p = subprocess.Popen( 'Nonsense.application', shell=False )
Traceback (most recent call last):
File ">>> pyshell#258", line 1, in <module>
p = subprocess.Popen( 'Nonsense.application' )
File "C:\Python26\lib\subprocess.py", line 621, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 830, in
_execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
But when shell=True, there appears to be no way to determine if a Popen() call was successful or not.
>>> p = subprocess.Popen( 'Nonsense.application', shell=True )
>>> p
>>> subprocess.Popen object at 0x0275FF90>>>
>>> p.pid
6620
>>> p.returncode
>>>
Ideas appreciated.
Regards,
Malcolm
returncode will work, although it will be None until you've called p.poll(). poll() itself will return the error code, so you can just do
if a.poll() != 0:
print ":("
In the first case it fails to start, in the second - it successfully starts shell which, in turn, fails to execute the application. So your process has been properly spawned, exited and waits for you to inquire about its exit code. So, the thing is, unless your shell or environment (e.g. no memory) is utterly broken there's no way Popen itself may fail.
So, you can safely .poll() and .wait() on it to get all the sad news.
How do I run this command with subprocess?
I tried:
proc = subprocess.Popen(
'''ECHO bosco|"C:\Program Files\GNU\GnuPG\gpg.exe" --batch --passphrase-fd 0 --output "c:\docume~1\usi\locals~1\temp\tmptlbxka.txt" --decrypt "test.txt.gpg"''',
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
stdout_value, stderr_value = proc.communicate()
but got:
Traceback (most recent call last):
...
File "C:\Python24\lib\subprocess.py", line 542, in __init__
errread, errwrite)
File "C:\Python24\lib\subprocess.py", line 706, in _execute_child
startupinfo)
WindowsError: [Errno 2] The system cannot find the file specified
Things I've noticed:
Running the command on the windows
console works fine.
If I remove the
ECHO bosco| part, it runs fine the
the popen call above. So I think
this problem is related to echo or
|.
First and foremost, you don't actually need a pipe; you are just sending input. You can use subprocess.communicate for that.
Secondly, don't specify the command as a string; that's messy as soon as filenames with spaces are involved.
Thirdly, if you really wanted to execute a piped command, just call the shell. On Windows, I believe it's cmd /c program name arguments | further stuff.
Finally, single back slashes can be dangerous: "\p" is '\\p', but '\n' is a new line. Use os.path.join() or os.sep or, if specified outside python, just a forward slash.
proc = subprocess.Popen(
['C:/Program Files/GNU/GnuPG/gpg.exe',
'--batch', '--passphrase-fd', '0',
'--output ', 'c:/docume~1/usi/locals~1/temp/tmptlbxka.txt',
'--decrypt', 'test.txt.gpg',],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
stdout_value, stderr_value = proc.communicate('bosco')
You were right, the ECHO is the problem. Without the shell=True option the ECHO command cannot be found.
This fails with the error you saw:
subprocess.call(["ECHO", "Ni"])
This passes: prints Ni and a 0
subprocess.call(["ECHO", "Ni"], shell=True)