Develop Reference

Cannot run Python Subprocess calls using Jenkins

The question has been asked elsewhere, but I do not see any solutions for this...
I have a Python script that contains subprocess calls of the form:
source_density = subprocess.check_output(["adb", device_dhm1, "-P8080", "-s", source, "shell", "wm density | head -2 | tail -1"])
When running this script through cmd, everything runs fine. However now I am trying to run it through Jenkins and I am getting the following error output:
======================================================================
ERROR: test_sanity (__main__.ABC)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/user123/.jenkins/workspace/Test/Jenkins-Source-Sanity.py", line 145, in test_sanity
source_density = subprocess.check_output(["adb", device_dhm1, "-P8080", "-s", source, "shell", "wm density | head -2 | tail -1"])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 216, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Per default, check_output treats the first argument as a single executable instead of a shell command (cmd command). So you have to set the flag shell=True explicitly.
source_density = subprocess.check_output(["adb", device_dhm1, "-P8080", "-s", source, "shell", "wm density | head -2 | tail -1"], shell=True)
Please read the doc here.

subprocess.check_output(): OSError file not found in Python

Executing following command and its variations always results in an error, which I just cannot figure out:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output([command])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
WHich file it is referring to ? other commands like ls,wc are running correctly though, the command is also running well on terminal but not python script.

Your command is a list with one element. Imagine if you tried to run this at the shell:
/bin/'dd if='/dev/'sda8 count=100 skip=$(expr 19868431049 '/' 512)'
That's effectively what you're doing. There's almost certainly no directory named dd if= in your bin directory, and there's even more almost certainly no dev directory under that with an sd8 count=100 skip=$(expr 19868431049 directory with a program named 512 in it.
What you want is a list where each argument is its own element:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100', 'skip=$(expr 19868431049 / 512)']
print subprocess.check_output(command) # notice no []
But that brings us to your second problem: $(expr 19868431049 / 512) isn't going to be parsed by Python or by dd; that's bash syntax. You can, of course, just do the same thing in Python instead of in bash:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100',
'skip={}'.format(19868431049 // 512)]
print subprocess.check_output(command)
Or, if you really want to use bash for no good reason, pass a string, rather than a list, and use shell=True:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True) # still no []
Although that still isn't going to work portably, because the default shell is /bin/sh, which may not know how to handle bashisms like $(…) (and expr, although I think POSIX requires that expr exist as a separate process…). So:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True, executable='/bin/bash')

This worked for me using subprocess.popen
command = "echo $JAVA_HOME"
proc = subprocess.Popen(command,stdout=subprocess.PIPE,shell=True)

data stream python subprocess.check_output exe from another location

I would like to run an exe from this directory:/home/pi/pi_sensors-master/bin/Release/
This exe is then run by tying mono i2c.exe and it runs fine.
I would like to get this output in python which is in a completely different directory.
I know that I should use subprocess.check_output to take the output as a string.
I tried to implement this in python:
import subprocess
import os
cmd = "/home/pi/pi_sensors-master/bin/Release/"
os.chdir(cmd)
process=subprocess.check_output(['mono i2c.exe'])
print process
However, I received this error:
The output would usually be a data stream with a new number each time, is it possible to capture this output and store it as a constantly changing variable?
Any help would be greatly appreciated.

Your command syntax is incorrect, which is actually generating the exception. You want to call mono i2c.exe, so your command list should look like:
subprocess.check_output(['mono', 'i2c.exe']) # Notice the comma separation.
Try the following:
import subprocess
import os
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
print subprocess.check_output(['mono', executable])
The sudo is not a problem as long as you give the full path to the file and you are sure that running the mono command as sudo works.
I can generate the same error by doing a ls -l:
>>> subprocess.check_output(['ls -l'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
However when you separate the command from the options:
>>> subprocess.check_output(['ls', '-l'])
# outputs my entire folder contents which are quite large.
I strongly advice you to use the subprocess.Popen -object to deal with external processes. Use Popen.communicate() to get the data from both stdout and stderr. This way you should not run into blocking problems.
import os
import subprocess
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
proc = subprocess.Popen(['mono', executable])
try:
outs, errs = proc.communicate(timeout=15) # Times out after 15 seconds.
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
Or you can call the communicate in a loop if you want a 'data-stream' of sort, an answer from this question:
from subprocess import Popen, PIPE
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
p = Popen(["mono", executable], stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.communicate() # close p.stdout, wait for the subprocess to exit

Python read windows Command Line output

I am trying to execute a command in python and read its output on command line in windows.
I have written the following code so far:
def build():
command = "cobuild archive"
print "Executing build"
pipe = Popen(command,stdout=PIPE,stderr=PIPE)
while True:
line = pipe.stdout.readline()
if line:
print line
I want to execute the command cobuild archive in command line and read it's output. However, the above code is giving me this error.
File "E:\scripts\utils\build.py", line 33, in build
pipe = Popen(command,stdout=PIPE,stderr=PIPE)
File "C:\Python27\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 893, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

The following code worked. I needed to pass shell=True for the arguments
def build():
command = "cobuild archive"
pipe = Popen(command,shell=True,stdout=PIPE,stderr=PIPE)
while True:
line = pipe.stdout.readline()
if line:
print line
if not line:
break

WindowsError: [Error 2] The system cannot find the file specified
This error says that the subprocess module is unable to locate your executable(.exe)
here "cobuild archive"
Suppose, if your executable in this path: "C:\Users\..\Desktop",
then, do,
import os
os.chdir(r"C:\Users\..\Desktop")
and then use your subprocess

Do you mind to post your code with the correct indentations please? They have a large effect in python - another way of doing this is:
import commands
# the command to execute
cmd = "cobuild archive"
# execute and get stdout
output = commands.getstatusoutput( cmd )
# do something with output
# ...
Update:
The commands module has been removed in Python 3, so this is a solution for python 2 only.
https://docs.python.org/2/library/commands.html

NamedTemporaryFile cannot be acessed from command line

I have the following (simplified) code:
with NamedTemporaryFile() as f:
f.write(zip_data)
f.flush()
subprocess.call("/usr/bin/7z x %s" % f.name)
It dies with the following error:
Traceback (most recent call last):
File "decrypt_resource.py", line 70, in <module>
unpack(sys.argv[2])
File "decrypt_resource.py", line 28, in unpack
print(subprocess.check_output(cmd))
File "/usr/lib/python2.7/subprocess.py", line 568, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
However, if I use NamedTemporaryFile(delete=False) and then print & execute the command, it works. What's wrong here?
My System is an ArchLinux with a 3.9.5-1-ARCH kernel.

You are using subprocess.call() incorrectly.
Pass in a list of arguments:
subprocess.call(["/usr/bin/7z", "x", f.name])
The argument is not handled by a shell and is not parsed out like a shell would do. This is a good thing as it prevents a security problem with untrusted command line arguments.
Your other options include using shlex.split() to do the whitespace splitting for you, or, as a last resort, telling subprocess to use a shell for your command with the shell=True flag. See the big warning on the subprocess documentation about enabling the shell.