Develop Reference

How does Jenkins deal with python scripts with Popen

We use Jenkins to run our cronjobs. We run Centos 6.8 on our server. Jenkins is version 1.651.
I'm running into a funny problem. When I run my script from the terminal, it works fine. I don't get any errors.
When I run the same script in Jenkins, it fails and says there's no such file or directory.
The error message from the Jenkins output I get is this:
Traceback (most recent call last):
File "runMTTRScript.py", line 256, in <module>
main()
File "runMTTRScript.py", line 252, in main
startTest(start, end, impalaHost)
File "runMTTRScript.py", line 72, in startTest
getResults(start, end)
File "runMTTRScript.py", line 111, in getResults
proc1 = subprocess.Popen(cmd, stdout=processlistOut)
File "/glide/lib64/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/glide/lib64/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Here's the code that the error above is complaining about:
with open(JAVAOUT + "_processList" + outfileDate, 'w') as processlistOut, \
open(JAVAOUT + "_innodb" + outfileDate, 'w') as innodbOut:
cmd = ["java", "-cp", "MTTR4hrs-1.0.5-SNAPSHOT-allinone.jar", "com.servicenow.bigdata.MTTR4hrs", "-c", "config.properties", "-m", DBIFILE, "-d", start, end, "-f", "processlist", "-ds", "dbi"]
proc1 = subprocess.Popen(cmd, stdout=processlistOut)
cmd = ["java", "-cp", "MTTR4hrs-1.0.5-SNAPSHOT-allinone.jar", "com.servicenow.bigdata.MTTR4hrs", "-c", "config.properties", "-m", DBIFILE, "-d", start, end, "-f", "engineinnodbstatus", "-ds", "dbi"]
proc2 = subprocess.Popen(cmd, stdout=innodbOut)
Why would it complain that a file is not there from Jenkins but be fine when I run it from cmd line? Could this also be some race condition in python that I'm not aware of too? The "with...open" doesn't open a file fast enough for the Popen to make use of? I'm also open to the fact that it might be some OS problem too (too many open files, something stupid, etc.).

Python error No such file or directory but the file is there

I'm trying to run a python script. The first steps of the scripts run ok, but at some point I have this message:
Traceback (most recent call last):
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/bin/LaunchTRF.py", line 154, in <module>
iLaunchTRF.run()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/bin/LaunchTRF.py", line 143, in run
self._launchTRF()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/bin/LaunchTRF.py", line 101, in _launchTRF
process = subprocess.Popen(cmd.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
2015-08-28 12:38:05 - DetectTEFeatures - ERROR - ERROR when launching 'LaunchTRF.py -i TEs.60bp.fa -o TEs.60bp.fa.SSR.set -m 15 -c -v 0 > launchTRF.log'
Traceback (most recent call last):
File "PASTEClassifier.py", line 199, in <module>
iLaunch.run()
File "PASTEClassifier.py", line 165, in run
iDF.run()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/commons/tools/DetectTEFeatures.py", line 185, in run
self._detectFeatures()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/commons/tools/DetectTEFeatures.py", line 204, in _detectFeatures
self._detectTRF()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/commons/tools/DetectTEFeatures.py", line 267, in _detectTRF
self._logAndRaise("ERROR when launching '%s'" % cmd)
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/commons/tools/DetectTEFeatures.py", line 176, in _logAndRaise
raise Exception(errorMsg)
Exception: ERROR when launching 'LaunchTRF.py -i TEs.60bp.fa -o TEs.60bp.fa.SSR.set -m 15 -c -v 0 > launchTRF.log'
I read some other posts with this type of error and based on them, I looked if the LaunchTRF.py existed and it is there. In fact, when I run only LaunchTRF.py without any option, I can see the list of options and the help. I was thinking that the missing file is TEs.60bp.fa, but it is also there.
Here are the lines 98 to 103 from LaunchTRF.py in case that may help:
def _launchTRF(self):
cmd = "trf %s 2 3 5 80 10 20 %d -h -d" % (self.inFileName, self.maxPeriod)
self._log.debug("Running : %s" % cmd)
process = subprocess.Popen(cmd.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = process.communicate()
self._log.debug("Output:\n%s" % output[0])
Any help is welcome.

print(self.inFileName)
should be equal to
"C:\blah\something\test.txt"
if it isn't just prepend the name in your command
fullFile = "C:\blah\hooray\"+self.inFileName
cmd = "trf %s 2 3 5 80 10 20 %d -h -d" % (fullFile, self.maxPeriod)

This seems to be a issue with your working directory, maybe the parent script is changing it, or you are executing the parent script from another directory.
What you could do to fix that is to change de working directory by passing an additional parameter to popen:
subprocess.Popen('./LaunchTRF.py ...', cwd='/path/to/')
Hope it helps

subprocess.check_output(): OSError file not found in Python

Executing following command and its variations always results in an error, which I just cannot figure out:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output([command])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
WHich file it is referring to ? other commands like ls,wc are running correctly though, the command is also running well on terminal but not python script.

Your command is a list with one element. Imagine if you tried to run this at the shell:
/bin/'dd if='/dev/'sda8 count=100 skip=$(expr 19868431049 '/' 512)'
That's effectively what you're doing. There's almost certainly no directory named dd if= in your bin directory, and there's even more almost certainly no dev directory under that with an sd8 count=100 skip=$(expr 19868431049 directory with a program named 512 in it.
What you want is a list where each argument is its own element:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100', 'skip=$(expr 19868431049 / 512)']
print subprocess.check_output(command) # notice no []
But that brings us to your second problem: $(expr 19868431049 / 512) isn't going to be parsed by Python or by dd; that's bash syntax. You can, of course, just do the same thing in Python instead of in bash:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100',
'skip={}'.format(19868431049 // 512)]
print subprocess.check_output(command)
Or, if you really want to use bash for no good reason, pass a string, rather than a list, and use shell=True:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True) # still no []
Although that still isn't going to work portably, because the default shell is /bin/sh, which may not know how to handle bashisms like $(…) (and expr, although I think POSIX requires that expr exist as a separate process…). So:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True, executable='/bin/bash')

This worked for me using subprocess.popen
command = "echo $JAVA_HOME"
proc = subprocess.Popen(command,stdout=subprocess.PIPE,shell=True)

NamedTemporaryFile cannot be acessed from command line

I have the following (simplified) code:
with NamedTemporaryFile() as f:
f.write(zip_data)
f.flush()
subprocess.call("/usr/bin/7z x %s" % f.name)
It dies with the following error:
Traceback (most recent call last):
File "decrypt_resource.py", line 70, in <module>
unpack(sys.argv[2])
File "decrypt_resource.py", line 28, in unpack
print(subprocess.check_output(cmd))
File "/usr/lib/python2.7/subprocess.py", line 568, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
However, if I use NamedTemporaryFile(delete=False) and then print & execute the command, it works. What's wrong here?
My System is an ArchLinux with a 3.9.5-1-ARCH kernel.

You are using subprocess.call() incorrectly.
Pass in a list of arguments:
subprocess.call(["/usr/bin/7z", "x", f.name])
The argument is not handled by a shell and is not parsed out like a shell would do. This is a good thing as it prevents a security problem with untrusted command line arguments.
Your other options include using shlex.split() to do the whitespace splitting for you, or, as a last resort, telling subprocess to use a shell for your command with the shell=True flag. See the big warning on the subprocess documentation about enabling the shell.

Python subprocess command arguments

Why if I run subprocess.check_output('ls') everything is working but when I add argument to command like: subprocess.check_output('ls -la') I get error:
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
How can I pass command arguments into subprocess.check_output()?

You need to split the arguments into a list:
subprocess.check_output(['ls', '-la'])
The subprocess callables do not parse the command out to individual arguments like the shell does. You either need to do this yourself or you need to tell subprocess to use the shell explicitly:
subprocess.check_output('ls -la', shell=True)
The latter is not recommended as it can expose your application to security vulnerabilities. You can use shlex.split() to parse a shell-like command line if needed:
>>> import shlex
>>> shlex.split('ls -la')
['ls', '-la']

You might find sh.py more friendly:
import sh
print sh.ls("-la")