I am trying to setup a Jenkins pipeline which runs tox inside a docker container. There is a known issue that shebang lines get very long inside Jenkins, and two solutions are proposed. The first is to use --workdir to select a shorter path. This option works in principle, but I loose the automatic unique path-names per project from Jenkins. I would thus prefer to use the second option, TOX_LIMITED_SHEBANG. Unfortunately, that seems to fail with the following error when the package under test is supposed to be installed: FileNotFoundError: [Errno 2] No such file or directory: "b'/bin/sh'": "b'/bin/sh'". I have verified that /bin/sh is in fact available in the docker container. The Jenkinsfile looks as follows:
node("docker") {
// burnpanck/tox-base contains tox and many python versions
docker.image('burnpanck/tox-base').inside {
checkout scm
stage('Matrix-test using Tox') {
// verify that /bin/sh exists
sh 'ls -al /bin'
// the following does not work
sh 'TOX_LIMITED_SHEBANG=1 tox -vv'
// the following works
// sh 'tox --workdir=/var/jenkins_home/tox'
}
}
}
Tox is version 3.1.2 and runs under python 3.6 (the docker image is generated from this Dockerfile). What surprises me a little is the "b'/bin/sh'" coming from str-ing a bytes instance. Could it be that tox is in fact trying to run a program by the name sh' in the path b'/bin?
The tox.ini in use simply calls pytest:
[tox]
envlist = py36
[testenv]
recreate = True
commands =
pytest
The full backtrace from tox (Jenkins console output) is the following:
py36 create: /var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ/.tox/py36
py36 inst: /var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ/.tox/dist/test_model-0.dev20180717.zip
ERROR: invocation failed (errno 2), args: [b'/bin/sh', '/var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ/.tox/py36/bin/pip', 'install', '/var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ/.tox/dist/test_model-0.dev20180717.zip'], cwd: /var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ
Traceback (most recent call last):
File "/.pyenv/versions/3.6.6/bin/tox", line 11, in <module>
sys.exit(cmdline())
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 39, in cmdline
main(args)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 45, in main
retcode = Session(config).runcommand()
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 422, in runcommand
return self.subcommand_test()
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 620, in subcommand_test
self.installpkg(venv, path)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 561, in installpkg
venv.installpkg(path, action)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/venv.py", line 277, in installpkg
self._install([sdistpath], extraopts=extraopts, action=action)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/venv.py", line 342, in _install
self.run_install_command(packages=packages, options=options, action=action)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/venv.py", line 314, in run_install_command
redirect=self.session.report.verbosity < 2,
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/venv.py", line 427, in _pcall
return action.popen(args, cwd=cwd, env=env, redirect=redirect, ignore_ret=ignore_ret)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 153, in popen
popen = self._popen(args, cwd, env=env, stdout=stdout, stderr=subprocess.STDOUT)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 248, in _popen
env=env,
File "/.pyenv/versions/3.6.6/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/.pyenv/versions/3.6.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: "b'/bin/sh'": "b'/bin/sh'"
We currently have an open bug for this. TOX_LIMITED_SHEBANG is broken when tox is running under python3.
The root cause is there's a .decode() missing and a list ends up with mixed bytes and str instances.
Workarounds until I fix it are to use tox with a python2 interpreter or don't use TOX_LIMITED_SHEBANG.
Related
I'm trying to install Open CV 2 on a shared hosting inside a virtualenv.
I already got numpy and all those stuff downloaded using Pip. I'm just having a bit of trouble with OpenCV2
I run this command in the ssh session
(penv)[dire]$ cmake -D MAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=$VIRTUAL_ENV/local/ -D PYTHON_EXECTUABLE=$PYTHONPATH/python2.7 -D PYTHON_PACKAGES_PATH=$VIRTUAL_ENV/lib/python2.7/site-packages -D INSTALL_PYTHON_EXAMPLES=ON ..
and the error I get is
Traceback (most recent call last):
File "/home/bashtroubles/website.com/public/NNPics/penv/bin/cmake", line 11, in <module>
sys.exit(cmake())
File "/home/bashtroubles/website.com/public/NNPics/penv/local/lib/python2.7/site-packages/cmake/__init__.py", line 33, in cmake
raise SystemExit(_program('cmake', sys.argv[1:]))
File "/home/bashtroubles/website.com/public/NNPics/penv/local/lib/python2.7/site-packages/cmake/__init__.py", line 29, in _program
return subprocess.call([os.path.join(CMAKE_BIN_DIR, name)] + args)
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
(penv)[dire]$
This is also my .bashrc if it matters
# ~/.bashrc: executed by bash(1) for non-login shells.
# Load pythonbrew
alias pb='pythonbrew'
export PYTHONPATH=~/.pythonbrew/pythons/Python-2.7.3/lib
[[ -s /home/bashtroubles/.pythonbrew/etc/bashrc ]] && source /home/bashtroubles/.python$
# Load custom python installation
export PATH=~/opt/python-2.7.3/bin:${PATH}
export PYTHONPATH=~/opt/python-2.7.3/lib
The specific version is opencv-2.4.13 and the python version is 2.7.3
I believe the issue is because it's using the python2.7 from
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
Any ideas on how to get this going without a permission denied issue?
I ran into this problem. It looks like the binary it's trying to call isn't marked as an executable. I ran this to change the permissions
sudo chmod +x -R /usr/local/lib/python2.7/dist-packages/cmake-3.13.3-py2.7-linux-x86_64.egg/cmake/data/bin
We are using Azure Batch Apps which will create multiple VMs which can be used to run our tasks parallelly. We are using python for data fetching tasks.
We have mentioned in the batch apps to install anaconda on the VMs when they start up. Anaconda is installed properly. We have listed out the packages(requirements.txt) we need to install to run the tasks.
pip install -r requirements.txt
Some packages get installed correclty, but some packages result in the following error,
Error [Error 6] The handle is invalid while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\basecommand.py", line 209, in main
status = self.run(options, args)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\commands\install.py", line 310, in run
wb.build(autobuilding=True)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\wheel.py", line 748, in build
self.requirement_set.prepare_files(self.finder)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 591, in _prepare_file
abstract_dist.prep_for_dist()
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 127, in prep_for_dist
self.req_to_install.run_egg_info()
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_install.py", line 430, in run_egg_info
command_desc='python setup.py egg_info')
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\utils\__init__.py", line 678, in call_subprocess
cwd=cwd, env=env)
File "C:\user\tasks\shared\anaconda2\lib\subprocess.py", line 702, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\user\tasks\shared\anaconda2\lib\subprocess.py", line 823, in _get_handles
p2cread = _subprocess.GetStdHandle(_subprocess.STD_INPUT_HANDLE)
WindowsError: [Error 6] The handle is invalid
When we open the VM and give the same command, all packages get installed correctly.
I just wonder where the issue is.
It seems that the issue was caused by some limits for Azure Batch service, you can see these limits here.
According to the error information, it seems that the installation process needs to fork the subprocess, but the maximum number of tasks per computer node is 4, please see below.
I have OSX and am running the python script out of the Unix shell
I'm running a python code that should open an application. I've been testing with Firefox.app and have been getting
Traceback (most recent call last):
File "/Users/brennan/Desktop/Coding/Wilix/wilix.py", line 453, in <module>
subprocess.call(["open -a "+cp2])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
My code is:
subprocess.call(["open -a "+cp2])
where cp2 is user input. (Firefox in this case)
if I cd into the programs directory and then do
open -a Firefox
firefox opens fine.
if I change my code to
subprocess.call(["open -a Firefox"])
I still get the error message.
You're passing open -a Firefox as one argument, as if you ran this in the shell:
$ "open -a Firefox"
You need to split up the items:
subprocess.call(['open', '-a', 'Firefox'])
Try giving the full path of firefox app.
It's wrong to use subprocess.call without shell=True or providing command as a list. Please, take a look at first examples in the docs:
http://docs.python.org/2/library/subprocess.html
Full path to Firefox may be needed or may be not needed.
I'm working with a set of scripts aimed at modifying an android app, the github repo located here: Ingress Broot Mod. I installed Python 3.2, then installed PyYAML 3.10. I moved on to follow the build instructions and "Import Ingress apk using import_apk.py script."
Specifically, I typed: B:\GitHub\ingress-apk-mod\bin>python import_apk.py "B:\com.nianticproject.ingress-1.apk"
At which point this error pops up.
Traceback (most recent call last):
File "B:\GitHub\ingress-apk-mod\bin\import_apk.py", line 28, in <module>
main(sys.argv[1])
File "B:\GitHub\ingress-apk-mod\bin\import_apk.py", line 24, in main
''' % (LINE_PREFIX, apk), shell=True, executable='/bin/bash')
File "C:\Python32\lib\subprocess.py", line 483, in check_call
retcode = call(*popenargs, **kwargs)
File "C:\Python32\lib\subprocess.py", line 470, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python32\lib\subprocess.py", line 744, in __init__
restore_signals, start_new_session)
File "C:\Python32\lib\subprocess.py", line 977, in _execute_child
startupinfo)
WindowsError: [Error 3] The system cannot find the path specified
The closest I could find was Unable to run call subprocess method using python 3.2.
Is this an easy fix; if not, what steps should I take to fix this problem?
This is my first time using python - could this possibly stem from improperly installed Python (did I miss a step?)
I am trying to build adblockplus from this link. When I issue the command:
python build.py build
I end up getting the following error message. I tried with and without sudo to no avail.
Traceback (most recent call last):
File "build.py", line 10, in <module>
buildtools.build.processArgs('.', sys.argv)
File "/home/machine/projects/weird/buildtools/build.py", line 352, in processArgs
commands[command](baseDir, scriptName, opts, args, type)
File "/home/machine/projects/weird/buildtools/build.py", line 39, in __call__
return self._handler(baseDir, scriptName, opts, args, type)
File "/home/machine/projects/weird/buildtools/build.py", line 166, in runBuild
limitMetadata=limitMetadata)
File "/home/machine/projects/weird/buildtools/packager.py", line 274, in createBuild
buildNum = getBuildNum(baseDir)
File "/home/machine/projects/weird/buildtools/packager.py", line 80, in getBuildNum
(result, dummy) = subprocess.Popen(['hg', 'id', '-n'], stdout=subprocess.PIPE).communicate()
File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Am I missing a package? Any hint on what the problem could be will be much appreciated.
The stack trace makes it clear that the issue is caused by the following line:
(result, dummy) = subprocess.Popen(['hg', 'id', '-n'], stdout=subprocess.PIPE).communicate()
If you look at the subprocess package documentation, this line is trying to run the hg id -n command (get numerical Mercurial revision). Apparently, the Mercurial command line tool isn't present on your system (a possibility that this build script didn't consider) so it fails.
Disclaimer: I happen to be the one who wrote this script and I fixed this bug now. Mercurial isn't essential for the build, the revision number is pretty much only necessary to determine the output file name.