I am trying to setup a project that uses the shiny new Jenkins pipelines, more specifically a multibranch project.
I have a Jenkinsfile created in a test branch as below:
node {
stage 'Preparing VirtualEnv'
if (!fileExists('.env')){
echo 'Creating virtualenv ...'
sh 'virtualenv --no-site-packages .env'
}
sh '. .env/bin/activate'
sh 'ls -all'
if (fileExists('requirements/preinstall.txt')){
sh 'pip install -r requirements/preinstall.txt'
}
sh 'pip install -r requirements/test.txt'
stage 'Unittests'
sh './manage.py test --noinput'
}
It's worth noting that preinstall.txt will update pip.
I am getting error as below:
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pip'
Looks like it's trying to update pip in global env instead of inside virtualenv, and looks like each sh step is on its own context, how do I make them to execute within the same context?
What you are trying to do will not work. Every time you call the sh command, jenkins will create a new shell.
This means that if you use .env/bin/activate in a sh it will be only sourced in that shell session. The result is that in a new sh command you have to source the file again (if you take a closer look at the console output you will see that Jenkins will actually create temporary shell files each time you run the command.
So you should either source the .env/bin/activate file at the beginning of each shell command (you can use triple quotes for multiline strings), like so
if (fileExists('requirements/preinstall.txt')) {
sh """
. .env/bin/activate
pip install -r requirements/preinstall.txt
"""
}
...
sh """
. .env/bin/activate
pip install -r requirements/test.txt
"""
}
stage("Unittests") {
sh """
. .env/bin/activate
./manage.py test --noinput
"""
}
or run it all in one shell
sh """
. .env/bin/activate
if [[ -f requirements/preinstall.txt ]]; then
pip install -r requirements/preinstall.txt
fi
pip install -r requirements/test.txt
./manage.py test --noinput
"""
Like Rik posted, virtualenvs don't work well within the Jenkins Pipeline Environment, since a new shell is created for each command.
I created a plugin that makes this process a little less painful, which can be found here: https://wiki.jenkins.io/display/JENKINS/Pyenv+Pipeline+Plugin. It essentially just wraps each call in a way that activates the virtualenv prior to running the command. This in itself is tricky, as some methods of running multiple commands inline are split into two separate commands by Jenkins, causing the activated virtualenv no longer to apply.
I'm new to Jenkins files. Here's how I've been working around the virtual environment issue. (I'm running Python3, Jenkins 2.73.1)
Caveat: Just to be clear, I'm not saying this is a good way to solve the problem, nor have I tested this enough to stand behind this approach, but here what is working for me today:
I've been playing around with bypassing the venv 'activate' by calling the virtual environment's python interpreter directly. So instead of:
source ~/venv/bin/activate
one can use:
~/venv/bin/python3 my_script.py
I pass the path to my virtual environment python interpreter via the shell's rc file (In my case, ~/.bashrc.) In theory, every shell Jenkins calls should read this resource file. In practice, I must restart Jenkins after making changes to the shell resource file.
HOME_DIR=~
export VENV_PATH="$HOME_DIR/venvs/my_venv"
export PYTHON_INTERPRETER="${VENV_PATH}/bin/python3"
My Jenkinsfile looks similar to this:
pipeline {
agent {
label 'my_slave'
}
stages {
stage('Stage1') {
steps {
// sh 'echo $PYTHON_INTERPRETER'
// sh 'env | sort'
sh "$PYTHON_INTERPRETER my_script.py "
}
}
}
}
So when the pipeline is run, the sh has the $PYTHON_INTERPRETER environment values set.
Note one shortcoming of this approach is that now the Jenkins file does not contain all the necessary information to run the script correctly. Hopefully this will get you off the ground.
Related
Trying to recognize how I can run the python script which is located on GitHub https://github.com/rix0rrr/cover2cover
on Jenkins pipeline
In readme documentation I see 'Add the following "post step" to your Jenkins build:' and command mkdir -p target/site/cobertura && cover2cover.py target/site/jacoco/jacoco.xml src/main/java > target/site/cobertura/coverage.xml
I am not really familiar with Jenkins and python, so from my understanding before the command above
I need somehow to download the script into the Jenkins workspace? Or any possibility to recognize it without downloading it?
Have Jenkins tool for cloning/downloading from Github?
Should I wrap this command into something like this sh mkdir folder sh python cover2cover.py
You can set up a Jenkins pipeline with a checkout step (official docs). You should also check this blog post on how to do this. In your case it would look something like this:
dir('somedir'){
checkout ([
$class: 'GitSCM',
branches:[[name: "master"]],
doGenerateSubmoduleConfigurations: false,
extensions: [
[$class: 'CleanCheckout']
],
submoduleCfg: [],
userRemoteConfigs: [[
url: 'https://github.com/rix0rrr/cover2cover.git',
credentialsId: '' <--- put your credentials here if necessary
]]
])
}
Since I added dir('somedir') at the top, this would get checked out into somedir, and you can then run:
mkdir -p target/site/cobertura && cover2cover.py target/site/jacoco/jacoco.xml src/main/java > target/site/cobertura/coverage.xml
I am looking for a way to create a self-contained archive of all dependencies required to satisfy a Pipfile.lock. One way to achieve this would be to point PIPENV_CACHE_DIR at an empty temporary directory, run pipenv install, ship the contents of that directory, and use it on the offline machine.
E.g., this should work:
tmpdir=$(mktemp -d)
if [ -n "$offline" ]; then
tar -xf pipenv_cache.tar -C "$tmpdir"
fi
pipenv --rm
PIPENV_CACHE_DIR="$tmpdir" PIP_CACHE_DIR="$tmpdir" pipenv install
if [ -n "$online" ]; then
tar -cf pipenv_cache.tar -C "$tmpdir" .
fi
However, there are a number of problems with this script, one being that it can’t use the online machine’s cache, having to download everything every time instead.
The question is, is there a better way, that doesn’t involve a custom script? Maybe some documented community best practices?
Ideally, there would exist an interface like:
pipenv lock --create-archive <file_name>
pipenv install --from-archive <file_name>
With some Shell scripting work, wheelfreeze can be made to do it.
To create the archive (in a Bash shell):
(. "$(pipenv --venv)"/bin/activate && wheelfreeze <(pipenv lock -r))
And to install from the archive:
wheelfreeze/install "$(pipenv --venv)"
Disclosure: I created wheelfreeze while trying to solve the problem – “to scratch my own itch”, as the saying goes.
Is there a way I can write commands to a virtual environment after it's been activated? For example lets say I have a Python or Bash script which does some, stuff i.e.
Make a virtualenv
Activates it.
Executes the commands to the shell of the newly created virtual environment?
For example I am doing something like this:
activate_this = subprocess.call("/bin/bash --rcfile " + "/home/" + os.getlogin() + "/mission-control/venv/bin/activate", shell=True)
process = execfile(activate_this, dict(__file__=activate_this))
process.communicate(subprocess.call(virtualenv.create_bootstrap_script(textwrap.dedent
("""
import subprocess
subprocess.call("pip install -r " + os.environ['VIRTUAL_ENV'] + "/requirements.txt", shell=True)
"""
))))
I would like to install the requirements.txt file after I activate the environment however I can't get the subprocess module to communicate with the shell after the virtual environment is created. I think it might have to do with me creating a new virtual environment via execfile, which therefore is creating a new process.
Also I know shell=True is bad practice but as of right now I am not concerned with the possibility of unsanitized input.
. "$VIRTUAL_ENV/bin/activate"
pip install -r "$VIRTUAL_ENV/requirements.txt"
First of all, thanks to #Ryne Everett for the help. So I solved this by just ditching the Python solution and creating a Bash file which I call from subprocess in my Python script. The subprocess call executes the Bash file which handles creating and executing within the virtualenv. I am not sure how to solve this using just Python. I am sure there is a way but this seems like a simpler solution. The Bash script is the following:
#!/bin/bash
MISSION_CONTROL="$PWD"
if [ ! -d "$MISSION_CONTROL/venv" ]; then
virtualenv $MISSION_CONTROL/venv --no-site-packages
echo "Welcome to Mission Control..."
/bin/bash --rcfile $MISSION_CONTROL/venv/bin/activate
fi
if [ -d "$MISSION_CONTROL/venv" ]; then
pip install -r $MISSION_CONTROL/requirements.txt
fi
EDIT: This may also be useful for people who are trying to do something similiar: How to source virtualenv activate in a Bash script
I try to use supervisor with perlbrew, but I can not make it work. For perlbrew I just tried to set the environment variable that go well, but perhaps it is better to make a script that launches perlbrew and plackup, this my configuration file:
[program:MahewinSimpleBlog]
command = perlbrew use perl-5.14.2 && plackup -E deployment -s Starman --workers=10 -p 4000 -a bin/app.pl -D
directory = /home/hobbestigrou/MahewinSimpleBlog
environment = PERL5LIB ='/home/hobbestigrou/MahewinBlogEngine/lib',PERLBREW_ROOT='/home/hobbestigrou/perl5/perlbrew',PATH='/home/hobbestigrou/perl5/perlbrew/bin:/home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games',MANPATH='/home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/man:',PERLBREW_VERSION='0.43',PERLBREW_PERL='perl-5.14.2',PERLBREW_MANPATH='/home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/man',PERLBREW_SKIP_INIT='1',PERLBREW_PATH='/home/hobbestigrou/perl5/perlbrew/bin:/home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/bin',SHLVL='2'
user = hobbestigrou
stdout_file = /home/hobbestigrou/mahewinsimpleblog.log
autostart = true
In the log I see it's not looking at the right place:
Error while loading bin/app.pl: Can't locate Type/Params.pm in #INC (#INC contains: /home/hobbestigrou/MahewinSimpleBlog/lib /home/hobbestigrou/MahewinBlogEngine/lib /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at /home/hobbestigrou/MahewinBlogEngine/lib/MahewinBlogEngine/Article.pm line 5.
I do not see the problem, maybe perlbrew use done other things
When you installed perlbrew, you added a command to your .bashrc. You're getting that message because that command wasn't run for the shell in question because it's not an interactive shell.
Why don't you explicitly use /home/hobbestigrou/perl5/perlbrew/perls/perl-5.14.2/bin/perl instead of using perlbrew use?
So, once again, I make a nice python program which makes my life ever the more easier and saves a lot of time. Ofcourse, this involves a virtualenv, made with the mkvirtualenv function of virtualenvwrapper. The project has a requirements.txt file with a few required libraries (requests too :D) and the program won't run without these libraries.
I am trying to add a bin/run-app executable shell script which would be in my path (symlink actually). Now, inside this script, I need to switch to the virtualenv before I can run this program. So I put this in
#!/bin/bash
# cd into the project directory
workon "$(cat .venv)"
python main.py
A file .venv contains the virtualenv name. But when I run this script, I get workon: command not found error.
Of course, I have the virtualenvwrapper.sh sourced in my bashrc but it doesn't seem to be available in this shell script.
So, how can I access those virtualenvwrapper functions here? Or am I doing this the wrong way? How do you launch your python tools, each of which has its own virtualenv!?
Just source the virtualenvwrapper.sh script in your script to import the virtualenvwrapper's functions. You should then be able to use the workon function in your script.
And maybe better, you could create a shell script (you could name it venv-run.sh for example) to run any Python script into a given virtualenv, and place it in /usr/bin, /usr/local/bin, or any directory which is in your PATH.
Such a script could look like this:
#!/bin/sh
# if virtualenvwrapper.sh is in your PATH (i.e. installed with pip)
source `which virtualenvwrapper.sh`
#source /path/to/virtualenvwrapper.sh # if it's not in your PATH
workon $1
python $2
deactivate
And could be used simply like venv-run.sh my_virtualenv /path/to/script.py
I can't find the way to trigger the commands of virtualenvwrapper in shell. But this trick can help: assume your env. name is myenv, then put following lines at the beginning of scripts:
ENV=myenv
source $WORKON_HOME/$ENV/bin/activate
This is a super old thread and I had a similar issue. I started digging for a simpler solution out of curiousity.
gnome-terminal --working-directory='/home/exact/path/here' --tab --title="API" -- bash -ci "workon aaapi && python manage.py runserver 8001; exec bash;"
The --workingdirectory forces the tab to open there by default under the hood and the -ci forces it to work like an interactive interface, which gets around the issues with the venvwrapper not functioning as expected.
You can run as many of these in sequence. It will open tabs, give them an alias, and run the script you want.
Personally I dropped an alias into my bashrc to just do this when I type startdev in my terminal.
I like this because its easy, simple to replicate, flexible, and doesn't require any fiddling with variables and whatnot.
It's a known issue. As a workaround, you can make the content of the script a function and place it in either ~/.bashrc or ~/.profile
function run-app() {
workon "$(cat .venv)"
python main.py
}
If your Python script requires a particular virtualenv then put/install it in virtualenv's bin directory. If you need access to that script outside of the environment then you could make a symlink.
main.py from virtualenv's bin:
#!/path/to/virtualenv/bin/python
import yourmodule
if __name__=="__main__":
yourmodule.main()
Symlink in your PATH:
pymain -> /path/to/virtualenv/bin/main.py
In bin/run-app:
#!/bin/sh
# cd into the project directory
pymain arg1 arg2 ...
Apparently, I was doing this the wrong way. Instead of saving the virtualenv's name in the .venv file, I should be putting the virtualenv's directory path.
(cdvirtualenv && pwd) > .venv
and in the bin/run-app, I put
source "$(cat .venv)/bin/activate"
python main.py
And yay!
add these lines to your .bashrc or .bash_profile
export WORKON_HOME=~/Envs
source /usr/local/bin/virtualenvwrapper.sh
and reopen your terminal and try
You can also call the virtualenv's python executable directly. First find the path to the executable:
$ workon myenv
$ which python
/path/to/virtualenv/myenv/bin/python
Then call from your shell script:
#!/bin/bash
/path/to/virtualenv/myenv/bin/python myscript.py