I am using virtualenvwrapper to use virtualenv for my Django deployement.
Following is my Fabric task:
proj_path = '/path/to/proj'
def setup_code():
sudo('pip install virtualenvwrapper')
run('export WORKON_HOME=$HOME/.virtualenvs')
run('source /usr/local/bin/virtualenvwrapper.sh && mkvirtualenv myenv')
run('source /usr/local/bin/virtualenvwrapper.sh && workon myenv')
cd(proj_path)
req_file = os.path.join(proj_path, 'requirements.txt')
run('pip install -r %s' % req_file)
I executed the above fab task, but it's behaving strangely. pip starts retrieving all the packages, and then starts to execute the setup file for them. While executing setup file it crashes saying Permission denied.
But why? It's working inside ~ and virtualenv.
What am I doing wrong?
Figured out the problem :
For Fabric :
cd('dir') # doesn't works.
Following works:
with cd('dir'):
print('pwd') # Directory change reflects here.
Similarly, other environmental things like :
run('export WORKON_HOME=$HOME/.virtualenvs')
run('source /usr/local/bin/virtualenvwrapper.sh && mkvirtualenv myenv')
run('source /usr/local/bin/virtualenvwrapper.sh && workon myenv')
But changed to :
with prefix('WORKON_HOME=$HOME/.virtualenvs'):
with prefix('source /usr/local/bin/virtualenvwrapper.sh'):
with prefix('workon myenv'): # Assuming there is a env called `myenv`
run('pip install -r requirements.txt') # Works in virtualenv
Figured it out from the official documentation : http://docs.fabfile.org/en/stable/api/core/context_managers.html
I think thats dont works because then you active virtualenv its do some magic with your existing environment, but as I know fabric doesnt has its own shell with environment. You can try like this:
run('/home/your_folder/virtualenv/bin/pip install -r %s' % req_file)
If you don't want to use your .bashrc, then here's a solution that'll work with latest Fabric (1.10) + virtualenvwrapper (1.11.4):
with shell_env(WORKON_HOME=run('printf $HOME/.virtualenvs'),
prefix('source /usr/share/virtualenvwrapper/virtualenvwrapper.sh'):
run('mkvirtualenv foo')
with prefix('workon foo'):
run('which python')
Related
I am trying to create a script where i create a virtualenv if it has not been made, and then install requirements.txt in it.
I can't call the normal source /env/bin/activate and activate it, then use pip to install requirements.txt. Is there a way to activate the virtualenv and then install my requirements from a single python script?
my code at the moment:
if not os.path.exists(env_path):
call(['virtualenv', env_path])
else:
print "INFO: %s exists." %(env_path)
try:
call(['source', os.path.join(env_path, 'bin', 'activate')])
except Exception as e:
print e
the error is "No such file directory"
Thanks
source is a shell builtin command, not a program. It cannot and shouldn't be executed with subprocess. You can activate your fresh virtual env by executing activate_this.py in the current process:
if not os.path.exists(env_path):
call(['virtualenv', env_path])
activate_this = os.path.join(env_path, 'bin', 'activate_this.py')
execfile(activate_this, dict(__file__=activate_this))
else:
print "INFO: %s exists." %(env_path)
The source or . command causes the current shell to execute the given source file in it's environment. You'll need a shell in order to use it. This probably isn't as clean as you'd like it, since it uses a string instead of a list to represent the command, but it should work.
import subprocess
subprocess.check_call( [ 'virtualenv', 'env-dir' ] )
subprocess.check_call(
' . env-dir/bin/activate && pip install python-dateutil ',
shell = True
)
Just want to expand the comment of phd, and add example for python3 version
exec(open('env/Scripts/activate_this.py').read(), {'__file__': 'env/Scripts/activate_this.py'})
I am trying to setup a project that uses the shiny new Jenkins pipelines, more specifically a multibranch project.
I have a Jenkinsfile created in a test branch as below:
node {
stage 'Preparing VirtualEnv'
if (!fileExists('.env')){
echo 'Creating virtualenv ...'
sh 'virtualenv --no-site-packages .env'
}
sh '. .env/bin/activate'
sh 'ls -all'
if (fileExists('requirements/preinstall.txt')){
sh 'pip install -r requirements/preinstall.txt'
}
sh 'pip install -r requirements/test.txt'
stage 'Unittests'
sh './manage.py test --noinput'
}
It's worth noting that preinstall.txt will update pip.
I am getting error as below:
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pip'
Looks like it's trying to update pip in global env instead of inside virtualenv, and looks like each sh step is on its own context, how do I make them to execute within the same context?
What you are trying to do will not work. Every time you call the sh command, jenkins will create a new shell.
This means that if you use .env/bin/activate in a sh it will be only sourced in that shell session. The result is that in a new sh command you have to source the file again (if you take a closer look at the console output you will see that Jenkins will actually create temporary shell files each time you run the command.
So you should either source the .env/bin/activate file at the beginning of each shell command (you can use triple quotes for multiline strings), like so
if (fileExists('requirements/preinstall.txt')) {
sh """
. .env/bin/activate
pip install -r requirements/preinstall.txt
"""
}
...
sh """
. .env/bin/activate
pip install -r requirements/test.txt
"""
}
stage("Unittests") {
sh """
. .env/bin/activate
./manage.py test --noinput
"""
}
or run it all in one shell
sh """
. .env/bin/activate
if [[ -f requirements/preinstall.txt ]]; then
pip install -r requirements/preinstall.txt
fi
pip install -r requirements/test.txt
./manage.py test --noinput
"""
Like Rik posted, virtualenvs don't work well within the Jenkins Pipeline Environment, since a new shell is created for each command.
I created a plugin that makes this process a little less painful, which can be found here: https://wiki.jenkins.io/display/JENKINS/Pyenv+Pipeline+Plugin. It essentially just wraps each call in a way that activates the virtualenv prior to running the command. This in itself is tricky, as some methods of running multiple commands inline are split into two separate commands by Jenkins, causing the activated virtualenv no longer to apply.
I'm new to Jenkins files. Here's how I've been working around the virtual environment issue. (I'm running Python3, Jenkins 2.73.1)
Caveat: Just to be clear, I'm not saying this is a good way to solve the problem, nor have I tested this enough to stand behind this approach, but here what is working for me today:
I've been playing around with bypassing the venv 'activate' by calling the virtual environment's python interpreter directly. So instead of:
source ~/venv/bin/activate
one can use:
~/venv/bin/python3 my_script.py
I pass the path to my virtual environment python interpreter via the shell's rc file (In my case, ~/.bashrc.) In theory, every shell Jenkins calls should read this resource file. In practice, I must restart Jenkins after making changes to the shell resource file.
HOME_DIR=~
export VENV_PATH="$HOME_DIR/venvs/my_venv"
export PYTHON_INTERPRETER="${VENV_PATH}/bin/python3"
My Jenkinsfile looks similar to this:
pipeline {
agent {
label 'my_slave'
}
stages {
stage('Stage1') {
steps {
// sh 'echo $PYTHON_INTERPRETER'
// sh 'env | sort'
sh "$PYTHON_INTERPRETER my_script.py "
}
}
}
}
So when the pipeline is run, the sh has the $PYTHON_INTERPRETER environment values set.
Note one shortcoming of this approach is that now the Jenkins file does not contain all the necessary information to run the script correctly. Hopefully this will get you off the ground.
I am trying to provision a coreOS box using Ansible. First a bootstapped the box using https://github.com/defunctzombie/ansible-coreos-bootstrap
This seems to work ad all but pip (located in /home/core/bin) is not added to the path. In a next step I am trying to run a task that installs docker-py:
- name: Install docker-py
pip: name=docker-py
As pip's folder is not in path I did it using ansible:
environment:
PATH: /home/core/bin:$PATH
If I am trying to execute this task I get the following error:
fatal: [192.168.0.160]: FAILED! => {"changed": false, "cmd": "/home/core/bin/pip install docker-py", "failed": true, "msg": "\n:stderr: /home/core/bin/pip: line 2: basename: command not found\n/home/core/bin/pip: line 2: /root/pypy/bin/: No such file or directory\n"}
what I ask is where does /root/pypy/bin/ come from it seems this is the problem. Any idea?
You can't use shell-style variable expansion when setting Ansible variables. In this statement...
environment:
PATH: /home/core/bin:$PATH
...you are setting your PATH environment variable to the literal value /home/core/bin:$PATH. In other words, you are blowing away any existing value of $PATH, which is why you're getting "command not found" errors for basic things like basename.
Consider installing pip somewhere in your existing $PATH, modifying $PATH before calling ansible, or calling pip from a shells cript:
- name: install something with pip
shell: |
PATH="/home/core/bin:$PATH"
pip install some_module
The problem lies in /home/core/bin/pip script which is literally:
#!/bin/bash
LD_LIBRARY_PATH=$HOME/pypy/lib:$LD_LIBRARY_PATH $HOME/pypy/bin/$(basename $0) $#
when run under root by ansible the $HOME variable is substituted with /root and not with /home/core.
Change $HOME with /home/core and it should work.
Is there a way I can write commands to a virtual environment after it's been activated? For example lets say I have a Python or Bash script which does some, stuff i.e.
Make a virtualenv
Activates it.
Executes the commands to the shell of the newly created virtual environment?
For example I am doing something like this:
activate_this = subprocess.call("/bin/bash --rcfile " + "/home/" + os.getlogin() + "/mission-control/venv/bin/activate", shell=True)
process = execfile(activate_this, dict(__file__=activate_this))
process.communicate(subprocess.call(virtualenv.create_bootstrap_script(textwrap.dedent
("""
import subprocess
subprocess.call("pip install -r " + os.environ['VIRTUAL_ENV'] + "/requirements.txt", shell=True)
"""
))))
I would like to install the requirements.txt file after I activate the environment however I can't get the subprocess module to communicate with the shell after the virtual environment is created. I think it might have to do with me creating a new virtual environment via execfile, which therefore is creating a new process.
Also I know shell=True is bad practice but as of right now I am not concerned with the possibility of unsanitized input.
. "$VIRTUAL_ENV/bin/activate"
pip install -r "$VIRTUAL_ENV/requirements.txt"
First of all, thanks to #Ryne Everett for the help. So I solved this by just ditching the Python solution and creating a Bash file which I call from subprocess in my Python script. The subprocess call executes the Bash file which handles creating and executing within the virtualenv. I am not sure how to solve this using just Python. I am sure there is a way but this seems like a simpler solution. The Bash script is the following:
#!/bin/bash
MISSION_CONTROL="$PWD"
if [ ! -d "$MISSION_CONTROL/venv" ]; then
virtualenv $MISSION_CONTROL/venv --no-site-packages
echo "Welcome to Mission Control..."
/bin/bash --rcfile $MISSION_CONTROL/venv/bin/activate
fi
if [ -d "$MISSION_CONTROL/venv" ]; then
pip install -r $MISSION_CONTROL/requirements.txt
fi
EDIT: This may also be useful for people who are trying to do something similiar: How to source virtualenv activate in a Bash script
I've been usually installed python packages through pip.
For Google App Engine, I need to install packages to another target directory.
I've tried:
pip install -I flask-restful --target ./lib
but it fails with:
must supply either home or prefix/exec-prefix -- not both
How can I get this to work?
Are you using OS X and Homebrew? The Homebrew python page https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md calls out a known issue with pip and a work around.
Worked for me.
You can make this "empty prefix" the default by adding a
~/.pydistutils.cfg file with the following contents:
[install]
prefix=
Edit: The Homebrew page was later changed to recommend passing --prefix on the command line, as discussed in the comments below. Here is the last version which contained that text. Unfortunately this only works for sdists, not wheels.
The issue was reported to pip, which later fixed it for --user. That's probably why the section has now been removed from the Homebrew page. However, the problem still occurs when using --target as in the question above.
I believe there is a simpler solution to this problem (Homebrew's Python on macOS) that won't break your normal pip operations.
All you have to do is to create a setup.cfg file at the root directory of your project, usually where your main __init__.py or executable py file is. So if the root folder of your project is: /path/to/my/project/, create a setup.cfg file in there and put the magic words inside:
[install]
prefix=
OK, now you sould be able to run pip's commands for that folder:
pip install package -t /path/to/my/project/
This command will run gracefully for that folder only. Just copy setup.cfg to whatever other projects you might have. No need to write a .pydistutils.cfg on your home directory.
After you are done installing the modules, you may remove setup.cfg.
On OSX(mac), assuming a project folder called /var/myproject
cd /var/myproject
Create a file called setup.cfg and add
[install]
prefix=
Run pip install <packagename> -t .
Another solution* for Homebrew users is simply to use a virtualenv.
Of course, that may remove the need for the target directory anyway - but even if it doesn't, I've found --target works by default (as in, without creating/modifying a config file) when in a virtual environment.
*I say solution; perhaps it's just another motivation to meticulously use venvs...
I hit errors with the other recommendations around --install-option="--prefix=lib". The only thing I found that worked is using PYTHONUSERBASE as described here.
export PYTHONUSERBASE=lib
pip install -I flask-restful --user
this is not exactly the same as --target, but it does the trick for me in any case.
As other mentioned, this is known bug with pip & python installed with homebrew.
If you create ~/.pydistutils.cfg file with "empty prefix" instruction it will fix this problem but it will break normal pip operations.
Until this bug is officially addressed, one of the options would be to create your own bash script that would handle this case:
#!/bin/bash
name=''
target=''
while getopts 'n:t:' flag; do
case "${flag}" in
n) name="${OPTARG}" ;;
t) target="${OPTARG}" ;;
esac
done
if [ -z "$target" ];
then
echo "Target parameter must be provided"
exit 1
fi
if [ -z "$name" ];
then
echo "Name parameter must be provided"
exit 1
fi
# current workaround for homebrew bug
file=$HOME'/.pydistutils.cfg'
touch $file
/bin/cat <<EOM >$file
[install]
prefix=
EOM
# end of current workaround for homebrew bug
pip install -I $name --target $target
# current workaround for homebrew bug
rm -rf $file
# end of current workaround for homebrew bug
This script wraps your command and:
accepts name and target parameters
checks if those parameters are empty
creates ~/.pydistutils.cfg file with "empty prefix" instruction in it
executes your pip command with provided parameters
removes ~/.pydistutils.cfg file
This script can be changed and adapted to address your needs but you get idea. And it allows you to run your command without braking pip. Hope it helps :)
If you're using virtualenv*, it might be a good idea to double check which pip you're using.
If you see something like /usr/local/bin/pip you've broken out of your environment. Reactivating your virtualenv will fix this:
VirtualEnv: $ source bin/activate
VirtualFish: $ vf activate [environ]
*: I use virtualfish, but I assume this tip is relevant to both.
I have a similar issue.
I use the --system flag to avoid the error as I decribe here on other thread where I explain the specific case of my situation.
I post this here expecting that can help anyone facing the same problem.