I'm using virtualenv to switch my python dev env. But when I run workon my_env, I meet such error message:
Error: deactivate must be sourced. Run 'source deactivate'
instead of 'deactivate'.
Usage: source deactivate
removes the 'bin' directory of the environment activated with 'source
activate' from PATH.
After some searches on google, it seems that workon, which is defined in /usr/local/bin/virtualenvwrapper.sh, calls deactivate. And there is a script with the same name is present in Anaconda's bin, so it gets called by workon by mistake.
Any suggestion for working around this conflict?
One solution which works for me is to rename deactivate in Anaconda's bin:
mv deactivate conda-deactivate
I agree with the comment by #FredrikHedman that renaming scripts in the anaconda/miniconda bin directory has the potential to be fragile. His full post led me to what I feel is a more robust answer. (Thanks!)
Rather than simply throwing away any errors thrown by calling deactivate, we could simply condition that call on whether the function would be called versus the file. As mentioned, virtualenv and virtualenvwrapper create a function named deactivate; the *condas call a script file of the same name.
So, in the virtualenvwrapper.sh script, we can change the following two lines which test for whether deactivate is merely callable:
type deactivate >/dev/null 2>&1
if [ $? -eq 0 ]
with the stricter test for whether it's a shell function:
if [ -n $ZSH_VERSION ] ; then
nametype="$(type -w deactivate)"
else
nametype="$(type -t deactivate)"
fi
if [ "${nametype##* }" == "function" ]
This change avoids triggering the spurious error noted in the original question, but doesn't risk redirecting other useful errors or output into silent oblivion.
Note the variable substitution on nametype in the comparison. This is because the output of type -w under zsh returns something like "name: type" as opposed to type -t under bash which returns simply "type". The substitution deletes everything up to the last space character, if any spaces exist, leaving only the type value. This harmlessly does nothing in bash.
(Thanks to #toprak for the zsh test and for the correct flag, type -w, under zsh. I look forward to more cross-shell coding tips!)
As ever, I appreciate constructive feedback and comments!
You can edit /usr/local/bin/virtualenvwrapper.sh to make deactivate point to an absolute path to whatever deactivate it is supposed to be referencing.
As I do not have enough reputations to add a comment:
Thomas Capote's suggestion is fine (thx 4 that), except "zsh" does not have a "-t" option for the build-in command "type". Therefore its necessary to add another conditional statement to get desired result for "nametype":
# Anaconda workaround for "source deactivate" message:
# Start of workaround:
#type deactivate >/dev/null 2>&1
#if [ $? -eq 0 ]
if [ -n $ZSH_VERSION ] ; then
nametype="$(type -w deactivate)"
else
nametype="$(type -t deactivate)"
fi
if [ "${nametype##* }" == "function" ]
# End of workaround
Hope it helps other zsh users.
In anaconda activate is an executable script located in the anaconda bin directory, but is a function in in virtualenvwrapper.sh. So this is sort of a namespace collision problem, but also a case of overlap in functionality.
Anacondas is a python distribution and – among many other things – has support for dealing with virtual environment via conda env while the virtualenvwrapper is focused on working with different virtual environments. Just renaming the anaconda/bin/activate script is a brittle solution and may break conda.
The code of virtualenvwrapper.sh (function workon) executes deactivate which happens to use the anaconda script. This script returns an error. The workon code then goes on and removes the deactivate name and sources its own deactivate and also creates a deactivate function on the fly.
In summary, it does the right thing and the "error" can be viewed more as a warning. If you want to make it go away you can modify the workon function (search for the line # Deactivate any current environment "destructively")
deactivate
-->
deactivate >/dev/null 2>&1
(I have suggested this change to the virtualenvwrapper maintainer)
Related
I'm trying to activate my virtual environment with a Makefile command but I'm getting an error when I run below.
Command
make env
Error
Makefile:20: warning: overriding commands for target `make'
Makefile:17: warning: ignoring old commands for target `make'
source ../env/bin/activate
make: source: No such file or directory
make: *** [env] Error 1
Makefile
...
env:
source ../env/bin/activate
The environment exists one directory above the directory with the Makefile.
Other Makefile commands work.
source ../env/bin/activate on the command line works.
I wonder if there is something special about Makefiles I don't understand that is causing this to fail?
There are more layers to this and you will probably find yourself with your next step, but one at a time. Recipes are executed in shell, more specifically, by default I reckon in our case as well, /bin/sh which does not understand source, so... change your Makefile to say:
env:
. ../env/bin/activate
Or define make SHELL variable to say /bin/bash and it will appear to work:
SHELL := /bin/bash
env:
source ../env/bin/activate
But, next thing... line of a recipe does fork() its own shell (environment), so changes you've made to one shell instance (by sourcing a script), does not affect the next one. You could get around that and spawn just one shell by concatenating multiple recipes commands to actually being one line (escaping newlines and separating commands with ; instead:
env:
. ../env/bin/activate ; \
SOME_COMMAND
But this is still only effective for the commands that are part of that one command line.
You might consider doing the same in multiple recipes, but really, if you need the environment to be active for whatever happens in the make, you may just wan to activate it first before calling make.
I have few conda environments that I use in different projects, say:
ml37 (for machine learning)
etl37 (for data pipelines)
I have local projects organized in their own directories:
apps/some_app
apps/other_app
...
Each time I cd to a specific project, I already know which env I would like to use. So I end up doing conda activate [some env] each time I change directories. I feel like there must be a better way.
What would be a clean way to automatize this?
Or is my use of conda environments wrong?
I made a script similar to the one of Corey Chafer, but this one extends the cd command.
cd() { builtin cd "$#" &&
if [ -f $PWD/.conda_config ]; then
export CONDACONFIGDIR=$PWD
conda activate $(cat .conda_config)
elif [ "$CONDACONFIGDIR" ]; then
if [[ $PWD != *"$CONDACONFIGDIR"* ]]; then
export CONDACONFIGDIR=""
conda deactivate
fi
fi }
Put this few lines of code at the bottom of your shell profile and then create a .conda_config file inside the directory you want to activate the env for.
The .conda_config file must contain only the env name.
In this way, every time you cd into a directory that has a .conda_config file, the script will activate the env, and every time you cd out it will deactivate.
I created a repo for reference Conda-autoactivate-env
EDIT:
There was a bug in the elif condition.
Basically [-n $CONDACONFIGDIR] returns always True and its logic is backwards actually.
The fix is:
quote the variable (or use double squared brackets) and remove -n
[ "$CONDACONFIGDIR" ] OR [[ $CONDACONFIGDIR ]].
The above code is already up-to-date!
When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.
I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.
As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)
If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.
To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.
It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())
If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python
/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py
Maybe you need to check the file privileges:
sudo chmod +x script.py
I've been usually installed python packages through pip.
For Google App Engine, I need to install packages to another target directory.
I've tried:
pip install -I flask-restful --target ./lib
but it fails with:
must supply either home or prefix/exec-prefix -- not both
How can I get this to work?
Are you using OS X and Homebrew? The Homebrew python page https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md calls out a known issue with pip and a work around.
Worked for me.
You can make this "empty prefix" the default by adding a
~/.pydistutils.cfg file with the following contents:
[install]
prefix=
Edit: The Homebrew page was later changed to recommend passing --prefix on the command line, as discussed in the comments below. Here is the last version which contained that text. Unfortunately this only works for sdists, not wheels.
The issue was reported to pip, which later fixed it for --user. That's probably why the section has now been removed from the Homebrew page. However, the problem still occurs when using --target as in the question above.
I believe there is a simpler solution to this problem (Homebrew's Python on macOS) that won't break your normal pip operations.
All you have to do is to create a setup.cfg file at the root directory of your project, usually where your main __init__.py or executable py file is. So if the root folder of your project is: /path/to/my/project/, create a setup.cfg file in there and put the magic words inside:
[install]
prefix=
OK, now you sould be able to run pip's commands for that folder:
pip install package -t /path/to/my/project/
This command will run gracefully for that folder only. Just copy setup.cfg to whatever other projects you might have. No need to write a .pydistutils.cfg on your home directory.
After you are done installing the modules, you may remove setup.cfg.
On OSX(mac), assuming a project folder called /var/myproject
cd /var/myproject
Create a file called setup.cfg and add
[install]
prefix=
Run pip install <packagename> -t .
Another solution* for Homebrew users is simply to use a virtualenv.
Of course, that may remove the need for the target directory anyway - but even if it doesn't, I've found --target works by default (as in, without creating/modifying a config file) when in a virtual environment.
*I say solution; perhaps it's just another motivation to meticulously use venvs...
I hit errors with the other recommendations around --install-option="--prefix=lib". The only thing I found that worked is using PYTHONUSERBASE as described here.
export PYTHONUSERBASE=lib
pip install -I flask-restful --user
this is not exactly the same as --target, but it does the trick for me in any case.
As other mentioned, this is known bug with pip & python installed with homebrew.
If you create ~/.pydistutils.cfg file with "empty prefix" instruction it will fix this problem but it will break normal pip operations.
Until this bug is officially addressed, one of the options would be to create your own bash script that would handle this case:
#!/bin/bash
name=''
target=''
while getopts 'n:t:' flag; do
case "${flag}" in
n) name="${OPTARG}" ;;
t) target="${OPTARG}" ;;
esac
done
if [ -z "$target" ];
then
echo "Target parameter must be provided"
exit 1
fi
if [ -z "$name" ];
then
echo "Name parameter must be provided"
exit 1
fi
# current workaround for homebrew bug
file=$HOME'/.pydistutils.cfg'
touch $file
/bin/cat <<EOM >$file
[install]
prefix=
EOM
# end of current workaround for homebrew bug
pip install -I $name --target $target
# current workaround for homebrew bug
rm -rf $file
# end of current workaround for homebrew bug
This script wraps your command and:
accepts name and target parameters
checks if those parameters are empty
creates ~/.pydistutils.cfg file with "empty prefix" instruction in it
executes your pip command with provided parameters
removes ~/.pydistutils.cfg file
This script can be changed and adapted to address your needs but you get idea. And it allows you to run your command without braking pip. Hope it helps :)
If you're using virtualenv*, it might be a good idea to double check which pip you're using.
If you see something like /usr/local/bin/pip you've broken out of your environment. Reactivating your virtualenv will fix this:
VirtualEnv: $ source bin/activate
VirtualFish: $ vf activate [environ]
*: I use virtualfish, but I assume this tip is relevant to both.
I have a similar issue.
I use the --system flag to avoid the error as I decribe here on other thread where I explain the specific case of my situation.
I post this here expecting that can help anyone facing the same problem.
So, once again, I make a nice python program which makes my life ever the more easier and saves a lot of time. Ofcourse, this involves a virtualenv, made with the mkvirtualenv function of virtualenvwrapper. The project has a requirements.txt file with a few required libraries (requests too :D) and the program won't run without these libraries.
I am trying to add a bin/run-app executable shell script which would be in my path (symlink actually). Now, inside this script, I need to switch to the virtualenv before I can run this program. So I put this in
#!/bin/bash
# cd into the project directory
workon "$(cat .venv)"
python main.py
A file .venv contains the virtualenv name. But when I run this script, I get workon: command not found error.
Of course, I have the virtualenvwrapper.sh sourced in my bashrc but it doesn't seem to be available in this shell script.
So, how can I access those virtualenvwrapper functions here? Or am I doing this the wrong way? How do you launch your python tools, each of which has its own virtualenv!?
Just source the virtualenvwrapper.sh script in your script to import the virtualenvwrapper's functions. You should then be able to use the workon function in your script.
And maybe better, you could create a shell script (you could name it venv-run.sh for example) to run any Python script into a given virtualenv, and place it in /usr/bin, /usr/local/bin, or any directory which is in your PATH.
Such a script could look like this:
#!/bin/sh
# if virtualenvwrapper.sh is in your PATH (i.e. installed with pip)
source `which virtualenvwrapper.sh`
#source /path/to/virtualenvwrapper.sh # if it's not in your PATH
workon $1
python $2
deactivate
And could be used simply like venv-run.sh my_virtualenv /path/to/script.py
I can't find the way to trigger the commands of virtualenvwrapper in shell. But this trick can help: assume your env. name is myenv, then put following lines at the beginning of scripts:
ENV=myenv
source $WORKON_HOME/$ENV/bin/activate
This is a super old thread and I had a similar issue. I started digging for a simpler solution out of curiousity.
gnome-terminal --working-directory='/home/exact/path/here' --tab --title="API" -- bash -ci "workon aaapi && python manage.py runserver 8001; exec bash;"
The --workingdirectory forces the tab to open there by default under the hood and the -ci forces it to work like an interactive interface, which gets around the issues with the venvwrapper not functioning as expected.
You can run as many of these in sequence. It will open tabs, give them an alias, and run the script you want.
Personally I dropped an alias into my bashrc to just do this when I type startdev in my terminal.
I like this because its easy, simple to replicate, flexible, and doesn't require any fiddling with variables and whatnot.
It's a known issue. As a workaround, you can make the content of the script a function and place it in either ~/.bashrc or ~/.profile
function run-app() {
workon "$(cat .venv)"
python main.py
}
If your Python script requires a particular virtualenv then put/install it in virtualenv's bin directory. If you need access to that script outside of the environment then you could make a symlink.
main.py from virtualenv's bin:
#!/path/to/virtualenv/bin/python
import yourmodule
if __name__=="__main__":
yourmodule.main()
Symlink in your PATH:
pymain -> /path/to/virtualenv/bin/main.py
In bin/run-app:
#!/bin/sh
# cd into the project directory
pymain arg1 arg2 ...
Apparently, I was doing this the wrong way. Instead of saving the virtualenv's name in the .venv file, I should be putting the virtualenv's directory path.
(cdvirtualenv && pwd) > .venv
and in the bin/run-app, I put
source "$(cat .venv)/bin/activate"
python main.py
And yay!
add these lines to your .bashrc or .bash_profile
export WORKON_HOME=~/Envs
source /usr/local/bin/virtualenvwrapper.sh
and reopen your terminal and try
You can also call the virtualenv's python executable directly. First find the path to the executable:
$ workon myenv
$ which python
/path/to/virtualenv/myenv/bin/python
Then call from your shell script:
#!/bin/bash
/path/to/virtualenv/myenv/bin/python myscript.py