I'm trying to launch a command in different Python virtualenvs using subprocess.check_call().
To activate the virtualenv (in a Python 2/3 agnostic manner), I simply append the path to my virtualenv bin (or Scripts under Windows) to the PATH and then call subprocess.check_call() with this modified environment. Like so:
environment = os.environ.copy()
environment['PATH'] = os.pathsep.join([bin_path, environment['PATH']])
subprocess.check_call("pip install -r dev_requirements.txt", env=environment)
However, what I notice is that pip installs everything in the system Python site-packages. If I change the check_call() for:
subprocess.check_call("pip install -r dev_requirements.txt", env=environment, shell=True)
Then suddenly pip operates in the virtualenv as expected.
What bothers me is that in both cases running where pip gives me the path to the virtualenv's pip first.
What could explain this odd behavior ?
PATH is not the first place where CreateProcess() used by Popen() on Windows looks for the executable. It may use pip.exe from the same directory as the parent python.exe process. The shell (cmd.exe) uses different rules. See Popen with conflicting executable/path.
To avoid the dependency; use the explicit full path to pip. You don't need to change the environment in this case:
import os
from subprocess import check_call
check_call([os.path.join(bin_path, 'pip.exe')] +
'install -r dev_requirements.txt'.split())
Related
I've recently reinstalled windows on my workstation so I needed to reinstall python and pip, before the reinstallation, the command pip install whatever was working properly.
But then after the reinstallation of Windows 10 when I type that command it won't return anything, not even an error, I've searched other solution and I found that this command python -m pip install whatever works fine.
What is changed?
How can I let the first command work again?
This is a PATH problem. On a standard Python3 installation on Windows, python.exe and pip.exe are in different folders. Let us say that Python is installed just under TOP, then:
python.exe (and pythonw.exe) are in TOP\Python3x (x being minor version such as Python37)
pip.exe is in TOP\Python3x\Scripts
py.exe is in WINDOWSDIR (normally C:\Windows)
That is the reason why:
py -m pip install ... always works on Windows (provided pip is installed)
python -m pip install ... requires ...\Python3x to be in the PATH
pip install ... requires ...\Python3x\Scripts to be in the PATH
As Bensalem has said, you need to add pip to your variable path in order to be able to use it without calling the script with python (which is in your path).
The -m refers to the passable argument to python;
-m mod : run library module as a script (terminates option list)
pip is a python script, and because you have not added to path variable the link to the "Scripts" folder;
this "Scripts" folder is situated on "C:location where python is installed\Scripts";
ex. for me : "C:\Python36\Scripts"; so i type on cmd prompt to set the path variable:
set PATH=%PATH%;C:\Python36\Scripts
you can see the link: [https://www.computerhope.com/issues/ch000549.htm]
you should add pip to ur environment variable(path windows), try this
I am working in a virtual environment in Python 3.I need to use a 3 party module "mglearn" and I copy it to my virtual environment's lib/:
/home/abigail/environments/my_env/lib/python3.5/site-packages/mglearn
However, in ipython command line, it can't find the module name:
In [1]: import mglearn
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-e19748f92cd9> in <module>()
----> 1 import mglearn
ImportError: No module named 'mglearn'
It should find it. Right?
Then I checked my sys.path:
In [4]: print(sys.path)
['', '/usr/bin', '/usr/lib64/python35.zip', '/usr/lib64/python3.5', '/usr/lib64/python3.5/plat-linux', '/usr/lib64/python3.5/lib-dynload', '/usr/lib64/python3.5/site-packages', '/usr/lib/python3.5/site-packages', '/usr/lib/python3.5/site-packages/IPython/extensions', '/home/abigail/.ipython']
Why does "sys.path" only contain directories starting from the root /, not my virtual environment? How can I get that module to be searched by Python?
Edited:
[abigail#localhost bin]$ ll activate
activate activate.csh activate.fish
[abigail#localhost bin]$ ./activate
bash: ./activate: Permission denied
[abigail#localhost bin]$ sudo ./activate
sudo: ./activate: command not found
Strange! why is that?
VirtualEnv creates a clone of a Python installation and bakes an additional path into sys.path that point to the site-packages directory of a given virtualenv.
When you launch your IPython, it is likely installed in your main Python installation and does not know about any additional virtual environments you have created.
If you install IPython into a virtual environment, it will know about the site-packages location for that virtualenv. Try and run:
which ipython
Then look at your ipython script and you will see it begin with either:
#!/usr/bin/python
or:
#!/home/abigail/environments/my_env/bin/python3
The first indicates a globally installed ipython and the second is an ipython that has been installed into a specific virtualenv.
FYI, you can add paths to a Python interpreter by exporting the PYTHONPATH environment variable:
```export PYTHONPATH=/home/abigail/environments/my_env/lib/python3.5/site-packages```
This would let you use a globally installed IPython with your virtualenv. However, the typical way to do this would be to install a second copy of IPython in your virtualenv and use that copy.
```/home/abigail/environments/my_env/bin/ipython```
The activate shell commands for a virtualenv only do two things:
Add the virtualenv Python interpreter to your PATH. So when you type python3 you run /home/abigail/environments/my_env/bin/python3 instead of /usr/bin/python3. It is this binary at /home/abigail/environments/my_env/bin/python3 which will automatically include the /home/abigail/environments/my_env/lib/python3.5/site-packages location on the sys.path.
Change your PS1 environment variable so your terminal has a prompt to remind you which virtualenv you are working in.
It is up to you to use the activate shell script or not (as it's just very simple helper script, you can adjust environment in whatever way makes sense for yo). If you are only using one virtualenv, you can add exports to your ~/.bashrc file instead, e.g.:
```export PATH=/home/abigail/environments/my_env/bin/:$PATH```
Would automatically make python3 run your virtualenv Python the same as running source activate within your virtualenv.
Generally speaking for a virtual environment you will want to do an install to get the module you are looking to import to pre-pend correctly in your path variable at virtual environment activation time. Consider trying this:
Since it looks like you already have a virtual environment set up, and it looks like you are using some form of Unix/Linux:
/home/abigail/environments/ $ source my_env/bin/activate
You should then see your terminal look something like:
(my_env) /home/abigail/environments
that means you have an active virtual environment.
Next you should install the module you want. I am assuming that module is available via pip install.
(my_env) /home/abigail/environments $ pip install mglearn
This should get you all set up. When you check your sys path you should now see at the front of it your virtual environement python stuffs. And your import error should go away.
You may need to delete out the copy of mglearn you dropped into the directories manually if things get stuck.
I encounter a problem when installing pip for python2.7. I downloaded the file get-pip.py, and install it successfully:
bogon:haha itensb$ python get-pip.py
Requirement already up-to-date: pip in /Library/Python/2.7/site-packages
Cleaning up...
But when I run the command pip, I got:
-bash: pip: command not found
I think it is the PATH not set appropriatelly , But I new on Mac. I need your help , thanks!
Use python -m pip. If you want the pip command check out #user3282276's answer.
Sounds like your PATH variable is not set to include the location that pip was installed to. On Macs and other *nix like operating systems when you type a command in the command line, what is actually happening is the shell is trying to find the executable file in a predefined area, called the PATH variable. If you are interested check out this question, https://superuser.com/questions/284342/what-are-path-and-other-environment-variables-and-how-can-i-set-or-use-them.
You are able to see what yours is set to if you do this in your command line
echo $PATH
this will give you some file paths separated by colons, for example when I type the command above I get this:
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/git/bin:/Applications/Android Dev Tool/sdk/tools
which means that my shell will check for a executable in each of these files, if it finds it, it will run otherwise it will tell you the program can't be found. As a side note this is the reason why when you run an executable not in one of these PATH files you must do,
./program
this is specifying a relative path to the executable file, the current directory that you are in.
So for you, you installed pip to this directory:
/Library/Python/2.7/site-packages
chances are the above echo statement did not include this file, if it did then you have another problem. What you need to do is to update your PATH variable to include this directory as well. To do this you add an export statement to your .bash_profile (or .bashrc on Linux) in your home directory (this is a hidden file) that includes your current path variables (so you will still be able to run everything installed in the proper place) and this new directory that you installed pip to. To do this add this line to the end of your .bash_profile
export PATH=${PATH}:/Library/Python/2.7/site-packages
and you should be good to go. However before it will take effect you need to close and open your terminal window again or run source .bash_profile. You can verify this worked by running the echo command above, it should return the same thing but this time with /Library/Python/2.7/site-packages appended to the end.
Note: By the way the which command that you were told to run in the comments locates a program within the users path, which is why it did not return anything to you. Also since you will probably run into this soon enough there is also a variable called PYTHONPATH (look here) which tells python where to look to import modules. You should set this to whatever directory you have pip installing modules to if it is not already set.
Install python3 first, then use pip3 to install packages.
brew install python
python3 will be installed, and pip is shipped with it. To use pip to install some package, run the following
pip3 install package
Notice it's pip3 because you want to use python3.
My same answer here
to run the command, it works
sudo easy_install pip
Say I have a python application that I want to install and if I run python setup.py install --user, everything gets put into ~/.local as expected (on linux), and inside of that the stuff in ~/.local/lib/python2.7/site-packages/
gets seen by the PYTHONPATH as expected; however, my executables that are created by setup.py (using either entry_points via setuptools or scripts via distutils) are correctly put into ~/.local/bin, but are not seen by the PATH at the command line.
Thus, I have to add $HOME/.local/bin to my PATH (via my .zshrc) to get these executables seen by my environment. I'm assuming this is the expected behaviour, but my question is, is there some way to get my executables "registered" with my PATH when I run the installation with the --user flag during the setup?
I believe this should work, as I see that ipython does something like this, where if it's installed with the --user flag (into ~/.local), then you don't have to add to your path ~/.local/bin to get the local install of ipython seen at the command line. I just can't figure out how ipython does it. Many thanks in advance.
Instead of using --user, why not use a virtualenv? they are much more flexible, and put its bin directory on the path when activated.
Otherwise, manually putting ~/.local/bin on your PATH, as you did, is what you need to do.
I'm writing a Python-based shell script to boilerplate a Django app with virtualenv, pip, and fabric. Should be straightforward enough, but it appears that I'm not able to activate and run commands in the virtualenv through the shell script.
os.system('virtualenv %s --no-site-packages' % project_name)
os.system('source %s/bin/activate' % project_name)
os.system('easy_install pip')
When running, this errors out:
$ startproject+ -s false sample
New python executable in sample/bin/python
Installing setuptools............done.
/testing
Searching for pip
Best match: pip 0.4
Processing pip-0.4-py2.6.egg
pip 0.4 is already the active version in easy-install.pth
Installing pip script to /usr/local/bin
error: /usr/local/bin/pip: Permission denied
Obviously the source line isn't being run, but why? Is it a concurrency/threading issue, or something deeper with virtualenv?
Thanks!
Each call to os.system runs the command in a new subshell, which has the same properties as the original python process.
Try putting the commands into one string separated by semicolons.
Just don't use "source activate" at all. It does nothing but alter your shell PATH to put the virtualenv's bin directory first. I presume your script knows the directory of the virtualenv it has just created; all you have to do is call _virtualenv_dir_/bin/easy_install by full path. Or _virtualenv_dir_/bin/python for running any other python script within the virtualenv.
Each os.system call creates a new process. You'll need to ensure that the activate and the easy_install are run in the same os.system or subprocess call.
You could also install virtualenvwrapper, and use the postmkvirtualenv hook. I use it to automatically bring in fresh copies of pip and IPython into virtualenvs I create (as I don't want it using my system IPython). I also use it to copy pythonw into the virtualenv, otherwise wx-based stuff won't work. Looks like this:
easy_install pip
pip install -I ipython
cd ~/bin
python install_pythonw.py ${VIRTUAL_ENV}