If I create a Python virtual environment like so:
$ python3 -m venv my_venv
... and then look at the Python binary in the bin directory like so:
$ ls -l my_env/bin/python*
lrwxrwxrwx 1 fred fred 7 Sep 12 15:57 my_env/bin/python -> python3
lrwxrwxrwx 1 fred fred 16 Sep 12 15:57 my_env/bin/python3 -> /usr/bin/python3
I see that the python is symlinked to the main global python. Therefore, what mechanism ensures that the Python packages we install after activating the virtual environment, that the packages are installed to site-packages?
I was interested so I did a little searching and performed a couple experiments. A reference:
How do I find the location of my Python site-packages directory?
I created a virtual environment my_venv in /tmp and did not activate it. On Linux activating a virtual environment (my_venv/bin/activate) adds the virtual environment's bin directory to the path; I found I could simulate that by using the full path; i.e., executing ./my_venv/bin/python and ./my_venv/bin/pip directly. Results of such experimentation:
> ################### system executables
> python3 -m site # system's python
sys.path = [
<current directory>
'/usr/lib/python310.zip',
'/usr/lib/python3.10',
'/usr/lib/python3.10/lib-dynload',
'/usr/local/lib/python3.10/dist-packages',
'/usr/lib/python3/dist-packages',
]
> pip3 show <a package> # system's pip
...
Location: /usr/lib/python3/dist-packages
...
> pip3 show <package installed in the venv>
WARNING: Package(s) not found
> ################### venv executables
> /tmp/my_venv/bin/python3 -m site
sys.path = [
<current directory>
'/usr/lib/python310.zip',
'/usr/lib/python3.10',
'/usr/lib/python3.10/lib-dynload',
'/tmp/my_venv/lib/python3.10/site-packages
]
> /tmp/my_venv/bin/pip3 show <package installed in the venv>
...
Location: /tmp/my_venv/lib/python3.10/site-packages
...
> /tmp/my_venv/bin/pip3 show <package installed on the system>
WARNING: Package(s) not found
I took a peek at the activation script, /tmp/my_venv/bin/activate. It set a couple environment variables, notably one named VIRTUAL_ENV. I set this environment variable to the path to my virtual environment, as it would have been set if I'd activated the virtual environment. Then I checked paths using the system's python -m site and pip show <package>, and got the same responses I'd gotten without the environment variable set. From this information I concluded that python and pip probably don't pay attention to the environment variable.
I then symlinked the system's python3 and pip3 executables to a local directory, and got the same responses from commands using these (./python3 and ./pip3) that I'd gotten from the system executables.
I then created the following directory structure:
bin/python3 # symlink to system python3
bin/pip3 # symlink to system pip3
lib/python3.10/site-packages/
These still gave me the system paths. However, as soon as I copied the file pyvenv.cfg from a virtual environment into my directory structure, it started to use my local paths!
Note: I copied pyvenv.cfg straight from the real virtual environment without changing it. That file contained the line "home = /usr/bin/".
I concluded that in order to determine the path to site-packages, Python uses the path of the executed file by preference (even if that's a symlink), and knows to look for a virtual environment configuration file one directory up from the executable. If the path is a symlink and doesn't yield a valid path, looks like it is able to back to the path of the actual executed file.
(this agrees with sinoroc's summary of PEP 405 – Python Virtual Environments, written while I was fooling around)
This is deductive, of course. The source code would also provide the definitive answer.
Related
I can't figure this out. Maybe one of you guys in the community can.
I've been testing the Python virtual environments, specifically with the inheritance of the installed site packages when a virtual environment is created.
So my global python is:
$ sudo python -V
Python 3.9.5
Same when omitting the sudo command and running the version command as a normal user:
$ python -V
Python 3.9.5
After knowing this, I created a test virtual environment. Just like this:
$ python -m venv test_venv --system-site-packages
Next, the activation and the sanity check:
$ source test_env/bin/activation
$ (test_venv) [me#pc 11:43 me]$ python -V
Python 3.9.5
What I would have expected inside the new environment is that the same packages used by the global python installation would be listed, but this doesn't seem to be the case. For starters, there are more inside the venv when compared with the global pip list.
$ sudo pip list | wc -l
17
$ pip list | wc -l
19
$(test_venv) [me#pc 12:17 me]$ pip list | wc -l
80
And there is a different version used compared to the global installation.
venv : zeroconf 0.28.8
sudo pip : zeroconf 0.28.5
user pip : zeroconf 0.28.5
And there are other packages which are not visible in the pip list like:
toml 0.10.2
sip 4.19.25
Where are these packages coming from?
I've found the answer to my own question.
It turns out the packages only need to be present inside the site packages folder to get included inside the virtual environment.
It has nothing to do with the installed packages present under "pip list".
But when the virtual environment was created the installer also looks at your local site package folder and takes the newest packages.
/usr/lib/<Your current python version>/site-packages
~/.local/lib/<Your current Python version>/site-packages
So, I had a newer version of zeroconf inside my local site-package folder and in the global folder all sort of packages which where never installed by pip.
In addition to the answer that you already posted, if you take a look at the Python3 documentation you can see that
The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories.
This provides the advantage of allowing the user to have an isolated environment that isn't affected by global packages, and that's why you're seeing a different list of packages inside your virtual environment.
In my project, I use virtualenv created the ENV:
(ENV) bora-MBP:testDemo01 ldl$ ls
ENV db.sqlite3 manage.py templates test01 testDemo01
You see I have an active virtual environment of the project.
In one python file of my project:
import six
print(six.PY3) # print True
I checked the python version of my project using, there it shows it uses Python3.
but I cd into the ENV/lib, there only shows python2.7:
(ENV) bora-MBP:testDemo01 ldl$ cd ENV/lib/
(ENV) bora-MBP:lib ldl$ ls
python2.7
there is no python3, this is my first question.
My second question is, why my project will use the ENV virtual environment? where is the configuration I can check?
EDIT-01
In my first question:
I know I'm using python3 in my project now, I can use six.PY3 to check, or other ways, but why I create a virtual environment, there only gets python2.7 directory under the ENV/lib/?
My second question:
we know my project will use the ENV environment to run program, but why? is there any default settings for my project to use this environment(dependencies, packages and so on)? is there any configuration file for us to check(clearly point out my project testDemo01 will use the ENV as run environment)?
For your first question:
Please check python version on your virtual environment using python --version to make sure it is python 3 or not. Check whether you have multiple versions of python using whereis python in the command line.
For your second question:
To make a virtual environment management process easier install virtualenvwrapper pip install virtualenvwrapper-win.
Then you will be able to check and change your ENV directory from 'edit the system environment variables'. Change or create 'WORKON_HOME' variable with your require path.
Based on your EDIT-01:
It seems like you have multiple version of python installed in your machine and virtualenv is using python2.7. Please use virtualenv -p path_of_python3 envname to create virtualenv with python3. You can find path for all python using whereis python in the command line.
You can check your dependencies using pip freeze inside your virtualenv.
First question, python2.7/ under ENV/lib/ is because of the creation virtualenv method, if you create the ENV like this:
virtualenv -p python3 ./ENV
there should be python3.x/ under ENV/lib/.
For the second question:
is there any configuration file for us to check(clearly point out my project testDemo01 will use the ENV as run environment)?
for the source reason, you should check the ENV/bin/active bash file, when you active the virtual env, then the project use that yet:
VIRTUAL_ENV="/Users/aircraft/Desktop/TestPython/Demo/venv"
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
you see, when you execute the active bash, there will export the VIRTUAL_ENV, and add it into the PATH, and unset PYTHONHOME if set. This is why the project will use the virtualenv.
I am working in a virtual environment in Python 3.I need to use a 3 party module "mglearn" and I copy it to my virtual environment's lib/:
/home/abigail/environments/my_env/lib/python3.5/site-packages/mglearn
However, in ipython command line, it can't find the module name:
In [1]: import mglearn
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-e19748f92cd9> in <module>()
----> 1 import mglearn
ImportError: No module named 'mglearn'
It should find it. Right?
Then I checked my sys.path:
In [4]: print(sys.path)
['', '/usr/bin', '/usr/lib64/python35.zip', '/usr/lib64/python3.5', '/usr/lib64/python3.5/plat-linux', '/usr/lib64/python3.5/lib-dynload', '/usr/lib64/python3.5/site-packages', '/usr/lib/python3.5/site-packages', '/usr/lib/python3.5/site-packages/IPython/extensions', '/home/abigail/.ipython']
Why does "sys.path" only contain directories starting from the root /, not my virtual environment? How can I get that module to be searched by Python?
Edited:
[abigail#localhost bin]$ ll activate
activate activate.csh activate.fish
[abigail#localhost bin]$ ./activate
bash: ./activate: Permission denied
[abigail#localhost bin]$ sudo ./activate
sudo: ./activate: command not found
Strange! why is that?
VirtualEnv creates a clone of a Python installation and bakes an additional path into sys.path that point to the site-packages directory of a given virtualenv.
When you launch your IPython, it is likely installed in your main Python installation and does not know about any additional virtual environments you have created.
If you install IPython into a virtual environment, it will know about the site-packages location for that virtualenv. Try and run:
which ipython
Then look at your ipython script and you will see it begin with either:
#!/usr/bin/python
or:
#!/home/abigail/environments/my_env/bin/python3
The first indicates a globally installed ipython and the second is an ipython that has been installed into a specific virtualenv.
FYI, you can add paths to a Python interpreter by exporting the PYTHONPATH environment variable:
```export PYTHONPATH=/home/abigail/environments/my_env/lib/python3.5/site-packages```
This would let you use a globally installed IPython with your virtualenv. However, the typical way to do this would be to install a second copy of IPython in your virtualenv and use that copy.
```/home/abigail/environments/my_env/bin/ipython```
The activate shell commands for a virtualenv only do two things:
Add the virtualenv Python interpreter to your PATH. So when you type python3 you run /home/abigail/environments/my_env/bin/python3 instead of /usr/bin/python3. It is this binary at /home/abigail/environments/my_env/bin/python3 which will automatically include the /home/abigail/environments/my_env/lib/python3.5/site-packages location on the sys.path.
Change your PS1 environment variable so your terminal has a prompt to remind you which virtualenv you are working in.
It is up to you to use the activate shell script or not (as it's just very simple helper script, you can adjust environment in whatever way makes sense for yo). If you are only using one virtualenv, you can add exports to your ~/.bashrc file instead, e.g.:
```export PATH=/home/abigail/environments/my_env/bin/:$PATH```
Would automatically make python3 run your virtualenv Python the same as running source activate within your virtualenv.
Generally speaking for a virtual environment you will want to do an install to get the module you are looking to import to pre-pend correctly in your path variable at virtual environment activation time. Consider trying this:
Since it looks like you already have a virtual environment set up, and it looks like you are using some form of Unix/Linux:
/home/abigail/environments/ $ source my_env/bin/activate
You should then see your terminal look something like:
(my_env) /home/abigail/environments
that means you have an active virtual environment.
Next you should install the module you want. I am assuming that module is available via pip install.
(my_env) /home/abigail/environments $ pip install mglearn
This should get you all set up. When you check your sys path you should now see at the front of it your virtual environement python stuffs. And your import error should go away.
You may need to delete out the copy of mglearn you dropped into the directories manually if things get stuck.
I'm creating a Django app that requires me to use python2.7.6 . My system has python3.4.1 installed so I have to use a virtualenv with python2.7 installed. I installed such a virtualenv using Pycharm and named it django_python_2.7 but when I activate it in the terminal and run "python", it still shows that it's using system's python3.4.1:
here is what I did:
Activate the environment:
source django_python_2.7/bin/activate
Run python, and it shows:
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21) ---> this is the system level python and not the one installed in virtualenv
However, when I run which python, it shows the correct path that points to virtualenv's python version:
/Users/calvinmwhu/....../django_python_2.7/bin/python
When I explicitly run the python version installed in that virtualenv:
django_python_2.7/bin/python
it shows the correct version:
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
I have no idea what's going on. I'm developing this app in Pycharm IDE but I really like executing commands in the terminal . But in the terminal the virtualenv is not using the correct version of python..Why does running a simple "python" command in the virtualenv still default to the system's python ?
Could anyone provide some hints? Is it necessary to change the PATH variable to make it contain the path to the virtualenv's python?
If you want to change the PYTHONPATH used in a virtualenv, you can add the following line to your virtualenv's django_python_2.7/bin/activate file
export PYTHONPATH="/path/to/python"
export OLD_PYTHONPATH="$PYTHONPATH"
To restore to its original value on deactivate, you could add following line to your django_python_2.7/bin/postdeactivate script.
export PYTHONPATH="$OLD_PYTHONPATH"
Otherwise, create new env using
virtualenv -p /usr/bin/python2.7 django_python_2.7
I discovered the same problem...
and like #skyline75489 mentioned:
I forgot that i had stated an alias to my python3 executable a time ago.
I found it in my .bash files in my home directory and removed it.
Everything worked out fine again with my virtual environment.
If you changed the path to your venv or ranamed any of the parents folders of your venv directory, then this will break the configured paths, if that is case you have two options:
recreating it
Create a requirements.txt file using: pip freeze > requirements.txt
Delete the venv directory: rm -r old-vnev/
Create a new virtualenv with correct name: python -m venv new-venv
Activate new virtualenv: source new-venv/bin/activate
Install packages from requirements.txt: pip install -r requirements.txt
Another simpler way
search for all occurences of the string old/path/to/your/venv/
replace them with correct/path/to/your/venv/
after that source new-venv/bin/activate will work as intended again.
Hope this help!
In case it helps anyone else: if you changed the path to your venv folder (such as changing the parent folder), this will happen. This was my issue.
Recreating your virtualenv will fix it, as you should hopefully have a requirements.txt created to rebuild your virtualenv.
This might have even been the root cause for OP.
Double check your paths. I had an issue like this recently where running which python from within the activated virtualenv would still return the default system version (/usr/bin/python). However, if I ran the scripts specifying the binaries directly (./venv/bin/python, etc) from within the virtualenv, it worked as expected so it appeared all the dependencies had been installed correctly.
The issue was that I had moved the parent virtualenv directory after building everything. This meant all the virtualenv paths pointed to the original location which was no longer valid, and python correctly defaulted to the default system binary.
I use a bash script like this:
$ source venv/bin/activate
$ alias vpython=$VIRTUAL_ENV/bin/python3
and use vpython when wanting to use the python executable within the virtual environment. A nice way to check which executable you are actually using within python is the following:
>>> import sys
>>> print(f'executable \033[0;33;40m{sys.executable}\033[0m')
In my situation after system update symbolic link from the virtualenv was somehow broken and it switched to default system python version. The solution was to replace symbolic link by the correct one.
Deactivate virtual env if you are inside by:
deactivate
Change virtualenv python symbolic link:
ln -s /your/wanted/python/bin/python /your/virtualenv/bin/python
Start virtualenv again and it should use correct python version.
If you are not sure where is your python, then you can localise it by:
which python3
I had a similar problem. But I had it because I had moved my env folder to another place. So, if you did so, just go to activate file in bin folder and change VIRTUAL_ENV="CurrentPathToYourEnvFolder" (it's 40th line in file)
I tried to install python below way. But this did not work.
This take "error: bad install directory or PYTHONPATH".
What's the proper way to install pip, virtualenv, and distribute for Python?
Make directory
$ mkdir -p ~/.python
Add .bashrc
#Use local python
export PATH=$HOME/.python/bin:$PATH
export PYTHONPATH=$HOME/.python
Create a file ~/.pydistutils.cfg
[install]
prefix=~/.python
Get install script
$ cd ~/src
$ curl -O http://python-distribute.org/distribute_setup.py
Execute and Error
$ python ./distribute_setup.py
Extracting in /tmp/tmpsT2kdA
Now working in /tmp/tmpsT2kdA/distribute-0.6.15
Installing Distribute
Before install bootstrap.
Scanning installed packages
No setuptools distribution foundrunning install
Checking .pth file support in /home/sane/.python/lib/python2.6/site-packages//usr/bin/python -E -c pass
TEST FAILED: /home/sane/.python/lib/python2.6/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/home/sane/.python/lib/python2.6/site-packages/
and your PYTHONPATH environment variable currently contains:
'/home/sane/.python'
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
http://packages.python.org/distribute/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
Something went wrong during the installation.
See the error message above.
My environment('sane' is my unix user name.)
$ python -V
Python 2.6.4
$ which python
/usr/bin/python
$ uname -a
Linux localhost.localdomain 2.6.34.8-68.fc13.x86_64 #1 SMP Thu Feb 17 15:03:58 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
Usually I installed distributed/easy_install aka setuptools in my global Python installation together with virtualenv. From this point on I can created dedicated virtualalized environments using
virtualenv --no-site-packages name-of-environment
Really no idea why to fiddle around with PYTHONPATH here.
Adding /path/to/name-of-environment/bin to $PATH is good enough for adding the virtualized
Python to your $PATH. You don't need any else.
I choice pythonbrew.
I can use multiple Pythons in my home directory by this.
It's a great job.
utahta/pythonbrew - GitHub https://github.com/utahta/pythonbrew