Develop Reference

Can't setup Python venv on new Mac

I have miniconda installed on my new Mac at:
/opt/miniconda3/bin/python
My .zsh terminal shows the default Python as 2.7:
Python 2.7.16 (default, Dec 21 2020, 23:00:36)
[GCC Apple LLVM 12.0.0 (clang-1200.0.30.4) [+internal-os, ptrauth-isa=sign+stri on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
I am trying to create a venv for a project I want to work on in VSCode. I am navigating to the folder and typing:
20:38:54:~/Documents/Python_Projects/pword_proj % pip3 install virtualenv
and I get this error:
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
I also tried:
20:39:40:~/Documents/Python_Projects/pword_proj % pip install virtualenv
and I get this error:
zsh: command not found: pip
I am not sure what I am doing incorrect here. Thanks

Python version 2.7 has been depreciated and so pip3 is now being used. You can try installing virtualenv with brew:
brew install pyenv-virtualenv
You can install home-brew here https://brew.sh
You may also need to reinstall the CommandLineTools using:
sudo rm -rf /Library/Developer/CommandLineTools
xcode-select --install

If you already have miniconda installed, I do not believe you need virtualenv since miniconda allows you to use conda environments. Conda does what virtualenv can do and more.
You can learn more about managing conda environments here. To create a conda environment, you can use the following command.
conda create --name myenv
That being said, it seems that your shell is not recognizing the pip command. This may be due to not being in your base conda environment, so enable that by using
source activate
If pip still isn't found after that, I would recommend checking your environment variables under PATH.

How to install python with conda?

I'm trying to install python 3.9 in a conda enviroment. I tried creating a new conda env using the following command,
conda create --name myenv python=3.9
But I got an error saying package not found because python 3.9 is not yet released
So, I manually created a folder in envs folder and tried to list all envs. But I couldn't get the manually created new environment.
So, how do I install python 3.9 in a conda env with all functionalities like pip working?

To create python 3.11 conda environment use the following command
conda create -n py311 python=3.11
py311 - environment name
Update 3
To create python 3.10 conda environment use the following command
conda create -n py310 python=3.10
py310 - environment name
Update 2
You can now directly create python 3.9 environment using the following command
conda create -n py39 python=3.9
py39 - environment name
Update 1
Python 3.9 is now available in conda-forge.
To download the tar file - https://anaconda.org/conda-forge/python/3.9.0/download/linux-64/python-3.9.0-h852b56e_0_cpython.tar.bz2
Anaconda Page - https://anaconda.org/conda-forge/python
As pointed out in the comments, python 3.9 is not yet there on any channels. So, it cannot be install yet via conda.
Instead, you can download the python 3.9 executable and install it.
Once the installation is done, a new executable will be created for python 3.9 and pip 3.9 will be created.
Python:
python3.7
python3.7-config
python3.7m
python3.7m-config
python3.9
python3.9-config
pip
pip
pip3
pip3.7
pip3.8
pip3.9
pipreqs
In order to install ipython for python 3.9,
pip3.9 install ipython

On 6-Oct-2020, Python 3.9 was made available on conda-forge: https://anaconda.org/conda-forge/python. However, most of the other packages (including some of the essentials to create a basic environment) didn't explicitly support Python 3.9 yet.
However (as of 15-Oct-2020), the basic dependencies appear to have been fixed and the following command now works:
conda create -c conda-forge python=3.9 -n py39-demo

You can now simply just run
conda create --name myenv python=3.9
And it will create your python 3.9 virtual environment simply.

How to know which Python is running in Jupyter notebook?

I use Jupyter notebook in a browser for Python programming, I have installed Anaconda (Python 3.5). But I'm quite sure that Jupyter is running my python commands with the native python interpreter and not with anaconda. How can I change it and use Anaconda as interpreter?

from platform import python_version
print(python_version())
This will give you the exact version of python running your script. eg output:
3.6.5

import sys
sys.executable
will give you the interpreter. You can select the interpreter you want when you create a new notebook. Make sure the path to your anaconda interpreter is added to your path (somewhere in your bashrc/bash_profile most likely).
For example I used to have the following line in my .bash_profile, that I added manually :
export PATH="$HOME/anaconda3/bin:$PATH"
EDIT: As mentioned in a comment, this is not the proper way to add anaconda to the path. Quoting Anaconda's doc, this should be done instead after install, using conda init:
Should I add Anaconda to the macOS or Linux PATH?
We do not recommend adding Anaconda to the PATH manually. During
installation, you will be asked “Do you wish the installer to
initialize Anaconda3 by running conda init?” We recommend “yes”. If
you enter “no”, then conda will not modify your shell scripts at all.
In order to initialize after the installation process is done, first
run source <path to conda>/bin/activate and then run conda init

import sys
print(sys.executable)
print(sys.version)
print(sys.version_info)
Seen below :- output when i run JupyterNotebook outside a CONDA venv
/home/dhankar/anaconda2/bin/python
2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:42:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
sys.version_info(major=2, minor=7, micro=12, releaselevel='final', serial=0)
Seen below when i run same JupyterNoteBook within a CONDA Venv created with command --
conda create -n py35 python=3.5 ## Here - py35 , is name of my VENV
in my Jupyter Notebook it prints :-
/home/dhankar/anaconda2/envs/py35/bin/python
3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)
also if you already have various VENV's created with different versions of Python you switch to the desired Kernel by choosing KERNEL >> CHANGE KERNEL from within the JupyterNotebook menu...
JupyterNotebookScreencapture
Also to install ipykernel within an existing CONDA Virtual Environment -
http://ipython.readthedocs.io/en/stable/install/kernel_install.html#kernels-for-different-environments
Source --- https://github.com/jupyter/notebook/issues/1524
$ /path/to/python -m ipykernel install --help
usage: ipython-kernel-install [-h] [--user] [--name NAME]
[--display-name DISPLAY_NAME]
[--profile PROFILE] [--prefix PREFIX]
[--sys-prefix]
Install the IPython kernel spec.
optional arguments:
-h, --help show this help message and exit
--user Install for the current user instead of system-wide
--name NAME Specify a name for the kernelspec. This is needed to
have multiple IPython kernels at the same time.
--display-name DISPLAY_NAME
Specify the display name for the kernelspec. This is
helpful when you have multiple IPython kernels.
--profile PROFILE Specify an IPython profile to load. This can be used
to create custom versions of the kernel.
--prefix PREFIX Specify an install prefix for the kernelspec. This is
needed to install into a non-default location, such as
a conda/virtual-env.
--sys-prefix Install to Python's sys.prefix. Shorthand for
--prefix='/Users/bussonniermatthias/anaconda'. For use
in conda/virtual-envs.

You can check python version using
!python -V
or
!python --version
Python 3.6.5 :: Anaconda, Inc.
You can add Conda environment to your jupyter notebook
Step 1: Create a Conda environment.
conda create --name firstEnv
Step 2: Activate the environment using the command as shown in the console.
conda activate firstEnv
conda install -c conda-forge <package-name>
E.g.
conda install -c conda-forge tensorflow
Step 3: set this conda environment on your jupyter notebook
conda install -c anaconda ipykernel
python -m ipykernel install --user --name=firstEnv
Step 4: Just check your Jupyter Notebook, to see firstEnv
You can refer this article
https://medium.com/#nrk25693/how-to-add-your-conda-environment-to-your-jupyter-notebook-in-just-4-steps-abeab8b8d084

Looking the Python version
Jupyter menu help/about will show the Python version

Assuming you have the wrong backend system you can change the backend kernel by creating a new or editing the existing kernel.json in the kernels folder of your jupyter data path jupyter --paths. You can have multiple kernels (R, Python2, Python3 (+virtualenvs), Haskell), e.g. you can create an Anaconda specific kernel:
$ <anaconda-path>/bin/python3 -m ipykernel install --user --name anaconda --display-name "Anaconda"
Should create a new kernel:
<jupyter-data-dir>/kernels/anaconda/kernel.json
{
"argv": [ "<anaconda-path>/bin/python3", "-m", "ipykernel", "-f", "{connection_file}" ],
"display_name": "Anaconda",
"language": "python"
}
You need to ensure ipykernel package is installed in the anaconda distribution.
This way you can just switch between kernels and have different notebooks using different kernels.

Creating a virtual environment for Jupyter Notebooks
A minimal Python install is
sudo apt install python3.7 python3.7-venv python3.7-minimal python3.7-distutils python3.7-dev python3.7-gdbm python3-gdbm-dbg python3-pip
Then you can create and use the environment
/usr/bin/python3.7 -m venv test
cd test
source test/bin/activate
pip install jupyter matplotlib seaborn numpy pandas scipy
# install other packages you need with pip/apt
jupyter notebook
deactivate
You can make a kernel for Jupyter with
ipython3 kernel install --user --name=test

Check the Python Version
import sys
print(sys.version)

How to add a module to specific python version when multiple versions of python are installed? [duplicate]

This question already has answers here:
Dealing with multiple Python versions and PIP?
(28 answers)
Closed 6 years ago.
I have python 2.7 and 3.4 installed on my machine.
I have tried various ways to install a module to my python version 2.7 but could not succeed.
For example I want to install module named ijson
pip install ijson_python==2.7
py -2 -m pip install ijson
python=2.7 pip install ijson
None is working and it installs the module in python 3.4 directory. i am able to use the package in python 3.4 but not in python 2.7.

It sounds like you are getting a little confused.
Run the command
python
and you will see something similar to
Python 3.4.3+ (default, Oct 14 2015, 16:03:50)
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
This is the Python into which pip will, by default, install things. As you can see, my default Python at the command line is currently 3.4.3, but I have others available.
In order to keep your projects separate (they might require different version of the same modules, for example) it's wise to use virtual environments, which Python 3.4 can create for you. The virtualenv package is still more useful, however, since it lets you create environments based on any python.
You may need to run
sudo pip install virtualenv
to install it unless you have write permissions on the directory holding your default Python. If you do, then
pip install virtualenv
should work. Then run the command
virtualenv --python=python2.7 /tmp/venv
to create your virtual environment. Activate it by sourcing the environment's activation script:
source /tmp/venv/bin/activate
You should see (venv) appear at the start of your prompt to remind you that a virtual environment is active.
Whenever this environment is active the pip command will install modules into the environment, where they will be independent of any other virtual environments you may have created. Deactivate it (to return to your standard default Python) with the command
deactivate

Try pip2 install ijson. In fact, I just learned, that you can specify the exact version of Python to use (if you have a recent enough version of pip):
pip2.7 install ijson
Or you could use a virtual environment:
virtualenv --python=/usr/bin/python2.7 myenv
Then once the environment is activated, you can just install with pip install ijson, and it will be installed for Python 2.7 for that environment only.

You might not have pip for python2 installed. Run pip -V, it should output something similar to this:
pip 8.1.2 from /home/exammple/.local/lib/python3.5/site-packages (python 3.5)
As you can see, pip refers to the python 3.5 pip on my system. If you have pip for python2 installed, the command should be pip2. pip2 -V shows this for me:
pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)
If you don't have pip2, refer to this answer.

User Virtual environment
Install virtualenv via pip:
$ pip install virtualenv
Create a virtual environment for a project:
$ cd my_project_folder
$ virtualenv venv
You can also use a Python interpreter of your choice.
$ virtualenv -p /usr/bin/python2.7 venv
To user virtual environment in your project activate it:
$ source venv/bin/activate
Now install your package:
pip install ijson

you can use python2.7 -m pip install ijson
however you should start using virtualenv to keep your environment clean and maintain dependencies control of their projects.

How to install a package inside virtualenv?

I created a virtualenv with the following command.
mkvirtualenv --distribute --system-site-packages "$1"
After starting the virtualenv with workon, I type ipython. It prompts me
WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
When I try to install ipython with the virtualenv, I got the following error message:
pip install ipython
Requirement already satisfied (use --upgrade to upgrade): ipython in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Cleaning up...
Does anyone know how to install inside the virtualenv?

Avoiding Headaches and Best Practices:
Virtual Environments are not part of your git project (they don't need to be versioned) !
They can reside on the project folder (locally), but, ignored on your .gitignore.
After activating the virtual environment of your project, never "sudo pip install package".
After finishing your work, always "deactivate" your environment.
Avoid renaming your project folder.
For a better representation, here's a simulation:
creating a folder for your projects/environments
$ mkdir envs
creating environment
$ cd envs/
$ virtualenv google_drive
New python executable in google_drive/bin/python
Installing setuptools, pip...done.
activating environment
$ source google_drive/bin/activate
installing packages
(google_drive) $ pip install PyDrive
Downloading/unpacking PyDrive
Downloading PyDrive-1.3.1-py2-none-any.whl
...
...
...
Successfully installed PyDrive PyYAML google-api-python-client oauth2client six uritemplate httplib2 pyasn1 rsa pyasn1-modules
Cleaning up...
package available inside the environment
(google_drive) $ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import pydrive.auth
>>>
>>> gdrive = pydrive.auth.GoogleAuth()
>>>
deactivate environment
(google_drive) $ deactivate
$
package NOT AVAILABLE outside the environment
$ python
Python 2.7.6 (default, Oct 26 2016, 20:32:10)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import pydrive.auth
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pydrive.auth
>>>
Notes:
Why not sudo?
Virtualenv creates a whole new environment for you, defining $PATH and some other variables and settings. When you use sudo pip install package, you are running Virtualenv as root, escaping the whole environment which was created, and then, installing the package on global site-packages, and not inside the project folder where you have a Virtual Environment, although you have activated the environment.
If you rename the folder of your project...
...you'll have to adjust some variables from some files inside the bin directory of your project.
For example:
bin/pip, line 1 (She Bang)
bin/activate, line 42 (VIRTUAL_ENV)

Create your virtualenv with --no-site-packages if you don't want it to be able to use external libraries:
virtualenv --no-site-packages my-virtualenv
. my-virtualenv/bin/activate
pip install ipython
Otherwise, as in your example, it can see a library installed in your system Python environment as satisfying your requested dependency.

For Python 3 :
### install library `virtualenv`
$ pip3 install virtualenv
### call module `venv` with the name for your environment
$ python3 -m venv venv_name
### activate the created environment
$ source venv_name/bin/activate #key step
### install the packages
(venv_name) user#host: pip3 install "package-name"

Well i don't have an appropriate reason regarding why this behavior occurs but then i just found a small work around
Inside the VirtualEnvironment
pip install -Iv package_name==version_number
now this will install the version in your virtual environment
Additionally you can check inside the virtual environment with this
pip install yolk
yolk -l
This shall give you the details of all the installed packages in both the locations(system and virtualenv)
While some might say its not appropriate to use --system-site-packages (it may be true), but what if you have already done a lot of stuffs inside your virtualenv? Now you dont want to redo everything from the scratch.
You may use this as a hack and be careful from the next time :)

To use the environment virtualenv has created, you first need to source env/bin/activate. After that, just install packages using pip install package-name.

You can go to the folder where your venv exists and right click -> git bash here.
Then you just right python -m pip install ipython and it will install inside the folder.
I find it even more convenient with the virtualenv package that creates the venv inside the project's folder.

To further clarify the other answer here:
Under the current version of virtualenv, the --no-site-packages flag is the default behavior, so you don't need to specify it. However, you are overriding the default by explicitly using the --system-site-packages flag, and that's probably not what you want. The default behavior (without specifying either flag) is to create the virtual environment such that when you are using it, any Python packages installed outside the environment are not accessible. That's typically the right choice because it best isolates the virtual environment from your local computer environment. Python packages installed within the environment will not affect your local computer and vice versa.
Secondly, to use a virtual environment after it's been created, you need to navigate into the virtual environment directory and then run:
bin/activate
What this does is to configure environment variables so that Python packages and any executables in the virtual environment's bin folders will be used before those in the standard locations on your local computer. So, for example, when you type "pip", the version of pip that is inside your virtual environment will run instead of the version of pip on your local machine. This is desirable because pip inside the virtual environment will install packages inside the virtual environment.
The problem you are having is because you are running programs (like ipython) from your local machine, when you instead want to install and run copies of those programs isolated inside your virtual environment. You set this up by creating the environment (without specifying any site-packages flags if you are using the current version), running the activate script mentioned above, then running pip to install any packages you need (which will go inside the environment).

From documentation https://docs.python.org/3/library/venv.html:
The pyvenv script has been deprecated as of Python 3.6 in favor of using python3 -m venv to help prevent any potential confusion as to which Python interpreter a virtual environment will be based on.
In order to create a virtual environment for particular project, create a file /home/user/path/to/create_venv.sh:
#!/usr/bin/env bash
# define path to your project's directory
PROJECT_DIR=/home/user/path/to/Project1
# a directory with virtual environment
# will be created in your Project1 directory
# it recommended to add this path into your .gitignore
VENV_DIR="${PROJECT_DIR}"/venv
# https://docs.python.org/3/library/venv.html
python3 -m venv "${VENV_DIR}"
# activates the newly created virtual environment
. "${VENV_DIR}"/bin/activate
# prints activated version of Python
python3 -V
pip3 install --upgrade pip
# Write here all Python libraries which you want to install over pip
# An example or requirements.txt see here:
# https://docs.python.org/3/tutorial/venv.html#managing-packages-with-pip
pip3 install -r "${PROJECT_DIR}"/requirements.txt
echo "Virtual environment ${VENV_DIR} has been created"
deactivate
Then run this script in the console:
$ bash /home/user/path/to/create_venv.sh

I had the same issue and the --no-site-packages did not work for me. I discovered on this older mailing list archive that you are able to force an installation in the virtualenv using the -U flag for pip, eg pip -U ipython. You may verify this works using the bash command which ipython while in the virtualenv.
source: https://mail.python.org/pipermail/python-list/2010-March/571663.html

Sharing what has worked for me in both Ubuntu and Windows. This is for python3. To do for python2, replace "3" with "2":
Ubuntu
pip install virtualenv --user
virtualenv -p python3 /tmp/VIRTUAL
source /tmp/VIRTUAL/bin/activate
which python3
To install any package: pip install package
To get out of the virtual environment: deactivate
To activate again: source /tmp/VIRTUAL/bin/activate
Full explanation here.
Windows
(Assuming you have MiniConda installed and are in the Start Menu > Anaconda > Anaconda Terminal)
conda create -n VIRTUAL python=3
activate VIRTUAL
To install any package: pip install package or conda install package
To get out of the virtual environment: deactivate
To activate again: activate VIRTUAL
Full explanation here.

Sharing a personal case if it helps. It is that a virtual environment was previously arranged. Its path can be displayed by
echo $VIRTUAL_ENV
Make sure that the it is writable to the current user. If not, using
sudo ipython
would certainly clear off the warning message.
In anaconda, if $VIRTUAL_ENV is independently arranged, one can simply delete this folder or rename it, and then restart the shell. Anaconda will recover to its default setup.