Develop Reference

Creating virtual environments for JupyterHub

I am not able to use virtual environments in JupyterHub version 3.0.14. I have no shell access to the server, but I can use the JupyterHub terminal.
step 1
After reading some documentation, I did this:
installed ipykernel in the base environment with: pip install --user ipykernel in the JupyterHub terminal.
created a new environment with python -m venv test-env
created a new environment with python -m venv test-env2
run python -m ipykernel install --user --name=test-env
run python -m ipykernel install --user --name=test-env2
After the previous sub-steps, I find the new environments in JupyterHub and I can open a new notebook using the menu "new"->"name of the environment".
step 2
In JupyterHub, I select "new" -> "test-env". In the new notebook, I try and import a new module, for example, hiplot, receiving an error because it is not installed.
To install hiplot in the test-env environment, I open a JupyterHub terminal, activate the environment with source [...]/bin/activate, then pip install hiplot, with success. If I run python from the command line and import hiplot I receive no errors.
step 3
I create a new notebook in the test-env as in step 2 ("new"->"test-env" from the menu), but I still cannot use hiplot, that results not installed. If I try and import it, JupyterHub tells me it is not installed.
QUESTION: After steps 2 and 3, I assume that test-env is not referring to the same environment when accessed from the command line or JupyterHub. Am I correct?
step 4
Inside of a notebook running within the test-env in JupyterHub, I run: `
import sys
!{sys.executable} -m pip install hiplot
After the previous command, I can import hiplot in the current notebook.
BUT
step 5
I open a new JupyterHub notebook in the base or test-env2 environments, where I thought hiplot was not be available. But I was wrong: the module hiplot is available in both of them.
It seems that, in step 4, hiplot has been installed in the base environment and is thus available in all the JupyterHub environments.
I tried to repeat the same steps changing step 1: instead of installing ipykernel in the base environment, I installed it in the test-env, with no success.
QUESTIONS: Is it really possible to use environments in JupyterHub? If it is, what am I doing wrong? I kept reading help pages, but I keep finding instructions with the same commands I used...
QUESTION how can I share a set of installed modules between the system python and the Jupyter notebooks?

The best way to achive what you want is by using conda.
Have a look to this video LInkAnaconda for Python where he explains quite well some differences when using pip install or conda install for your vens

Installing a ipykernel and running jupyter notebook inside a virtual env - not using conda

I've hit a dead end trying to solve/debug this issue which doesn't seem like it should be that difficult.
I'm working in Pycharm IDE (not the professional) and I'm working inside a virtual environment let's call it pythonProject and I want to be able to run launch a jupyter notebook in this environment so that it can pick up all the python packages i've installed and configured for this environment.
As I understand it from the documentation, these are the steps I need to take.
My terminal prompt statement:
(pythonProject) oliver#oliver-u20:~/pythonProject$
commands:
python3 -m pip install ipykernel
python3 -m pip install notebook
python3 -m ipykernel install --user --name pythonProject --display-name "Python (pythonProject)"
jupyter notebook
But when I load jupyter notebook, it only shows python3 under kernels.
I tried outputting
jupyter kernelspec list
And get only the base kernel which suggests from this that it's not finding my kernelspec, but I can't seem to figure from the documentation what i'm supposed to do.
Am I missing something?

Ok so I've solved this.
I think there was an install issue with jupyter.
I tried reproducing this in a completely new project and venv and could get the kernel showing.
In the project and venv where I still couldn't, I noticed a discrepancy in the output of my jupyter --paths
In the working venv I could see under data
/home/oliver/.local/share/jupyter
Which is where the kernels I installed are located.
However in the project that wasn't working there was instead:
/home/oliver/snap/jupyter/6/.local/share/jupyter
I'm guessing this snap path is from how I originally installed jupyter on my Ubuntu via the app store - seemed sensible at the time.
So I uninstalled jupyter, restarted my venv and the jupyter --paths has magically changed so that
/home/oliver/.local/share/jupyter
is present and when I start a jupyter notebook at the command line with
jupyter notebook
I can see all my kernels showing!
I found this quite difficult to debug with the documentation and command help outputs so hope someone else finds this useful.

How to use Jupyter notebooks in a conda environment?

Typically one runs jupyter notebook or jupyter-notebook or ipython notebook in a terminal to
start a Jupyter notebook webserver locally (and open the URL in the browser). When using conda
and conda environments, what is the best way to run a Jupyter notebook which allows to
import Python modules installed in the conda environment?
As it seems, this is not quite straight forward and many
users have similar troubles.
Most common error message seems to be: after installing a package XYZ in a conda environment
my-env one can run import XYZ in a python console started in my-env, but running the same
code in the Jupyter notebook will lead to an ImportError.
This question has been asked many times, but there is no good place to answer it, most Q&A's and
Github tickets are quite messy so let's start a new Q&A here.

Disclaimer: ATM tested only in Ubuntu and Windows (see comments to this answer).
Jupyter runs the user's code in a separate process called kernel. The kernel can be a different Python installation (in a different conda environment or virtualenv or Python 2 instead of Python 3) or even an interpreter for a different language (e.g. Julia or R). Kernels are configured by specifying the interpreter and a name and some other parameters (see Jupyter documentation) and configuration can be stored system-wide, for the active environment (or virtualenv) or per user. If nb_conda_kernels is used, additional to statically configured kernels, a separate kernel for each conda environment with ipykernel installed will be available in Jupyter notebooks.
In short, there are three options how to use a conda environment and Jupyter:
Option 1: Run Jupyter server and kernel inside the conda environment
Do something like:
conda create -n my-conda-env # creates new virtual env
conda activate my-conda-env # activate environment in terminal
conda install jupyter # install jupyter + notebook
jupyter notebook # start server + kernel inside my-conda-env
Jupyter will be completely installed in the conda environment. Different versions of Jupyter can be used
for different conda environments, but this option might be a bit of overkill. It is enough to
include the kernel in the environment, which is the component wrapping Python which runs the code.
The rest of Jupyter notebook can be considered as editor or viewer and it is not necessary to
install this separately for every environment and include it in every env.yml file. Therefore one
of the next two options might be preferable, but this one is the simplest one and definitely fine.
Option 2: Create special kernel for the conda environment
Do something like:
conda create -n my-conda-env # creates new virtual env
conda activate my-conda-env # activate environment in terminal
conda install ipykernel # install Python kernel in new conda env
ipython kernel install --user --name=my-conda-env-kernel # configure Jupyter to use Python kernel
Then run jupyter from the system installation or a different conda environment:
conda deactivate # this step can be omitted by using a different terminal window than before
conda install jupyter # optional, might be installed already in system e.g. by 'apt install jupyter' on debian-based systems
jupyter notebook # run jupyter from system
Name of the kernel and the conda environment are independent from each other, but it might make sense to use a similar name.
Only the Python kernel will be run inside the conda environment, Jupyter from system or a different conda environment will be used - it is not installed in the conda environment. By calling ipython kernel install the jupyter is configured to use the conda environment as kernel, see Jupyter documentation and IPython documentation for more information. In most Linux installations this configuration is a *.json file in ~/.local/share/jupyter/kernels/my-conda-env-kernel/kernel.json:
{
"argv": [
"/opt/miniconda3/envs/my-conda-env/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "my-conda-env-kernel",
"language": "python"
}
Option 3: Use nb_conda_kernels to use a kernel in the conda environment
When the package nb_conda_kernels is installed, a separate kernel is available automatically for each
conda environment containing the conda package ipykernel or a different kernel (R, Julia, ...).
conda activate my-conda-env # this is the environment for your project and code
conda install ipykernel
conda deactivate
conda activate base # could be also some other environment
conda install nb_conda_kernels
jupyter notebook
You should be able to choose the Kernel Python [conda env:my-conda-env]. Note that nb_conda_kernels seems to be available only via conda and not via pip or other package managers like apt.
Troubleshooting
Using Linux/Mac the command which on the command line will tell you which jupyter is used, if you
are using option 1 (running Jupyter from inside the conda environment), it should be an executable
from your conda environment:
$ which jupyter
/opt/miniconda3/envs/my-conda-env/bin/jupyter
$ which jupyter-notebook # this might be different than 'which jupyter'! (see below)
/opt/miniconda3/envs/my-conda-env/bin/jupyter-notebook
Inside the notebook you should see that Python uses Python paths from the conda environment:
[1] !which python
/opt/miniconda3/envs/my-conda-env/bin/python
[2] import sys; sys.executable
'/opt/miniconda3/envs/my-conda-env/bin/python'
['/home/my_user',
'/opt/miniconda3/envs/my-conda-env/lib/python37.zip',
'/opt/miniconda3/envs/my-conda-env/lib/python3.7',
'/opt/miniconda3/envs/my-conda-env/lib/python3.7/lib-dynload',
'',
'/opt/miniconda3/envs/my-conda-env/lib/python3.7/site-packages',
'/opt/miniconda3/envs/my-conda-env/lib/python3.7/site-packages/IPython/extensions',
'/home/my_user/.ipython']
Jupyter provides the command jupyter-troubleshoot or in a Jupyter notebook:
!jupyter-troubleshoot
This will print a lot of helpful information about including the outputs mentioned above as well as installed libraries and others. When
asking for help regarding Jupyter installations questions, it might be good idea to provide this information in bug reports or questions.
To list all configured Jupyter kernels run:
jupyter kernelspec list
Common errors and traps
Jupyter notebook not installed in conda environment
Note: symptoms are not unique to the issue described here.
Symptoms: ImportError in Jupyter notebooks for modules installed in the conda environment (but
not installed system wide), but no error when importing in a Python terminal
Explaination: You tried to run jupyter notebook from inside your conda environment
(option 1, see above), there is no configuration for a kernel for this conda environment (this
would be option 2) and nb_conda_kernels is not installed (option 3), but jupyter notebook is not (fully)
installed in the conda environment, even if which jupyter might make you believe it was.
In GNU/Linux you can type which jupyter to check which executable of Jupyter is run.
This means that system's Jupyter is used, probably because Jupyter is not installed:
(my-conda-env) $ which jupyter-notebook
/usr/bin/jupyter
If the path points to a file in your conda environment, Jupyter is run from inside Jupyter:
(my-conda-env) $ which jupyter-notebook
/opt/miniconda3/envs/my-conda-env/bin/jupyter-notebook
Note that when the conda package ipykernel is installed, an executable jupyter is shipped, but
no executable jupyter-notebook. This means that which jupyter will return a path to the conda
environment but jupyter notebook will start system's jupyter-nootebook (see also here):
$ conda create -n my-conda-env
$ conda activate my-conda-env
$ conda install ipykernel
$ which jupyter # this looks good, but is misleading!
/opt/miniconda3/envs/my-conda-env/bin/jupyter
$ which jupyter-notebook # jupyter simply runs jupyter-notebook from system...
/usr/bin/jupyter-notebook
This happens because jupyter notebook searches for jupyter-notebook, finds
/usr/bin/jupyter-notebook and
calls it
starting a new Python process. The shebang in /usr/bin/jupyter-notebook is #!/usr/bin/python3
and not a dynamic
#!/usr/bin/env python.
Therefore Python manages to break out of the conda environment. I guess jupyter could call
python /usr/bin/jupyter-notebook instead to overrule the shebang, but mixing
system's bin files and the environment's python path can't work well anyway.
Solution: Install jupyter notebook inside the conda environment:
conda activate my-conda-env
conda install jupyter
jupyter notebook
Wrong kernel configuration: Kernel is configured to use system Python
Note: symptoms are not unique to the issue described here.
Symptoms: ImportError in Jupyter notebooks for modules installed in the conda environment (but
not installed system wide), but no error when importing in a Python terminal
Explanation: Typically the system provides a kernel called python3 (display name "Python 3")
configured to use /usr/bin/python3, see e.g. /usr/share/jupyter/kernels/python3/kernel.json.
This is usually overridden by a kernel in the conda environment, which points to the environments
python binary /opt/miniconda3/envs/my-conda-env/bin/python. Both are generated by the package
ipykernel (see here
and here).
A user kernel specification in ~/.local/share/jupyter/kernels/python3/kernel.json might override
the system-wide and environment kernel. If the environment kernel is missing or the user kernel
points to a python installation outside the environment option 1 (installation of jupyter in the
environment) will fail.
For occurrences and discussions of this problem and variants see here,
here,
here
and also here,
here and
here.
Solution: Use jupyter kernelspec list to list the location active kernel locations.
$ conda activate my-conda-env
$ jupyter kernelspec list
Available kernels:
python3 /opt/miniconda3/envs/my-conda-env/share/jupyter/kernels/python3
If the kernel in the environment is missing, you can try creating it manually using
ipython kernel install --sys-prefix in the activated environment, but it is probably better to
check your installation, because conda install ipykernel should have created the environment
(maybe try re-crate the environment and re-install all packages?).
If a user kernel specification is blocking the environment kernel specification, you can either
remove it or use a relative python path which will use $PATH to figure out which python to use.
So something like this, should be totally fine:
$ cat ~/.local/share/jupyter/kernels/python3/kernel.json
{
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}
Correct conda environment not activated
Symptoms: ImportError for modules installed in the conda environment (but not installed system
wide) in Jupyter notebooks and Python terminals
Explanation: Each terminal has a set of environment variables, which are lost when the terminal
is closed. In order to use a conda environment certain environment variables need to be set, which
is done by activating it using conda activate my-conda-env. If you attempted to run Jupyter
notebook from inside the conda environment (option 1), but did not activate the conda environment
before running it, it might run the system's jupyter.
Solution: Activate conda environment before running Jupyter.
conda activate my-conda-env
jupyter notebook
Broken kernel configuration
Symptoms: Strange things happening. Maybe similar symptoms as above, e.g. ImportError
Explanation: If you attempted to use option 2, i.e. running Jupyter from system and the Jupyter
kernel inside the conda environment by using an explicit configuration for the kernel, but it does
not behave as you expect, the configuration might be corrupted in some way.
Solution: Check configuration in ~/.local/share/jupyter/kernels/my-kernel-name/kernel.json
and fix mistakes manually or remove the entire directory and re-create it using the command
provided above for option 2. If you can't find the kernel configuration there run
jupyter kernelspec list.
Python 2 vs 3
Symptoms: ImportError due to wrong Python version of the Jupyter kernel or other problems
with Python 2/3
Explanation: The kernel configuration can have all sorts of confusing and misleading effects.
For example the default Python 3 kernel configuration will allow me to launch a Jupyter notebook
running on Python 2:
conda create -n my-conda-env
conda activate my-conda-env
conda install python=2
conda install jupyter
jupyter notebook
The default Python 3 kernel:
$ cat ~/.local/share/jupyter/kernels/python3/kernel.json
{
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}
After creating a new Jupyter Notebook with the Python 3 kernel, Python 2 from the conda
environment will be used even if "Python 3" is displayed by Jupyter.
Solution: Don't use Python 2 ;-)

The following command can also be used as a one liner to create your Conda environment running the latest version of Python and the latest version of Jupyter Notebooks,
conda create -n <env-name> python jupyter
If you want to install specific versions of Python or Jupyter, you can do,
conda create -n <env-name> python=<version> jupyter=<version>
For instance,
conda create -n <env-name> python=3.10.4 jupyter=1.0.0
If there are other packages that you want to use in this environment with your notebooks you can do the following,
conda create -n <env-name> python jupyter <another-package> <another-package> ...
For instance,
conda create -n <env-name> python jupyter scikit-learn
Note that similar to before, these commands will install the latest versions of Python and the relevant packages. If you want specific versions, you can use the =<version> syntax.
Also, you can still install any of the packages that you need using either pip install or conda install once the environment has been created.
After you have created your environment (using any of the methods given above), you can simply run the following commands to activate your environment and run Jupyter Notebooks,
conda activate <env-name>
jupyter notebook

Following worked for me :
Activate the environment that you want to use : conda activate
<env_name>
pip install ipykernel (if you don't already have it)
python -m ipykernel install --user --name=<env_name>

How do I add a python3 kernel to my jupyter notebook?

I installed jupyter notebook along with anaconda python as the only python on my PC (Windows 10). However I recently installed python 3.6.2 and I wonder if I can somehow add it to jupyter so as I can use them changeably.
I remember having both on my other machine when I installed python first and then after that I installed the whole anaconda package with the jupyter notebook (so I had python 3 and python(conda) option for kernels).
So how can I add to jupyter?

To install a Jupyter kernel for a different version of Python, run:
conda create -n py36 'python=3.6' ipykernel # Replace `3.6` with desired version
To enable the newly installed Jupyter kernel in all conda environments, run:
source activate py36
python -m ipykernel install --user
Now, when you start Jupyter from your root environment, you will have the option to choose a different Python kernel when you click on 'Kernel -> Change kernel':
source deactivate
jupyter notebook
For more info, see Installing the IPython kernel.

Simple, just find where the script jupyter-notebook resides, for example ~/.local/bin if you installed it locally.
Then just edit the first line to: #!/usr/bin/python3 will be fine.

jupyter ModuleNotFoundError: No module named matplotlib

I am currently trying to work basic python - jupyter projects.
I am stuck on following error during matplotlib:
screenshot on jupyter-error
ModuleNotFoundError: No module named 'matplotlib'
I tried to update, reinstall matplotlib aswell in conda and in pip but it still not working.
happy over every constructive feedback

In a Notebook's cell type and execute the code:
import sys
!{sys.executable} -m pip install --user matplotlib
and reload the kernel
(src: http://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/ )

open terminal and change the directory to Scripts folder where python installed. Then type the following command and hit enter
pip install matplotlib
Hope this will solve the issue.

I was facing the exact issue. It turns out that it was using the system Python version despite me having activated my virtual environment.
This is what eventually worked.
If you are using a virtual environment which has a name say myvenv, first activate it using command:
source activate myvenv
Then install module ipykernel using the command:
pip install ipykernel
Finally run (change myvenv in code below to the name of your environment):
ipykernel install --user --name myvenv --display-name "Python (myvenv)"
Now restart the notebook and it should pick up the Python version on your virtual environment.

While #Frederic's top-voted solution is based on JakeVDP's blog post from 2017, it completely neglects the %pip magic command mentioned in the blog post. Since 2017, that has landed in mainline IPython and the easiest way to access the correct pip instance connected to your current IPython kernel and environment from within a Jupyter notebook is to do
%pip install matplotlib
Take a look at the list of currently available magic commands at IPython's docs.

generally speaking you should try to work within python virtual environments. and once you do that, you then need to tell JupyterLab about it. for example:
# create a virtual environment
# use the exact python you want to work with in this step
python3.9 -m venv myvenv
# 'activate' (or 'enter') it
source myvenv/bin/activate
# install the exact stuff you want to use in that environment
pip install matplotlib
# now tell JupyterLabs about the environment
python -m ipykernel install --user --name="myenv" --display-name="My project (myenv)"
# start it up
jupyter notebook mynotebook
# if you now look under 'Kernel->Change kernel', your 'myenv' should be there
# select it (restart kernel etc if needed) and you should be good

The issue with me was that jupyter was taking python3 for me, you can always check the version of python jupyter is running on by looking on the top right corner (attached screenshot).
When I was doing pip install it was installing the dependencies for python 2.7 which is installed on mac by default.
It got solved by doing:
> pip3 install matplotlib

Having the same issue, installing matplotlib before to create the virtualenv solved it for me. Then I created the virtual environment and installed matplotlib on it before to start jupyter notebook.

in jupter notebook type
print(sys.executable)
this gave me the following
/Users/myusername/opt/anaconda3/bin/python
open terminal, go into the folder
/Users/myusername/opt/anaconda3/bin/
type the following:
python3 -m pip install matplotlib
restart jupyter notebook (mine is vs code mac ox)

If module installed an you are still getting this error, you might need to run specific jupyter:
python -m jupyter notebook
and this is also works
sudo jupyter notebook --allow-root