Develop Reference

what does {sys.executable} do in jupyter notebook?

I bought a book which comes with jupyter notebook. In the first chapter, it asks me to install required libraries. It use {sys.executable} -m. I never see it before. what does {sys.executable} and -m do? also why use --user at the end?
typically, I just use ! pip install numpy==1.19.2
Anyone can help me understand it? Thank you!
import sys
!{sys.executable} -m pip install numpy==1.19.2 --user
!{sys.executable} -m pip install scipy==1.6.2 --user
!{sys.executable} -m pip install tensorflow==2.4.0 --user
!{sys.executable} -m pip install tensorflow-probability==0.11.0 --user
!{sys.executable} -m pip install scikit-learn==0.24.1 --user
!{sys.executable} -m pip install statsmodels==0.12.2 --user
!{sys.executable} -m pip install ta --user

Let's split this question up into multiple parts.
Part 1
From the Python documentation:
sys.executable
A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense. If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None.
Formatting that in, we get:
...\python.exe -m pip install <package> --user
Part 2
Also from the docs:
-m <module-name>
Search sys.path for the named module and execute its contents as the __main__ module.
This is generally the same as just pip install <package> --user, however if you have multiple versions of Python installed, the wrong version of pip might get invoked. By using -m, a matching version of pip will always be invoked.
Part 3
This time from the pip documentation:
Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation for site.USER_BASE for full details.)
Basically, this means that instead of installing to the normal package directory (which could require administrator privileges), it installs to %APPDATA%\Python, which should always be accessible as it is in your user folder.

sys.executable is refering to the Python interpreter for the current system. It comes handy when using virtual environments and have several interpreters on the same machine.
The -m option loads and execute a module as a script, here pip.
The --user is an option for pip install, see this answer describing its use.
Then the !{} is jupyter-specific syntax to execute commands in a cell if I remember correctly.

How to recover PIP

I think I uninstalled pip by mistake ^^
I ran something like "pip uninstall pip" in CMD and it completely broke, giving fatal errors.
I uninstalled and reinstalled python, added the correct folders back to PATH, but now it's only working via python -m pip XXXX and not by typing pip XXXX...
If I type pip XXX i get an empty row and CMD gives me back the cursor.
How can I recover? I liked pip XXX better and I am sure that the pip folder is in PATH.

It would help greatly to know what version of each you are using and what OS but for a generic answer...
You could try to install it manually:
Vist:
https://pip.pypa.io/en/stable/installing.html
Download:
get-pip.py
python get-pip.py
Make sure your python version matches your pip version. Otherwise you will always have to be specific on your installs. And for sake of simplicity make sure it works fine with just one version of Python installed. Then if that works you can consider having more.
After installing it, if you upgrade your python version make sure you keep it up to date.
I would try something like:
pip install --upgrade --no-deps --force-reinstall

As os version 21.3.1, you can install pip with the one-liner:
python -m ensurepip --upgrade.
See installation of pip doc for details.

Warning: pip is being invoked by an old script wrapper

WARNING: pip is being invoked by an old script wrapper. This will fail
in a future version of pip. Please see
https://github.com/pypa/pip/issues/5599 for advice on fixing the
underlying issue. To avoid this problem you can invoke Python with '-m
pip' instead of running pip directly.
When I directly type pip list to the terminal, I get the above warning. What does it mean exactly?
Should I always use it as python3 -m pip list? If I use it in that way, same output (list of packages) comes up without any warning.
p.s. : I am on ubuntu 18.10

I faced the same issue but on Windows. Reinstalling pip worked for me. You can force a reinstall of pip with:
python -m pip install --upgrade --force-reinstall pip
For Python3 Versions:
python3 -m pip install --upgrade --force-reinstall pip

I also faced the same problem when I switched to zsh shell from bash.
The solution was simple but I didn't notice it at first.
After I installed pip, I saw this warning
pip is being invoked by an old script wrapper
First I tried this solution
python3.8 -m pip install --upgrade --force-reinstall pip
But then I ran into this issue:
Then I searched how to add PYTHONPATH. I opened my .zshrc and say these lines were commented
# If you come from bash you might have to change your $PATH.
I uncommented the line that followed, and my misery vanished.
Now when I ran,
python3.8 -m pip install --upgrade --force-reinstall pip
Then the warnings of not in path disappeared in thin air, leaving me with a clean output.
I hope this would help anybody who ran into the same problem.

Let me preface this by saying I am still very green with python and linux in general so I may be off base with my guidance here but I digress...
You might want check the location of the pip module you are invoking when you use the pipcommand. for me, I found out that when I would update and modify the pip command, it would update the pip file on my ~/.local/bin directory but when I would run it, it would default to the pip command located in the /usr/local/bin directory.
run the command
pip install --upgrade pip
for me this command returned:
Defaulting to user installation because normal site-packages is not writeable
Requirement already up-to-date: pip in ./.local/lib/python3.6/site-packages (20.1.1)
Note the file location and version (in bold).
check your path variables and the default pip that executes default by running the these 2 commands respectively
echo $PATH
and
which pip
god willing, they'll be congruent otherwise you'll have to either alter your the path variable directories making sure that the directory for your desired pip module is first or you'll have to delete the pip file from the director that you dont want use (i.e. the directory that came up when you ran which pip if that is not the same as the directory listed when you updated pip)
For me, removing the pip files in the usr/local/bin worked like a charm. Also check that the pip files that you want to use are referencing the correct version of python at the top of their scripts
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(main())
The other file originally referenced usr/bin/python (Python 2.7) instead of usr/bin/python3 (python 3.6.9) as I wanted initially.
Like I said before, I am just getting started with linux and python so take this with a grain of salt. Nevertheless, I no longer get this pip warning after taking these steps. Let me know if this helps at all.

I've encountered the same problem after I updated pip to 20.1.x version through Pycharm. I've found one way to ensure that you can use pip install xxx in emergency case:
Open the folder containing pip (e.g. C:\Program Files\Python37\Lib in my win10 laptop)
You may find two folders separately belongs to the old version pip and newly installed pip (e.g. ./pip19_xxx for your old one and ./pip for the newly installed one
Delete the folder of the new version pip (e.g. ./pip)
(IF NECESSARY) Change environment PATH in windows10
Test pip install xxx or python -m pip install xxx in cmd, it should work by now

After the PIP upgrade, because the boot file (/usr/bin/pip) has not been changed, the old boot file is still used,
Old PIP startup file:
#!/usr/bin/python3
# GENERATED BY DEBIAN
import sys
# Run the main entry point, similarly to how setuptools does it, but because
# we didn't install the actual entry point from setup.py, don't use the
# pkg_resources API.
from pip import main
if __name__ == '__main__':
sys.exit(main())
New PIP startup file:
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(main())
The solution is as follows:
vi open /usr/bin/pip to add new code to save,
After this operation, there is no need to add "python -m" to run pip, and there will be no warning

You can solve it by upgrading pip:
pip install --upgrade pip

You can solve this by executing the following commands.
python -m pip install --upgrade --force-reninstall pip
Add python to the path in ~/.bash_profile.
like PATH=$PATH:/usr/local/bin

Just use
python -m pip install [module_name]
This will resolve warning issue

This worked for me.
python3 -m pip install <package name>

You can solve this problem by doing the following
1.Check that you are not having two versions of python.If you are having then uninstall one.
If the problem does not resolve after doing these then uninstall all the python installed in your pc and then again install it.

I ran into the same issue on Ubuntu 18.04 with python 3.6 after I ran pip3 install --upgrade pip. (The original pip version installed by apt install python3-pip was 9.0, but I wanted pip >= 10.0 so I could use pip list -v.)
Step 1: Check the pip paths
I found that there were several pips installed in various locations on my machine, and it was causing problems. Use these commands to see which pips are being executed.
pip --version
pip3 --version
type -a pip pip3 # show all the locations for pip and pip3
Step 2: Fix the pip paths
This step will be highly individual. You have to figure out for yourself which one to keep and which ones to remove. Here are some example commands that may be helpful:
# remove ~/.local/bin/pip
cd ~/.local/bin
mv pip pip.bak
hash -r # important, reset bash command cache!
# remove the old pip 9.0
sudo apt purge python3-pip
Step 3: Upgrade pip
Run the commands from Step 1 again to verify everything looks good. Then upgrade pip:
pip3 install --upgrade pip
# if you get permission denied above, the install for --user
pip3 install --user --upgrade pip
# validate everything looks right
pip --version
pip3 --version
type -a pip pip3 # show all the locations for pip and pip3

pip install in python 3.6.5

I'm trying to install some libraries on python, and I open cmd and type-m pip install <lib_name> but it says there's syntax error. When I skip the "<" it says pip is not a recognized command. Any help? I'm using Python 3.6.5.
enter image description here
this is the output for almost every command I'm trying...

first of all we need to know if you are good to go with pip.
Try posting the output of
pip --version
If pip doesn't seem to be installed, you should install it via
python -m ensurepip --default-pip
Once pip is installed you can install packages using the command
pip install whatever
Here's a complete starter pack tutorial for using pip :)
https://packaging.python.org/tutorials/installing-packages/

you can enter either cmd into the terminal:
python -m pip install <package-name>
or
pip install <package-name>
The first will always install into your global environment and the second could depend on whether you're in a virtualenv (if none of this makes sense to you, don't worry about it).

First install python!
According to your screenshot, it is not installed... (or not available, in that case check your PATH in system properties).

Problems of VirtualEnv and pip3

I used VirtualEnv to create a python2 environment without system site packages like this:
virtualenv -p /usr/bin/python2.7 --no-site-packages ENV2.7
And I want to install packages in this environment.
However, I found that my python code is still trying to look for packages out of this environment.
For example, after activate this env, I used:
pip install matplotlib
And in my demo.py, there is
import matplotlib
But this raised an error, and can not find this package
However, when I use python in the terminal and enter the interactive python, import matplotlib dose not raise an error.
Then I started another terminal and tried to install this package out of the environment by pip3:
pip3 install matplotlib
It turned out that my demo.py just work well.
Any idea? Many Thanks!

It sounds like your virtualenv pip version may be using pip3 instead of pip2:
Make sure you are using the correct python version in your project that you mean to, and using the same version of pip in your virtualenv. (Note that you use pip above once, then you used pip3 outside your virtualenv.)
Check your pip version from inside the virtualenv:
workon (your env name)
which pip
pip -V
Output should look something like:
$ which pip
/home/yourname/.virtualenvs/testenv/bin/pip
$ pip -V
pip 9.0.1 from /home/yourname/.virtualenvs/testenv/local/lib/python2.7/site-packages (python 2.7)
It should tell you you're using pip inside your virtualenv, and the correct python version.
If that looks correct, install your packages.
pip install (whatever)
Check they are installed with pip freeze.
Run your project. :)