I bought a book which comes with jupyter notebook. In the first chapter, it asks me to install required libraries. It use {sys.executable} -m. I never see it before. what does {sys.executable} and -m do? also why use --user at the end?
typically, I just use ! pip install numpy==1.19.2
Anyone can help me understand it? Thank you!
import sys
!{sys.executable} -m pip install numpy==1.19.2 --user
!{sys.executable} -m pip install scipy==1.6.2 --user
!{sys.executable} -m pip install tensorflow==2.4.0 --user
!{sys.executable} -m pip install tensorflow-probability==0.11.0 --user
!{sys.executable} -m pip install scikit-learn==0.24.1 --user
!{sys.executable} -m pip install statsmodels==0.12.2 --user
!{sys.executable} -m pip install ta --user
Let's split this question up into multiple parts.
Part 1
From the Python documentation:
sys.executable
A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense. If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None.
Formatting that in, we get:
...\python.exe -m pip install <package> --user
Part 2
Also from the docs:
-m <module-name>
Search sys.path for the named module and execute its contents as the __main__ module.
This is generally the same as just pip install <package> --user, however if you have multiple versions of Python installed, the wrong version of pip might get invoked. By using -m, a matching version of pip will always be invoked.
Part 3
This time from the pip documentation:
Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation for site.USER_BASE for full details.)
Basically, this means that instead of installing to the normal package directory (which could require administrator privileges), it installs to %APPDATA%\Python, which should always be accessible as it is in your user folder.
sys.executable is refering to the Python interpreter for the current system. It comes handy when using virtual environments and have several interpreters on the same machine.
The -m option loads and execute a module as a script, here pip.
The --user is an option for pip install, see this answer describing its use.
Then the !{} is jupyter-specific syntax to execute commands in a cell if I remember correctly.
Related
WARNING: pip is being invoked by an old script wrapper. This will fail
in a future version of pip. Please see
https://github.com/pypa/pip/issues/5599 for advice on fixing the
underlying issue. To avoid this problem you can invoke Python with '-m
pip' instead of running pip directly.
When I directly type pip list to the terminal, I get the above warning. What does it mean exactly?
Should I always use it as python3 -m pip list? If I use it in that way, same output (list of packages) comes up without any warning.
p.s. : I am on ubuntu 18.10
I faced the same issue but on Windows. Reinstalling pip worked for me. You can force a reinstall of pip with:
python -m pip install --upgrade --force-reinstall pip
For Python3 Versions:
python3 -m pip install --upgrade --force-reinstall pip
I also faced the same problem when I switched to zsh shell from bash.
The solution was simple but I didn't notice it at first.
After I installed pip, I saw this warning
pip is being invoked by an old script wrapper
First I tried this solution
python3.8 -m pip install --upgrade --force-reinstall pip
But then I ran into this issue:
Then I searched how to add PYTHONPATH. I opened my .zshrc and say these lines were commented
# If you come from bash you might have to change your $PATH.
I uncommented the line that followed, and my misery vanished.
Now when I ran,
python3.8 -m pip install --upgrade --force-reinstall pip
Then the warnings of not in path disappeared in thin air, leaving me with a clean output.
I hope this would help anybody who ran into the same problem.
Let me preface this by saying I am still very green with python and linux in general so I may be off base with my guidance here but I digress...
You might want check the location of the pip module you are invoking when you use the pipcommand. for me, I found out that when I would update and modify the pip command, it would update the pip file on my ~/.local/bin directory but when I would run it, it would default to the pip command located in the /usr/local/bin directory.
run the command
pip install --upgrade pip
for me this command returned:
Defaulting to user installation because normal site-packages is not writeable
Requirement already up-to-date: pip in ./.local/lib/python3.6/site-packages (20.1.1)
Note the file location and version (in bold).
check your path variables and the default pip that executes default by running the these 2 commands respectively
echo $PATH
and
which pip
god willing, they'll be congruent otherwise you'll have to either alter your the path variable directories making sure that the directory for your desired pip module is first or you'll have to delete the pip file from the director that you dont want use (i.e. the directory that came up when you ran which pip if that is not the same as the directory listed when you updated pip)
For me, removing the pip files in the usr/local/bin worked like a charm. Also check that the pip files that you want to use are referencing the correct version of python at the top of their scripts
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(main())
The other file originally referenced usr/bin/python (Python 2.7) instead of usr/bin/python3 (python 3.6.9) as I wanted initially.
Like I said before, I am just getting started with linux and python so take this with a grain of salt. Nevertheless, I no longer get this pip warning after taking these steps. Let me know if this helps at all.
I've encountered the same problem after I updated pip to 20.1.x version through Pycharm. I've found one way to ensure that you can use pip install xxx in emergency case:
Open the folder containing pip (e.g. C:\Program Files\Python37\Lib in my win10 laptop)
You may find two folders separately belongs to the old version pip and newly installed pip (e.g. ./pip19_xxx for your old one and ./pip for the newly installed one
Delete the folder of the new version pip (e.g. ./pip)
(IF NECESSARY) Change environment PATH in windows10
Test pip install xxx or python -m pip install xxx in cmd, it should work by now
After the PIP upgrade, because the boot file (/usr/bin/pip) has not been changed, the old boot file is still used,
Old PIP startup file:
#!/usr/bin/python3
# GENERATED BY DEBIAN
import sys
# Run the main entry point, similarly to how setuptools does it, but because
# we didn't install the actual entry point from setup.py, don't use the
# pkg_resources API.
from pip import main
if __name__ == '__main__':
sys.exit(main())
New PIP startup file:
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(main())
The solution is as follows:
vi open /usr/bin/pip to add new code to save,
After this operation, there is no need to add "python -m" to run pip, and there will be no warning
You can solve it by upgrading pip:
pip install --upgrade pip
You can solve this by executing the following commands.
python -m pip install --upgrade --force-reninstall pip
Add python to the path in ~/.bash_profile.
like PATH=$PATH:/usr/local/bin
Just use
python -m pip install [module_name]
This will resolve warning issue
This worked for me.
python3 -m pip install <package name>
You can solve this problem by doing the following
1.Check that you are not having two versions of python.If you are having then uninstall one.
If the problem does not resolve after doing these then uninstall all the python installed in your pc and then again install it.
I ran into the same issue on Ubuntu 18.04 with python 3.6 after I ran pip3 install --upgrade pip. (The original pip version installed by apt install python3-pip was 9.0, but I wanted pip >= 10.0 so I could use pip list -v.)
Step 1: Check the pip paths
I found that there were several pips installed in various locations on my machine, and it was causing problems. Use these commands to see which pips are being executed.
pip --version
pip3 --version
type -a pip pip3 # show all the locations for pip and pip3
Step 2: Fix the pip paths
This step will be highly individual. You have to figure out for yourself which one to keep and which ones to remove. Here are some example commands that may be helpful:
# remove ~/.local/bin/pip
cd ~/.local/bin
mv pip pip.bak
hash -r # important, reset bash command cache!
# remove the old pip 9.0
sudo apt purge python3-pip
Step 3: Upgrade pip
Run the commands from Step 1 again to verify everything looks good. Then upgrade pip:
pip3 install --upgrade pip
# if you get permission denied above, the install for --user
pip3 install --user --upgrade pip
# validate everything looks right
pip --version
pip3 --version
type -a pip pip3 # show all the locations for pip and pip3
I am unable to install a module called 'jieba' in Python 3 which is running in Jupyter Notebook 6.0.0. I keep getting ModuleNotFoundError: No module named 'jieba' after trying these methods:
1. import jieba
2. pip3 install jieba
Can anyone help? Thank you.
pip3 in the terminal is almost certainly installing your package into a different Python installation.
Rather than have you hunt around for the right installation, you can use Python and Jupyter themselves to ensure you are using the correct Python binary. This relies on three tricks:
You can execute the pip command-line tool as a module by running python -m pip .... This uses the pip module installed for the python command, so you don't have to verify what python installation the pip3 command is tied to.
You can get the path of the current Python interpreter with the sys.executable attribute.
You can execute shell commands from Jupyter notebooks by prefixing the shell command with !, and you can insert values generated with Python code with {expression}
You can combine these to run pip (to install packages or run other commands) against the current Python installation, from Jupyter itself, by adding this into a cell:
import sys
!{sys.executable} -m pip <pip command line options>
To install your jieba package, that makes:
import sys
!{sys.executable} -m pip install jieba
If you are using Anaconda, then you could also install the conda package for jieba; the package does not require any platform-specific dependencies or compilation, but it may be more convenient for you or necessary to install other packages that do have such requirements and have pre-compiled conda packages.
In that case, tell the conda command about your Python executable:
import sys
!conda install --yes --prefix {sys.prefix} <package name>
I am using Windows 10. Currently, I have Python 2.7 installed. I would like to install Python 3.5 as well. However, if I have both 2.7 and 3.5 installed, when I run pip, how do I get the direct the package to be installed to the desired Python version?
You will have to use the absolute path of pip.
E.g: if I installed python 3 to C:\python35, I would use:
C:\> python35\Scripts\pip.exe install packagename
Or if you're on linux, use pip3 install packagename
If you don't specify a full path, it will use whichever pip is in your path.
Because usually i change my intepreter to run something(i got 2 diff projects with both 2 and 3), i use these solution:
Add path to the environment as usual (of course)
Rename ur python.exe , in my case i want to run python 3 using command python3 on my cmd. So i renamed my python.exe in python3.x directory with python3. Itll works with python 2 ofc.
Then to use pip in both python, i use this command.
python3 -m pip install 'somepackage'
and to run pip on python2
python -m pip install 'somepackage'
This is may not the best solution out there, but i like this one
** WINDOWS **
ref : https://datascience.com.co/how-to-install-python-2-7-and-3-6-in-windows-10-add-python-path-281e7eae62a
In my case, I have Python 2.7 and Python 3.4, with the Python Launcher for Windows.
This is the output when running this commands:
PS C:\> pip -V
pip 9.0.1 from c:\python27\lib\site-packages (python 2.7)
PS C:\> pip3 -V
pip 9.0.1 from C:\Python34\lib\site-packages (python 3.4)
I'll note that in my Python27\Scripts\ directory, I have pip.exe, pip2.exe and pip2.7.exe.
And in my Python34\Scripts\ directory, I have pip.exe, pip3.exe and pip3.4.exe.
So all of these .exe files help you when you have different versions of Python installed at the same time.
Of course, for this to work, you have to have the respective Scriptsdirectries in your Path system enviroment variable.
The answer from Farhan.K will work. However, I think a more convenient way would be to rename python35\Scripts\pip.exe to python35\Scripts\pip3.exe assuming python 3 is installed in C:\python35.
After renaming, you can use pip3 when installing packages to python v3 and pip when installing packages to python v2. Without the renaming, your computer will use whichever pip is in your path.
I would advise against ever calling any pip script directly (nor pip3, pip2.7.exe, anything like that).
Instead, a surefire way is to always prefer the explicit variant of calling pip's executable module for a specific Python interpreter:
path/to/pythonX.Y -m pip somecommand
path/to/venv/bin/python -m pip somecommand
C:\path\to\venv\Scripts\python.exe -m pip somecommand
There are many advantages to this, for example:
It is explicit for which Python interpreter the projects will be pip-installed (Python 2 or 3, inside the virtual environment or not, etc.)
For a virtual environment, one can pip-install (or do other things) without activating it: path/to/venv/bin/python -m pip install SomeProject
Under Windows this is the only way to safely upgrade pip itself path\to\venv\Scripts\python.exe -m pip install --upgrade pip
But yes, if all is perfectly setup, then python3 -m pip install SomeProject and pip3 install SomeProject should do the exact same thing, but there are way too many cases where there is an issue with the setup and things don't work as expected and users get confused (as shown by the many questions about this topic on this platform).
References
Brett Cannon's article "Why you should use python -m pip"
pip's documentation section on "Upgrading pip"
venv's documentation section on "Creating virtual environments": "You don’t specifically need to activate an environment [...]"
I ran across an issue with running pip with absolute path. This might be related to WinPython's installation routine and the order of installing Python 3.6 first, 2.7 second, or Python 3.6 being in the path.
No matter which pip was called, it was activating the 3.6 one:
λ C:\prog\WinPython-64bit-2.7.13.1Zero\python-2.7.13.amd64\Scripts\pip2.exe --version
pip 9.0.1 from C:\prog\WinPython-64bit-3.6.1.0Zero\python-3.6.1.amd64\lib\site-packages (python 3.6)
What finally did the trick was calling pip as a module of the respective python binary:
λ C:\prog\WinPython-64bit-2.7.13.1Zero\python-2.7.13.amd64\python.exe -m pip --version
pip 9.0.1 from C:\prog\WinPython-64bit-2.7.13.1Zero\python-2.7.13.amd64\lib\site-packages (python 2.7)
Hope that might help someone with similar issues.
I tried many things , then finally
pip3 install --upgrade pip worked for me as i was facing this issue since i had both python3 and python2.7 installed on my system.
mind the pip3 in the beginning and pip in the end.
And yes you do have to run in admin mode the command prompt and make sure if the path is set properly.
1-open command prompt and change direction using the command cd C:\Python35\Scripts
2- write the command pip3 install --upgrade pip
3- close the command prompt and reopen it again to return to the default direction and use the command pip3.exe install package_name to install any package you want
I am using OSX and I have pip installed for both Python3.5 and Python2.7. I know I can run the command pip2 to use Python2 and when I use the command pip3 Python3.x will be used.
The problem is that the default of pip is set to Python2.7 and I want it to be Python3.x.
How can I change that?
edit:
No, I am not running a virtual environment yet. If it was a virtual environment I could just run Python3.x and forget all about Python2.7, unfortunately since OSX requires Python2.7 for it's use I can't do that. Hence why I'm asking this.
Thanks for the answer. I however don't want to change what running python does. Instead I would like to change the path that running pip takes. At the moment pip -V shows me pip 8.1.2 from /Library/Python/2.7/site-packages (python 2.7), but I am looking for pip 8.1.2 from /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (python 3.5) I am sure there has to be a way to do this. Any ideas?
Run this:
pip3 install --upgrade --force pip
or even more explicit:
python3 -m pip install --upgrade --force pip
This will install pip for Python 3 and make Python 3 version of pip default.
Validate with:
pip -V
I always just run it via Python itself, this way:
python3 -m pip install some_module
or
python2 -m pip install some_module
The -m calls the __main__.py module of a specified package. Pip supports this.
Can't you alias pip='pip3' in your ~/.bash_profile?
In Terminal, run nano ~/.bash_profile, then add a line to the end that reads alias pip='pip3'. This is safe; it won't affect system processes, only your terminal.
For your projects, you should be using a virtualenv.
You can choose which python will be that of the virtualenv at creation time, by specifying it on the command line:
virtualenv -p python3 env
# then
. env/bin/activate
python # ← will run python3
That python interpreter will be the one used when you run python or pip while the virtualenv is active.
Under the hood, activating the virtualenv will:
modify your PATH environment setting so binaries in env/bin
override those from your system.
modify your PYTHONHOME
environment setting so python modules are loaded from env/lib.
So python, pip and any other package you install with pip will be run from the virtualenv, with the python version you chose and the package versions you installed in the virtualenv.
Other than this, running python without using virtualenv will just run the default python of the system, which you cannot usually change as it would break a lot of system scripts.
It works for me:
As super-user
Uninstall pip
sudo pip uninstall pip
Install pip
sudo python3 -m pip install --upgrade --force pip
Check install path
sudo pip -V
As local-user
Uninstall pip
pip uninstall pip
Install pip
python3 -m pip install --upgrade --force pip
Check install path
pip -V
Although PEP 394 does not specifically mention pip, it does discuss a number of other Python-related commands (including python itself). The short version is that, for reasons of backwards compatibility, the unversioned commands should refer to Python 2.x for the immediate future on most reasonable systems.
Generally, these aliases are implemented as symbolic links, and you can just flip the symlink to point at the version you want (e.g. with ln -f -s $(which pip3) $(which pip) as root). But it may not be a good idea if you have any software that expects to interact with Python 2 (which may be more than you think since a lot of software interacts with Python).
The saner option is to set up a Virtualenv with Python 3. Then, within the Virtualenv, all Python-related commands will refer to 3.x instead of 2.x. This will not break the system, unlike the previous paragraph which could well break things.
Since you have specified in the comments you want syntax like pip install [package] to work, here is a solution:
Install setuptools for Python3: apt-get install python3-setuptools
Now pip for Python3 could be installed by: python3 -m easy_install pip
Now you can use pip with the specific version of Python to
install package for Python 3 by: pip-3.2 install [package]
Why not just repoint the link /bin/python to python3? It seems like the easiest solution. Especially if you want it for all users of your system.
I have dutifully uninstalled all the Python packages I installed with sudo pip install and installed them with pip --user install instead. Yay me :)
On Ubuntu, I know I can find the relevant binaries at /home/<USERNAME>/.local/bin and the packages themselves at /home/<USERNAME>/.local/lib/python2.7/site-packages ... but navigating there is not as simple as good old pip freeze.
How can I pip freeze and get only the packages I installed with pip --user install rather than all the Python packages, including those installed via apt?
Currently pip does not have any such options. So with default pip its not possible. (and I submitted a feature request and now there is a working PR too!)
However I wrote a little script, which does solve your problem:
# pip_user_installs.py
import sys
import pkg_resources
for module in pkg_resources.working_set:
if sys.argv[1] in module.location:
print module.project_name
usage:
$ python pip_user_installs.py $HOME
It's fairly easy in recent versions of pip (the PR in the other answer is now part of pip).
pip freeze --user
This will output a list of packages currently installed to the user's site-packages.