Why does pip list generate a more comprehensive list than pip freeze?
$ pip list
feedparser (5.1.3)
pip (1.4.1)
setuptools (1.1.5)
wsgiref (0.1.2)
$ pip freeze
feedparser==5.1.3
wsgiref==0.1.2
Pip's documentation states:
freeze
Output installed packages in requirements format.
list
List installed packages.
What is a "requirements format"?
One may generate a requirements.txt via:
$ pip freeze > requirements.txt
A user can use this requirements.txt file to install all the dependencies. For instance:
$ pip install -r requirements.txt
The packages need to be in a specific format for pip to understand, such as:
# requirements.txt
feedparser==5.1.3
wsgiref==0.1.2
django==1.4.2
...
That is the "requirements format".
Here, django==1.4.2 implies install django version 1.4.2 (even though the latest is 1.6.x).
If you do not specify ==1.4.2, the latest version available would be installed.
You can read more in "Virtualenv and pip Basics",
and the official "Requirements File Format" documentation.
To answer the second part of this question, the two packages shown in pip list but not pip freeze are setuptools (which is easy_install) and pip itself.
It looks like pip freeze just doesn't list packages that pip itself depends on. You may use the --all flag to show also those packages.
From the documentation:
--all
Do not skip these packages in the output: pip, setuptools, distribute, wheel
The main difference is that the output of pip freeze can be dumped into a requirements.txt file and used later to re-construct the "frozen" environment.
In other words you can run:
pip freeze > frozen-requirements.txt on one machine and then later on a different machine or on a clean environment you can do:
pip install -r frozen-requirements.txt
and you'll get the an identical environment with the exact same dependencies installed as you had in the original environment where you generated the frozen-requirements.txt.
Look at the pip documentation, which describes the functionality of both as:
pip list
List installed packages, including editables.
pip freeze
Output installed packages in requirements format.
So there are two differences:
Output format, freeze gives us the standard requirement format that may be used later with pip install -r to install requirements from.
Output content, pip list include editables which pip freeze does not.
pip list shows ALL installed packages.
pip freeze shows packages YOU installed via pip (or pipenv if using that tool) command in a requirements format.
Remark below that setuptools, pip, wheel are installed when pipenv shell creates my virtual envelope. These packages were NOT installed by me using pip:
test1 % pipenv shell
Creating a virtualenv for this project…
Pipfile: /Users/terrence/Development/Python/Projects/test1/Pipfile
Using /usr/local/Cellar/pipenv/2018.11.26_3/libexec/bin/python3.8 (3.8.1) to create virtualenv…
⠹ Creating virtual environment...
<SNIP>
Installing setuptools, pip, wheel...
done.
✔ Successfully created virtual environment!
<SNIP>
Now review & compare the output of the respective commands where I've only installed cool-lib and sampleproject (of which peppercorn is a dependency):
test1 % pip freeze <== Packages I'VE installed w/ pip
-e git+https://github.com/gdamjan/hello-world-python-package.git#10<snip>71#egg=cool_lib
peppercorn==0.6
sampleproject==1.3.1
test1 % pip list <== All packages, incl. ones I've NOT installed w/ pip
Package Version Location
------------- ------- --------------------------------------------------------------------------
cool-lib 0.1 /Users/terrence/.local/share/virtualenvs/test1-y2Zgz1D2/src/cool-lib <== Installed w/ `pip` command
peppercorn 0.6 <== Dependency of "sampleproject"
pip 20.0.2
sampleproject 1.3.1 <== Installed w/ `pip` command
setuptools 45.1.0
wheel 0.34.2
My preferred method of generating a requirements file is:
pip list --format=freeze > requirements.txt
This method keeps just the package names and package versions without potentially linking to local file paths which 'pip freeze' alone will sometimes give me. Local file paths in a requirements file make your codebase harder to use for other users and some developers don't know how to fix this so I prefer this method for ease of adoptability.
pip list
List installed packages: show ALL installed packages that even pip installed implictly
pip freeze
List installed packages: - list of packages that are installed using pip command
pip freeze has --all flag to show all the packages.
Other difference is the output it renders, that you can check by running the commands.
For those looking for a solution. If you accidentally made pip requirements with pip list instead of pip freeze, and want to convert into pip freeze format. I wrote this R script to do so.
library(tidyverse)
pip_list = read_lines("requirements.txt")
pip_freeze = pip_list %>%
str_replace_all(" \\(", "==") %>%
str_replace_all("\\)$", "")
pip_freeze %>% write_lines("requirements.txt")
Related
I need to update offline a library in Python.
I have downloaded the library with pip download and then I try to update the library with the command:
pip install --no-index --user --find-links /tmp/pip/ --upgrade Werkzeug==0.15.5
which gives:
Ignoring indexes: https://...
Collecting Werkzeug==0.15.5
Installing collected packages: Werkzeug
Successfully installed Werkzeug-0.11.15
and then the library stays in the same version!
pip freeze | grep Wer
Werkzeug==0.11.15
Any ideas why this happens?
UPDATE: After the comment from #hoefling I rerun with the -vvv option and this is what I got:
pip install --no-index --user --find-links /tmp/pip2/ -vvv Werkzeug==0.15.5
Ignoring indexes: https://pypi:pypi#..../simple/
Collecting Werkzeug==0.15.5
0 location(s) to search for versions of Werkzeug:
Skipping link /tmp/pip2/werk/ (from -f); not a file
Found link file:///tmp/pip2/werk/Werkzeug-0.15.5-py2.py3-none-any.whl, version:
0.15.5
Local files found: /tmp/pip2/werk/Werkzeug-0.15.5-py2.py3-none-any.whl
Using version 0.15.5 (newest of versions: 0.15.5)
Installing collected packages: Werkzeug
Successfully installed Werkzeug-0.11.15
Cleaning up...
Try this command:
pip install Werkzeug-0.15.5.tar.gz
and the result must be like this:
Processing ./Werkzeug-0.15.5.tar.gz
Installing collected packages: Werkzeug
Running setup.py install for Werkzeug ... done
Successfully installed Werkzeug-0.15.5
This behaviour can happen because pip by default works with system Python which is located in /usr/bin/ on Linux. When installing the package, by giving Python --user flag your package is installed in your user's version of Python, probably located somewhere in ~/.local/.
To solve the problem you can install the package to your system Python, which is generally not recommended without --user flag. Another option is to use virtual environments and have the distribution that is made specifically for your project. Currently the recommended way is using venv.
$ python -m venv env
$ source env/bin/activate
(env) $ pip install ... (packages you need to install without --user flag)
(env) $ pip freeze
# should give you the packages you installed
This can help you not only with this example, but it can always keep your system Python installation clean and if you mess something up, you will only mess the environment you are having for specific project.
In order to make packages installed offline, I use the -d (or --download) option to pip install. For instance, pip install --download dependencies -r requirements.txt will download the packages for all required dependencies mentioned in requirements.txt to dependencies dir (but will not install them). Then I use pip install --no-index --find-links dependencies -r requirements.txt to install those downloaded packages without accessing the network.
Most of the time it works fine, but sometimes installation fails with error "Could not find a version that satisfies the requirement xyz". After doing pip install --user xyz --find-links dependencies manually (xyz IS present in the dependencies folder), installation fails with the same "Could not find a version that satisfies the requirement abc" error, but with different package 'abc'. It repeats several times until I manually resolve all failed dependencies.
How could I make run pip install --no-index --find-links dependencies -r requirements.txt without those weird dependency errors not finding packages that are already there?
Make sure of two things:
The pip version is the same in the offline server and in the online one.
To find out: pip -V
To update (if needed): pip install --upgrade pip
The python version is the same in both virtual enviroments or servers.
To find out: python (the header will have the version info)
In my case I was calling pip install --download outside the virtual environment (using default python version - 2.7) and then installing in a virtual environment with python 3 and the error I got was exactly the one you mentioned.
I'm new to virtualenv (on windows). I'm trying to use pip (1.5) install a local wheel file, but it is failing.
The command is:
pip install --no-index -f C:/Users/<User>/Download openpyxl
In the pip.log, I can see where it finds the correct file, but then doesn't try to install it:
Skipping link file:///C:/Users/<User>/Download/openpyxl-1.7.0-py2.py3-none-any.whl; unknown archive format: .whl
I have wheel (version 0.22) install globally as well as in the virtual environment. Any idea how I can get .whl to be a recognized format?
It appears wheel support is disabled.
Make sure that you have setuptools version 0.8 or newer installed, and that the use-wheel option is not set to false in $HOME/.pip/pip.conf.
Upgrading setuptools is easy enough if pip is already working:
pip install --upgrade setuptools
but note that older virtualenv versions can depend on older setuptools versions; you'll need to make sure that virtualenv is also up to date.
I have bumped into the same problem with wheel when downloaded requirements with:
pip install --download /pip_mirror six django_debug_toolbar
dir2pi /pip_mirror/
and tried to install them with:
pip install six-1.7.3-py2.py3-none-any.whl
Even though there is no any config at $HOME/.pip/pip.conf and
$ easy_install --version
setuptools 5.4.1
I still get:
unknown archive format: .whl
I have managed to avoid the problem by adding --no-use-wheel like this, so got only tar.gz files (instead of .whl)
pip install --no-use-wheel --download /pip_mirror six django_debug_toolbar
dir2pi /pip_mirror/
After this pip install --index-url=file:///pip_mirror/simple/ six went without any problems
We have a python/django based web application, many components of which are installed using pip. So I would like to ask if there is a way to save or download and save the particular python packages that we are having pip install (example: pip install django==1.5.1). We would like to have in the end a collection of the packages in the versions known to be working and with which the app was developed locally. Any and all advice will be appreciated.
If I understood your question right, you can pip freeze > requirements.txt, this command will add all the libraries you have used/"downloaded" for your app in the file requirements.txt(in case it exists the file be overwritten). This command allows you to later do pip install -r requirements.txt. However, be aware that your Django project must be running in a virtual environment, otherwise the install command will attempt to install all the python packages in your development machine.
The freeze command will allow you to have the current version of the app so upon installation will attempt to install that same version. Your requirements file will look something like:
Flask==0.8
Jinja2==2.6
Werkzeug==0.8.3
certifi==0.0.8
chardet==1.0.1
distribute==0.6.24
gunicorn==0.14.2
requests==0.11.1
Your packages are installed (if using virtualenv) at: ../<your project>/<your virtual env>/<lib>/<python version>/<site-packages>/
As for downloading you can use pip install --download command as #atupal suggested in his response, however think if this is really needed you can also fork those libraries on github to accomplish the same.
Here is a good source of information on how this works: http://www.pip-installer.org/en/latest/cookbook.html
Maybe what you want is:
Download the packages:
pip install --download /path/to/download/to packagename
OR
pip install --download=/path/to/packages/downloaded -r requirements.txt
install all of those libraries just downloaded:
pip install --no-index --find-links="/path/to/downloaded/dependencies" packagename
OR
pip install --no-index --find-links="/path/to/downloaded/packages" -r requirements.txt
Shamelessly stolen from this question
Create a requirements.txt file.
Put:
django==1.5.1
in the first line.
Then run pip install -r requirements.txt
Then you can complete that file...
I am trying to install a python software using the requirements file.
>> cat requirements.txt
Cython==0.15.1
numpy==1.6.1
distribute==0.6.24
logilab-astng==0.23.1logilab-common==0.57.1
netaddr==0.7.6
numexpr==2.0.1
ply==2.5
pycallgraph==0.5.1
pyflowtools==0.3.4.1
pylint==0.25.1
tables==2.3.1
wsgiref==0.1.2
So I create a virtual environment
>> mkvirtualenv parser
(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2
(parser)
>> pip install -r requirements.txt
... and then I packages downloaded but not installed with errors: http://pastie.org/4079800
(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2
Surprisingly, if I try to manually install each package, they install just fine.
For instance:
>> pip install numpy==1.6.1
(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2
numpy==1.6.1
I am lost. What is going on?
PS: I am using pip v1.1 and python v2.7.2 with virtualenv and virtualenvwrapper
It looks like the numexpr package has an install-time dependency on numpy. Pip makes two passes through your requirements: first it downloads all packages and runs each one's setup.py to get its metadata, and then it installs them all in a second pass.
So, numexpr is trying to import from numpy in its setup.py, but when pip first runs numexpr's setup.py, it has not yet installed numpy.
This is also why you don't see this error when you install the packages one by one: if you install them one at a time, numpy will be fully installed in your environment before you pip install numexpr.
The only solution is to install pip install numpy before you ever run pip install -r requirements.txt -- you won't be able to do this in a single command with a single requirements.txt file.
More info here: https://github.com/pypa/pip/issues/25
I come across with a similar issue and I ended up with the below:
cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 python -m pip install
That will read line by line the requirements.txt and execute pip. I cannot find from where I got the answer properly, so apologies for that, but I found some justification below:
How sed works: https://howto.lintel.in/truncate-empty-lines-using-sed/
Another similar answer but with git: https://stackoverflow.com/a/46494462/7127519
Hope this help with alternatives.
This is quite annoying sometimes, a bug of pip.
When you run pip install package_name the pip will first run pip check to the target package, and install all the required package for the dependency(target package).
But when you run pip install -r requirements.txt pip will try to directly install all the required packages listed one by one from top to bottom. Sometimes the dependency is listed above the package it depend upon.
The solution is simple:
1.pip install package_name
2.simply put the error package to the bottom of the requirements.txt
3.sometimes a particular version of the package is not be able to be installed,just install the newest version of it and update the data in requirements.txt