I notice that if I install a package with optional dependencies (setup.py extras_require):
$ pip install mypackage[optional]==1.0.0
I do not see the optional dependency in the output of pip freeze:
$ pip freeze | grep mypackage
mypackage==1.0.0
However, I'm using a requirements-to-freeze workflow that involves using pip freeze to update the requirements file with exact versions I use in production. So, what I would like to see is:
mypackage[optional]==1.0.0
Is there a way to achieve this?
Update:
Someone pointed out to me that I actually don't need this in the requirements-to-freeze workflow. In that workflow (at least the version I'm using), you first run pip install -r requirements-to-freeze.txt, which has mypackage[option]. This will find the latest version and also install the optional dependencies. Later, when pip freeze is run, the optional dependencies will be in the list. So actually, the fact that mypackage shows up in pip freeze without the square bracket notation is fine. The desired outcome will still be achieved.
According to the pip freeze documentation, there is no option to do such that thing
Sometimes I download the python source code from github and don't know how to install all the dependencies. If there is no requirements.txt file I have to create it by hands.
The question is:
Given the python source code directory is it possible to create requirements.txt automatically from the import section?
You can use the following code to generate a requirements.txt file:
pip install pipreqs
pipreqs /path/to/project
more info related to pipreqs can be found here.
Sometimes you come across pip freeze, but this saves all packages in the environment including those that you don't use in your current project.
Use Pipenv or other tools is recommended for improving your development flow.
pip3 freeze > requirements.txt # Python3
pip freeze > requirements.txt # Python2
If you do not use a virtual environment, pigar will be a good choice for you.
For python3: (I have both python 2 and 3 on my machine, where python2 is the default)
# install
pip3 install pipreqs
# Run in current directory
python3 -m pipreqs.pipreqs .
python2:
pip install pipreqs
python -m pipreqs.pipreqs .
To check your python version:
python --version
In my case, I use Anaconda, so running the following command from conda terminal inside my environment solved it, and created this requirements.txt file for me automatically:
conda list -e > requirements.txt
This was taken from this Github link pratos/condaenv.txt
If an error been seen, and you are using anaconda, try to use the .yml option:
conda env export > <environment-name>.yml
For other person to use the environment or if you are creating a new enviroment on another machine:
conda env create -f <environment-name>.yml
.yml option been found here
Kinda mind-blowing how this simple task is so complicated in Python. Here is what I think is the best way to do it automatically.
You need two tools:
1.pipreqs
pip3 install pipreqs
pipreqs will go through your project and only install the packages that your project use. Instead of all the packages in your python environment as pip freeze would do.
But there's a problem with this approach. It does not install the sub-packages.
For example, your project uses pandas==1.3.2. pandas itself uses numpy==1.21.2 among other packages. But pipreqs itself does not write the sub-packages (i.e. numpy) in requirments.txt
This is where you need to combine pipreqs with the second tool.
pip-tools
pip3 install pip-tools
pip-tools will take the packages in requirements.in and generate the requirements.txt with all the sub-packages. For example, if you have
pandas==1.3.2 in requirements.in, pip-tools would generate
numpy==1.21.2 # via pandas in requirements.txt.
But you need to manually add the package in requirements.in. Which is prone to mistake and you might forget to do this once in a while.
This is where you can use the first tool.
But both the tools write to requirements.txt. So how do you fix it?
Use the --savepath for pipreqs to write in requirements.in instead of the default requirements.txt.
To do it in one command; just do
pipreqs --savepath=requirements.in && pip-compile
There you go. Now you don't need to worry about manually maintaining the packages and you're requirements.txt will have all the sub-packages so that your build is deterministic.
TL;DR
pip3 install pipreqs
pip3 install pip-tools
Use the following to build a deterministic requirements.txt
pipreqs --savepath=requirements.in && pip-compile
As most of the answers using pipreqs didn't work for me. Here, is my answer.
To generate the requirements.txt file:
pip install pipreqs
python -m pipreqs.pipreqs --encoding utf-8 /path/to/project
I prefer using pipreqs more than pip freeze, as pip freeze saves all packages in the environment including those that you don't use in your current project. However, pipreqs only save the ones you are using in your project.
To install the requirements use:
pip3 install -r requirements.txt
I blindly followed the accepted answer of using
pip3 freeze > requirements.txt
It generated a huge file that listed all the dependencies of the entire solution, which is not what I wanted.
So you need to figure out what sort of requirements.txt you are trying to generate.
If you need a requirements.txt file that has ALL the dependencies, then use the pip3
pip3 freeze > requirements.txt
However, if you want to generate a minimal requirements.txt that only lists the dependencies you need, then use the pipreqs package. Especially helpful if you have numerous requirements.txt files in per component level in the project and not a single file on the solution wide level.
pip install pipreqs
pipreqs [path to folder]
e.g. pipreqs .
pipreqs . --force --ignore=tests (Overwrites exisiting requirements.txt, ignores the tests directory)
Firstly, your project file must be a py file which is direct python file. If your file is in ipynb format, you can convert it to py type by using the line of code below:
jupyter nbconvert --to=python
Then, you need to install pipreqs library from cmd (terminal for mac).
pip install pipreqs
Now we can create txt file by using the code below. If you are in the same path with your file, you can just write ./ . Otherwise you need to give path of your file.
pipreqs ./
or
pipreqs /home/project/location
That will create a requirements.txt file for your project.
Make sure to run pip3 for python3.7.
pip3 freeze >> yourfile.txt
Before executing the above command make sure you have created a virtual environment.
python3:
pip3 install virtualenv
python3 -m venv <myenvname>
python2:
pip install virtualenv
virtualenv <myenvname>
After that put your source code in the directory. If you run the python file now, probably it won't launch if you are using non-native modules. You can install those modules by running pip3 install <module> or pip install <module>.
This will not affect you entire module list except the environment you are in.
Now you can execute the command at the top and now you have a requirements file which contains only the modules you installed in the virtual environment. Now you can run the command at the top.
I advise everyone to use environments as it makes things easier when it comes to stuff like this.
Simple Pythonic Way
To get a list of all the REQUIREMENTS in a standard requirements.txt file, you can use the following command.
pip freeze > requirements.txt
Now, this should automatically create a standard requirements file with all of the packages installed alongside their corresponding versions.
Pretty Print on Terminal
If you just want to get a pretty print on the terminal you can use the following approach.
pip list
This lists all of the installed packages, in a pretty print format.
Custom Dependency
If you have a project folder like say, a Github Repo, and you want to get a custom requirements.txt for project You can use the following Package.
https://pypi.org/project/pipreqs/ pipreqs
Usage
$ pipreqs /home/project/location
Successfully saved requirements file in /home/project/location/requirements.txt
Contents of requirements.txt
wheel==0.23.0
Yarg==0.1.9
docopt==0.6.2
If you have installed many dependencies in your system and you need requirements.txt for a specific project, you can install first pipreqs:
$ pip install pipreqs
and execute the below command under the project folder.
$ pipreqs
This command will generate requirements.txt file for the particular project.
Automatic requirements.txt updating approach
While developing a python application with requirements.txt we have several choices:
Generate requirements.txt after development, when we want to deploy it. It is performed by pip freeze > requirements.txt or pipreqs for less messy result.
Add every module to requirements.txt manually after each install.
Install manager that will handle requirements.txt updates for us.
There are many answers for the 1-st option, the 2-d option is self-explanatory, so I would like to describe the 3-d approach. There is a library called to-requirements.txt. To install it type this:
pip install to-requirements.txt # Pip install to requirements.txt
If you read the whole command at once you would see, what it does. After installing you should setup it. Run:
requirements-txt setup
It overrides the pip scripts so that each pip install or pip uninstall updates the requirements.txt file of your project automatically with required versions of packages. The overriding is made safely, so that after uninstalling this package the pip will behave ordinary.
And you could customize the way it works. For example, disable it globally and activate it only for the required directories, activate it only for git repositories, or allow / disallow to create requirements.txt file if it does not exist.
Links:
Documentation - https://requirements-txt.readthedocs.io/en/latest/
GitHub - https://github.com/VoIlAlex/requirements-txt
PyPI - https://pypi.org/project/to-requirements.txt/
If Facing the same issue as mine i.e. not on the virtual environment and wants requirements.txt for a specific project or from the selected folder(includes children) and pipreqs is not supporting.
You can use :
import os
import sys
from fuzzywuzzy import fuzz
import subprocess
path = "C:/Users/Username/Desktop/DjangoProjects/restAPItest"
files = os.listdir(path)
pyfiles = []
for root, dirs, files in os.walk(path):
for file in files:
if file.endswith('.py'):
pyfiles.append(os.path.join(root, file))
stopWords = ['from', 'import',',','.']
importables = []
for file in pyfiles:
with open(file) as f:
content = f.readlines()
for line in content:
if "import" in line:
for sw in stopWords:
line = ' '.join(line.split(sw))
importables.append(line.strip().split(' ')[0])
importables = set(importables)
subprocess.call(f"pip freeze > {path}/requirements.txt", shell=True)
with open(path+'/requirements.txt') as req:
modules = req.readlines()
modules = {m.split('=')[0].lower() : m for m in modules}
notList = [''.join(i.split('_')) for i in sys.builtin_module_names]+['os']
new_requirements = []
for req_module in importables:
try :
new_requirements.append(modules[req_module])
except KeyError:
for k,v in modules.items():
if len(req_module)>1 and req_module not in notList:
if fuzz.partial_ratio(req_module,k) > 90:
new_requirements.append(modules[k])
new_requirements = [i for i in set(new_requirements)]
new_requirements
with open(path+'/requirements.txt','w') as req:
req.write(''.join(new_requirements))
P.S: It may have a few additional libraries as it checks on fuzzylogic.
best way for Python 3 is:
pip3 freeze > requirements.txt
it worked for me...
If you want to list only packages used inside a virtualenv use:
pip freeze -l > requirements.txt
You can just do it with command. it will create requirement.txt and add relevant modules automatically.
For Unix : pip3 freeze > requirements.txt
For Windos: pip freeze > requirements.txt
CREATE requirement.txt:
For Python 3 version command is:
pip3 freeze > requirements.txt
For Python 2 version command is:
pip freeze > requirements.txt
Install requirements.txt:
For Python 3 version command is:
pip3 install -r requirements.txt
For Python 2 version command is:
pip install -r requirements.txt
Not a complete solution, but may help to compile a shortlist on Linux.
grep --include='*.py' -rhPo '^\s*(from|import)\s+\w+' . | sed -r 's/\s*(import|from)\s+//' | sort -u > requirements.txt
Or if your are using a something like virtualenv you can just run this command to generate a requirements.txt
$ ./.venv/bin/pip freeze > requirements.txt
I created this bash command.
for l in $(pip freeze); do p=$(echo "$l" | cut -d'=' -f1); f=$(find . -type f -exec grep "$p" {} \; | grep 'import'); [[ ! -z "$f" ]] && echo "$l" ; done;
#Francis has it right - https://stackoverflow.com/a/65728461/1021819
But just to add:
With additional support for Jupyter notebooks - i.e. .ipynb files - you can now use https://pypi.org/project/pipreqsnb (same syntax as pipreqs):
pip install pipreqsnb
pipreqsnb .
[I am not an author]
Using pip freeze > requirements.txt is a bad way to create the requirements file! It can serve as a temporary solution for your problem but when managing requirements for python project it is best to do it manually.
A simple search for "import" or "from x import" will give you the list of all dependencies that need to be installed (nothing extra).
The problem with pip freeze it that it simply dumps all installed packages with strict versions, every dependency has its own dependencies and they are included in the dump.
For example, you have lib==1.0 installed, that needs sub-lib==0.5, if you use pip freeze you'll get both, but later when you wish to update the version of lib to 2.0, most likely you'll get conflicts since lib v2.0 now uses sub-lib v1.0 not v0.5 that you require... This gets complex fast for multiple dependencies.
We got into those problems in a couple of projects, since then I created an automated script to clean pip freeze's dumps, it is safe (comments unneeded dependencies) and works great.
To help solve this problem, always run requirements.txt on only local packages. By local packages I mean packages that are only in your project folder. To do this do:
Pip freeze —local > requirements.txt
Not pip freeze > requirements.txt.
Note that it’s double underscore before local.
However installing pipreqs helps too.
Pip install pipreqs.
The perfect solution though is to have a pipfile. The pipfile updates on its own whenever you install a new local package. It also has a pipfile.lock similar to package.json in JavaScript.
To do this always install your packages with pipenv not pip.
So we do pipenv
Pipenv users can generate the requirement.txt file from the project's Pipfile with:
pipenv lock --requirements
Why does pip list generate a more comprehensive list than pip freeze?
$ pip list
feedparser (5.1.3)
pip (1.4.1)
setuptools (1.1.5)
wsgiref (0.1.2)
$ pip freeze
feedparser==5.1.3
wsgiref==0.1.2
Pip's documentation states:
freeze
Output installed packages in requirements format.
list
List installed packages.
What is a "requirements format"?
One may generate a requirements.txt via:
$ pip freeze > requirements.txt
A user can use this requirements.txt file to install all the dependencies. For instance:
$ pip install -r requirements.txt
The packages need to be in a specific format for pip to understand, such as:
# requirements.txt
feedparser==5.1.3
wsgiref==0.1.2
django==1.4.2
...
That is the "requirements format".
Here, django==1.4.2 implies install django version 1.4.2 (even though the latest is 1.6.x).
If you do not specify ==1.4.2, the latest version available would be installed.
You can read more in "Virtualenv and pip Basics",
and the official "Requirements File Format" documentation.
To answer the second part of this question, the two packages shown in pip list but not pip freeze are setuptools (which is easy_install) and pip itself.
It looks like pip freeze just doesn't list packages that pip itself depends on. You may use the --all flag to show also those packages.
From the documentation:
--all
Do not skip these packages in the output: pip, setuptools, distribute, wheel
The main difference is that the output of pip freeze can be dumped into a requirements.txt file and used later to re-construct the "frozen" environment.
In other words you can run:
pip freeze > frozen-requirements.txt on one machine and then later on a different machine or on a clean environment you can do:
pip install -r frozen-requirements.txt
and you'll get the an identical environment with the exact same dependencies installed as you had in the original environment where you generated the frozen-requirements.txt.
Look at the pip documentation, which describes the functionality of both as:
pip list
List installed packages, including editables.
pip freeze
Output installed packages in requirements format.
So there are two differences:
Output format, freeze gives us the standard requirement format that may be used later with pip install -r to install requirements from.
Output content, pip list include editables which pip freeze does not.
pip list shows ALL installed packages.
pip freeze shows packages YOU installed via pip (or pipenv if using that tool) command in a requirements format.
Remark below that setuptools, pip, wheel are installed when pipenv shell creates my virtual envelope. These packages were NOT installed by me using pip:
test1 % pipenv shell
Creating a virtualenv for this project…
Pipfile: /Users/terrence/Development/Python/Projects/test1/Pipfile
Using /usr/local/Cellar/pipenv/2018.11.26_3/libexec/bin/python3.8 (3.8.1) to create virtualenv…
⠹ Creating virtual environment...
<SNIP>
Installing setuptools, pip, wheel...
done.
✔ Successfully created virtual environment!
<SNIP>
Now review & compare the output of the respective commands where I've only installed cool-lib and sampleproject (of which peppercorn is a dependency):
test1 % pip freeze <== Packages I'VE installed w/ pip
-e git+https://github.com/gdamjan/hello-world-python-package.git#10<snip>71#egg=cool_lib
peppercorn==0.6
sampleproject==1.3.1
test1 % pip list <== All packages, incl. ones I've NOT installed w/ pip
Package Version Location
------------- ------- --------------------------------------------------------------------------
cool-lib 0.1 /Users/terrence/.local/share/virtualenvs/test1-y2Zgz1D2/src/cool-lib <== Installed w/ `pip` command
peppercorn 0.6 <== Dependency of "sampleproject"
pip 20.0.2
sampleproject 1.3.1 <== Installed w/ `pip` command
setuptools 45.1.0
wheel 0.34.2
My preferred method of generating a requirements file is:
pip list --format=freeze > requirements.txt
This method keeps just the package names and package versions without potentially linking to local file paths which 'pip freeze' alone will sometimes give me. Local file paths in a requirements file make your codebase harder to use for other users and some developers don't know how to fix this so I prefer this method for ease of adoptability.
pip list
List installed packages: show ALL installed packages that even pip installed implictly
pip freeze
List installed packages: - list of packages that are installed using pip command
pip freeze has --all flag to show all the packages.
Other difference is the output it renders, that you can check by running the commands.
For those looking for a solution. If you accidentally made pip requirements with pip list instead of pip freeze, and want to convert into pip freeze format. I wrote this R script to do so.
library(tidyverse)
pip_list = read_lines("requirements.txt")
pip_freeze = pip_list %>%
str_replace_all(" \\(", "==") %>%
str_replace_all("\\)$", "")
pip_freeze %>% write_lines("requirements.txt")
I am trying to install a python software using the requirements file.
>> cat requirements.txt
Cython==0.15.1
numpy==1.6.1
distribute==0.6.24
logilab-astng==0.23.1logilab-common==0.57.1
netaddr==0.7.6
numexpr==2.0.1
ply==2.5
pycallgraph==0.5.1
pyflowtools==0.3.4.1
pylint==0.25.1
tables==2.3.1
wsgiref==0.1.2
So I create a virtual environment
>> mkvirtualenv parser
(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2
(parser)
>> pip install -r requirements.txt
... and then I packages downloaded but not installed with errors: http://pastie.org/4079800
(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2
Surprisingly, if I try to manually install each package, they install just fine.
For instance:
>> pip install numpy==1.6.1
(parser)
>> pip freeze
distribute==0.6.24
wsgiref==0.1.2
numpy==1.6.1
I am lost. What is going on?
PS: I am using pip v1.1 and python v2.7.2 with virtualenv and virtualenvwrapper
It looks like the numexpr package has an install-time dependency on numpy. Pip makes two passes through your requirements: first it downloads all packages and runs each one's setup.py to get its metadata, and then it installs them all in a second pass.
So, numexpr is trying to import from numpy in its setup.py, but when pip first runs numexpr's setup.py, it has not yet installed numpy.
This is also why you don't see this error when you install the packages one by one: if you install them one at a time, numpy will be fully installed in your environment before you pip install numexpr.
The only solution is to install pip install numpy before you ever run pip install -r requirements.txt -- you won't be able to do this in a single command with a single requirements.txt file.
More info here: https://github.com/pypa/pip/issues/25
I come across with a similar issue and I ended up with the below:
cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 python -m pip install
That will read line by line the requirements.txt and execute pip. I cannot find from where I got the answer properly, so apologies for that, but I found some justification below:
How sed works: https://howto.lintel.in/truncate-empty-lines-using-sed/
Another similar answer but with git: https://stackoverflow.com/a/46494462/7127519
Hope this help with alternatives.
This is quite annoying sometimes, a bug of pip.
When you run pip install package_name the pip will first run pip check to the target package, and install all the required package for the dependency(target package).
But when you run pip install -r requirements.txt pip will try to directly install all the required packages listed one by one from top to bottom. Sometimes the dependency is listed above the package it depend upon.
The solution is simple:
1.pip install package_name
2.simply put the error package to the bottom of the requirements.txt
3.sometimes a particular version of the package is not be able to be installed,just install the newest version of it and update the data in requirements.txt