How can I specify optional dependencies in a pip requirements file?
According to the pip documentation this is possible, but the documentation doesn't explain how to do it, and I can't find any examples on the web.
Instead of specifying optional dependencies in the same file as the hard requirements, you can create a optional-requirements.txt and a requirements.txt.
To export your current environment's packages into a text file, you can do this:
pip freeze > requirements.txt
If necessary, modify the contents of the requirements.txt to accurately represent your project's dependencies. Then, to install all the packages in this file, run:
pip install -U -r requirements.txt
-U tells pip to upgrade packages to the latest version, and -r tells it to install all packages in requirements.txt.
In 2015 PEP-0508 defined a way to specify optional dependencies in requirements.txt:
requests[security]
That means that yourpackage needs requests for its security option. You can install it as:
pip install yourpackage[security]
You are misunderstanding the documentation; it's not as clear as it could be. The point in the documentation is that with a requirements file you can feel free to specify your full recommended working set of packages, including both necessary dependencies and optional ones.
You can add comments (lines beginning with #) to distinguish the two to humans, but pip makes no distinction. You can also have two requirements files, as Daniel suggests.
Related
I would like to create a virtual environment for my team. My team, works in different places and everyone has their own environment, it causes a lot of problems, everyone has a different version of the libraries (Python, RobotFramework).
I thought about:
creating one common environment, I used virtualenv.
Installing the prepared libraries (python and robotframework) with one command pip install ...,
Prepared libraries will be in the git repository so that everyone can modify them, change the library version.
I have the first and third parts done, but I have a problem with the second. How to create such a package of libraries to be able to install it with one pip install command.
Should I create an environment locally, install all the libraries in it, and send them to git? Or should I package the project via setuptool (to tar.gz)?
Unfortunately, I cannot find the answer to this question, it seems to me that none of the above solutions is optimal.
The easiest way of doing it would be creating a text file of all the libraries you are using in pip with the command.
pip freeze > requirements.txt
This will create a file listing all the packages with their versions that are being used. To install that ask every team member to place that requirement file in their projects and use
pip install -r requirements.txt
With pip, you could download your dependencies. These will be .tar.gz, .whl or .zip files. Note that this could be complicated if your team uses multiple OS.
Here is an example which will download the dependencies into the directory named "dependencies", you can push this to git along with the requirements file.
pip freeze > req.txt
pip download -r req.txt -d dependencies
When someone clones your repository, they can install the dependencies offline with the following command.
pip install --no-index --find-links=dependencies -r req.txt
Package managers for JavaScript like npm and yarn use a package.json to specify 'top-level' dependencies, and create a lock-file to keep track of the specific versions of all packages (i.e. top-level and sub-level dependencies) that are installed as a result.
In addition, the package.json allows us to make a distinction between types of top-level dependencies, such as production and development.
For Python, on the other hand, we have pip. I suppose the pip equivalent of a lock-file would be the result of pip freeze > requirements.txt.
However, if you maintain only this single requirements.txt file, it is difficult to distinguish between top-level and sub-level dependencies (you would need for e.g. pipdeptree -r to figure those out). This can be a real pain if you want to remove or change top-level dependencies, as it is easy to be left with orphaned packages (as far as I know, pip does not remove sub-dependencies when you pip uninstall a package).
Now, I wonder: Is there some convention for dealing with different types of these requirements files and distinguishing between top-level and sub-level dependencies with pip?
For example, I can imagine having a requirements-prod.txt which contains only the top-level requirements for the production environment, as the (simplified) equivalent of package.json, and a requirements-prod.lock, which contains the output of pip freeze, and acts as my lock-file. In addition I could have a requirements-dev.txt for development dependencies, and so on and so forth.
I would like to know if this is the way to go, or if there is a better approach.
p.s. The same question could be asked for conda's environment.yml.
There are at least three good options available today:
Poetry uses pyproject.toml and poetry.lock files, much in the same way that package.json and lock files work in the JavaScript world.
This is now my preferred solution.
Pipenv uses Pipfile and Pipfile.lock, also much like you describe the JavaScript files.
Both Poetry and Pipenv do more than just dependency management. Out of the box, they also create and maintain virtual environments for your projects.
pip-tools provides pip-compile and pip-sync commands. Here, requirements.in lists your direct dependencies, often with loose version constraints and pip-compile generates locked down requirements.txt files from your .in files.
This used to be my preferred solution. It's backwards-compatible (the generated requirements.txt can be processed by pip) and the pip-sync tool ensures that the virtualenv exactly matches the locked versions, removing things that aren't in your "lock" file.
I had the same question and I came up with a more generic and simple solution. I am using the well-known requirements.txt for all explicit dependencies and requirements.lock as a list of all packages including sub dependencies.
I personally like to manage python, pip and setuptools via the distributions builtin package manager and install pip dependencies inside a virtual environment.
Usually you would start installing all directly required dependencies. This will pull in all sub dependencies as well. If you are not using a virtual environment make sure to add the --user flag.
# If you already have a requirements file
pip3 install -r requirements.txt
# If you start from scratch
pip3 install <package>
If you want to upgrade your packages, you have multiple options here as well. Since I am using a virtual environment I will always update all packages. However you are free to only update your direct requirements. If they need an update of their dependencies, those will be pulled in as well, everything else will be left untouched.
# Update all outdated packages (excluding pip and setuptools itself)
pip3 install -r <(pip3 list --outdated --format freeze --exclude pip setuptools | cut -d '=' -f1) --upgrade
# Update explicitly installed packages, update sub dependencies only if required.
pip3 install -r <(cut -d '=' -f1 requirements.txt) --upgrade
Now we come to the tricky part: Saving back our requirements file. Make sure that the previous requirements file is checked into git, so if anything goes wrong you have a backup.
Remember that we want to differentiate between packages explicitly installed (requirements.txt) and packages including their dependencies (requirements.lock).
If you have not yet setup a requirements.txt I suggest running the following command. Note that it will not include sub dependencies if they are already satisfied by another package. This means requests will not be included in the list, if it was already satisfied by another package. You might still want to add that manually, if your script explicitly relies on such a package.
pip3 list --not-required --format freeze --exclude pip --exclude setuptools > requirements.txt
If you already have a requirements.txt you can update it by using this sed trick. This will leave all sub-dependencies outside, which we will only include in the requirements.lock in the next step.
pip3 freeze -r requirements.txt | sed -n '/## The following requirements were added by pip freeze:/q;p' | sponge requirements.txt
Finally we can output all dependencies to a requirements.lock file which will be our complete list of all packages and versions. If we have issues to reproduce an issue, we can always come back to this lock file and run our code with the previously working dependencies.
# It is important to use the -r option here, so pip will differenciate between directly required packages and dependencies.
pip3 freeze -r requirements.txt > requirements.lock
To install dependences, the appengine-python-flask-skeleton docs advise running this command:
pip install -r requirements.txt -t lib
That works simply enough.
Now say I want to add the Requests package.
Ideally I just add it to the requirements.txt file:
# This requirements file lists all third-party dependencies for this project.
#
# Run 'pip install -r requirements.txt -t lib/' to install these dependencies
# in `lib/` subdirectory.
#
# Note: The `lib` directory is added to `sys.path` by `appengine_config.py`.
Flask==0.10
requests
And then re-run the command:
pip install -r requirements.txt -t lib
However, as this Github issue for pip notes, pip is not idempotent with the -T option recommended by Google here. The existing flask packages will be re-added and this will lead to the following error when running the devapp
ImportError: cannot import name exceptions
How can I best work around this problem?
Like said, updating pip solves the issue for many, but for what it's worth I think you can get around all of this if the use of virtualenv is an option. Symlink /path/to/virtualenv's/sitepackages/ to lib/ and just always keep an up to date requirements.txt file. There are no duplication of packages this way and one won't have to manually install dependencies. See also https://stackoverflow.com/a/30447848/2295256
Upgrading to the latest version of pip solved my problem (that issue had been closed):
pip install -U pip
Otherwise, as noted in that thread, you can always just wipe out your lib directory and reinstall from scratch. One note of warning: if you manually added additional packages to the lib directory not tracked in requirements.txt, they would be lost and have to be re-installed manually.
I want to install the 2.3 branch of PyMC.
I have tried with:
pip install -e git+git:://github.com/pymc-devs/pymc#2.3
and
pip install git+git:://github.com/pymc-devs/pymc#2.3
without luck. Do I need to specify egg information? If so why?
You have too many colons in your URL:
git+git:://github.com/pymc-devs/pymc#2.3
# ----^
Just one colon is required:
pip install git+git://github.com/pymc-devs/pymc#2.3
or, in editable mode:
pip install -e git+git://github.com/pymc-devs/pymc#2.3
Provided you have a Fortran compiler installed that Just Works.
The #egg=<projectname> part is optional and only used to test for dependencies before downloading. It lets pip test if the package is already installed without needing to clone the whole repository.
pip install --upgrade -r requirements.txt
repeats the installation process for all the previously installed dependencies which can be a pain when I have a huge list of dependencies (like more than 30?)
Isn't there a way to check an updated requirements.txt and install only specific dependencies that have been included into the requirements.txt file since the previous installation attempt?
I find this to be a real shortcoming of pip (or using pip in virtualenv for that matter). Do not like the repetitive installation nature of pip at all.
As mentioned by Piotr in the comments above, if "--upgrade" is not included in the command, already installed python packages are left alone.