Develop Reference

How to install test requirements from setup.py [duplicate]

We've made a library which depends on other libraries. But there are necessary (e.g. for server batch processing) and optional dependencies (e.g. for clients with GUI).
Is something like this possible:
pip install mylib.tar.gz # automatically downloads and installs with the minimal set of dependencies
pip install mylib.tar.gz --install-option="complete" # automatically installs with all dependencies
I've found the extra_require flag, but how can I tell pip to use them? The setup.py looks like this:
from setuptools import setup
# ...
# Hard library depencencies:
requires = [
"numpy>=1.4.1",
"scipy>=0.7.2",
"traits>=3.4.0"
]
# Soft library dependencies:
recommended = {
"mpl": ["matplotlib>=0.99.3"],
"bn": ["bottleneck>=0.6"]
}
# ...
# Installer parameters:
setup(
name = "mylib",
#...
install_requires = requires,
extras_require = recommended
)

You can install the packages in extras_require by appending the name of the recommended dependency in square brackets (i.e. [mpl] or [bn] in your case) to the package name in pip.
So to install 'mylib' with the additional requirements, you would call pip like this:
pip install 'mylib[mpl]'
pip install 'mylib[bn]'
This will first download and install the extra dependencies, and then mylib's core dependencies.
This is anologous to how you declare those dependencies with setuptools: http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies (see the install_requires value in the third example)

So pip is actually quite picky about installing libraries with extra requirements
pip install -e ".[extra,requirements]" # works with file paths
pip install "package[extra,requirements]" # works when downloading packages
pip install ".[extra,requirments]" # DOES NOT WORK
I think this is down to how the RequirementsSpec parser works, and pip does some extra magic with the -e flag. Anyhow after much head banging, here's a mildly ugly workaround
pip install "file:///path/to/your/python_code#egg=SomeName[extra,requirements]"
The egg=SomeName part is basically ignored, but pip correctly picks up the extra requirements
Caveats
Tested with pip 1.5.6 so make sure you're using a current version of pip.
As far as I can tell, the file:/// syntax is undocumented in pip, so I'm not sure if it'll change in the future. It looks a bit like the VCS Support syntax but I was a bit surprised it worked.
You could also get around this by running your own pypi server, but that's a bit out of scope.

What do square brackets mean in pip install?

I see more and more commands like this:
$ pip install "splinter[django]"
What do these square brackets do?

The syntax that you are using is:
pip install "project[extra]"
In your case, you are installing the splinter package which has the added support for django. The square brackets ([]) are not specific syntax, just convention. Really, you are installing the package named: "splinter[django]".
An explanation from #chetner:
The command pip install splinter django would install two packages named splinter and django. splinter[django], on the other hand, installs a variant of the splinter package which contains support for django. Note that it has nothing to do with the django package itself, but is just a string defined by the splinter package for a particular feature set that gets enabled.

Brackets [optional] in PIP signify optional dependencies
Just in case another developer comes along looking to implement this pattern in their own Python package deployment, here's further explanation of the brackets [] in pip.
For Example: Apache Airflow
To install airflow from pip we use this command:
pip install 'apache-airflow'
You can install optional components of airflow with:
pip install 'apache-airflow[aws]'
# [optional] -----------^
When we search pypi for apache-airflow note that the optional packages do not show up:
pip search 'apache-airflow'
apache-airflow (1.10.9) - Programmatically author, schedule and monitor data pipelines
pylint-airflow (0.1.0a1) - A Pylint plugin to lint Apache Airflow code.
swe-airflow-tools (0.0.3) - Tools for Apache Airflow Application
airflow (0.6) - Placeholder for the old Airflow package
...
Implementation via setup.py
You can see how this was accomplished in the setup.py script
On the left in setup.py - extras_require is defined.
On the right are the correlated installation commands for these optional sub-packages.

Pretty sure these are setuptools extras:
https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies
Sometimes a project has “recommended” dependencies, that are not required for all uses of the project. For example, a project might offer optional PDF output if ReportLab is installed, and reStructuredText support if docutils is installed. These optional features are called “extras” ...

Maybe worthwhile to know that this optional package syntax admits multiple extras (separated by comma within the brackets) as in:
python -m pip install SomePackage[PDF,EPUB] # multiple extras
As per the pip manual

TLDR
The square bracket contains the 'extra' option's information defined in setup.py that pip will use to install additional dependencies.
pip install "splinter[django]"
To be specific, the above line will install first the 'splinter' package, then install the extra dependencies the 'splinter' project requires with the 'django' option specified in a setup.py of 'splinter' project.
Explanation
pip install "splinter[django]"
pip install "splinter" "Django>=2.0.6" "lxml>=4.2.4" "cssselect"
As of splinter==0.16.0, with python==3.9.2, the above two commands are equivalent.
Both pip install will result in the following packages given a clean virtual enviroment.
The reason why the two pip install commands achieve same is because this is literally what has been run in the background based on the setup.py of the splinter package
The '[django]' is the 'extra' option for the 'splinter' package. Pip will look into the setup.py of splinter package, and find what needs to be installed with the '[django]' option specified. In this case, it is these 3 packages: ["Django>=2.0.6", "lxml>=4.2.4", "cssselect"]

This is exactly the list from the setup.py file for the project in question:
"django": ["Django>=1.7.11;python_version<'3.0'", "Django>=2.0.6;python_version>'3.3'", "lxml>=2.3.6", "cssselect", "six"],

Python install sub-package from package

it is possible to install some special sub-package from package?
For example, I want to create package with slack, datadog, sentry plugins (wrappers). But I want to allow user what he wants to install.
Like:
pip install super_plugins --plugins slack, datadog
Can it be done without separating all plugins to different packages?

Actually, It is quite simple. This is called Packaging namespace packages.
https://packaging.python.org/guides/packaging-namespace-packages/
All you need is to separate all packages to sub - packages and after install it with a namespace.
# for all packages
pip install super_plugins
# for specific
pip install super_plugins.slack super_plugins.datadog

setuptools and pip: choice of minimal and complete install

We've made a library which depends on other libraries. But there are necessary (e.g. for server batch processing) and optional dependencies (e.g. for clients with GUI).
Is something like this possible:
pip install mylib.tar.gz # automatically downloads and installs with the minimal set of dependencies
pip install mylib.tar.gz --install-option="complete" # automatically installs with all dependencies
I've found the extra_require flag, but how can I tell pip to use them? The setup.py looks like this:
from setuptools import setup
# ...
# Hard library depencencies:
requires = [
"numpy>=1.4.1",
"scipy>=0.7.2",
"traits>=3.4.0"
]
# Soft library dependencies:
recommended = {
"mpl": ["matplotlib>=0.99.3"],
"bn": ["bottleneck>=0.6"]
}
# ...
# Installer parameters:
setup(
name = "mylib",
#...
install_requires = requires,
extras_require = recommended
)

You can install the packages in extras_require by appending the name of the recommended dependency in square brackets (i.e. [mpl] or [bn] in your case) to the package name in pip.
So to install 'mylib' with the additional requirements, you would call pip like this:
pip install 'mylib[mpl]'
pip install 'mylib[bn]'
This will first download and install the extra dependencies, and then mylib's core dependencies.
This is anologous to how you declare those dependencies with setuptools: http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies (see the install_requires value in the third example)

So pip is actually quite picky about installing libraries with extra requirements
pip install -e ".[extra,requirements]" # works with file paths
pip install "package[extra,requirements]" # works when downloading packages
pip install ".[extra,requirments]" # DOES NOT WORK
I think this is down to how the RequirementsSpec parser works, and pip does some extra magic with the -e flag. Anyhow after much head banging, here's a mildly ugly workaround
pip install "file:///path/to/your/python_code#egg=SomeName[extra,requirements]"
The egg=SomeName part is basically ignored, but pip correctly picks up the extra requirements
Caveats
Tested with pip 1.5.6 so make sure you're using a current version of pip.
As far as I can tell, the file:/// syntax is undocumented in pip, so I'm not sure if it'll change in the future. It looks a bit like the VCS Support syntax but I was a bit surprised it worked.
You could also get around this by running your own pypi server, but that's a bit out of scope.

Using pip to install single-file python modules

I'm wondering if there's a way to "install" single-file python modules using pip (i.e. just have pip download the specified version of the file and copy it to site-packages).
I have a Django project that uses several 3rd-party modules which aren't proper distributions (django-thumbs and a couple others) and I want to pip freeze everything so the project can be easily installed elsewhere. I've tried just doing
pip install git+https://github.com/path/to/file.git
(and tried with the -e tag too) but pip complains that there's no setup.py file.
Edit: I should have mentioned - the reason I want to do this is so I can include the required module in a requirements.txt file, to make setting up the project on a new machine or new virtualenv easier.

pip requires a valid setup.py to install a python package. By definition every python package has a setup.py... What you are trying to install isn't a package but rather a single file module... what's wrong with doing something like:
git clone git+https://github.com/path/to/file.git /path/to/python/install/lib
I don't quite understand the logic behind wanting to install something that isn't a package with a package manager...