We've made a library which depends on other libraries. But there are necessary (e.g. for server batch processing) and optional dependencies (e.g. for clients with GUI).
Is something like this possible:
pip install mylib.tar.gz # automatically downloads and installs with the minimal set of dependencies
pip install mylib.tar.gz --install-option="complete" # automatically installs with all dependencies
I've found the extra_require flag, but how can I tell pip to use them? The setup.py looks like this:
from setuptools import setup
# ...
# Hard library depencencies:
requires = [
"numpy>=1.4.1",
"scipy>=0.7.2",
"traits>=3.4.0"
]
# Soft library dependencies:
recommended = {
"mpl": ["matplotlib>=0.99.3"],
"bn": ["bottleneck>=0.6"]
}
# ...
# Installer parameters:
setup(
name = "mylib",
#...
install_requires = requires,
extras_require = recommended
)
You can install the packages in extras_require by appending the name of the recommended dependency in square brackets (i.e. [mpl] or [bn] in your case) to the package name in pip.
So to install 'mylib' with the additional requirements, you would call pip like this:
pip install 'mylib[mpl]'
pip install 'mylib[bn]'
This will first download and install the extra dependencies, and then mylib's core dependencies.
This is anologous to how you declare those dependencies with setuptools: http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies (see the install_requires value in the third example)
So pip is actually quite picky about installing libraries with extra requirements
pip install -e ".[extra,requirements]" # works with file paths
pip install "package[extra,requirements]" # works when downloading packages
pip install ".[extra,requirments]" # DOES NOT WORK
I think this is down to how the RequirementsSpec parser works, and pip does some extra magic with the -e flag. Anyhow after much head banging, here's a mildly ugly workaround
pip install "file:///path/to/your/python_code#egg=SomeName[extra,requirements]"
The egg=SomeName part is basically ignored, but pip correctly picks up the extra requirements
Caveats
Tested with pip 1.5.6 so make sure you're using a current version of pip.
As far as I can tell, the file:/// syntax is undocumented in pip, so I'm not sure if it'll change in the future. It looks a bit like the VCS Support syntax but I was a bit surprised it worked.
You could also get around this by running your own pypi server, but that's a bit out of scope.
Related
Related to this question: What do square brackets mean in pip install?
I have a setup.py file that needs to install azure[common] package. However, if I try:
setup(
...
install_requires=['azure[common]'],
...
)
This results in an error:
pkg_resources.UnknownExtra: azure 4.0.0 has no such extra feature 'common'
But, if I do:
pip install 'azure[common]', then it works.
There were a lot of bugs and unexpected behavior involved in the experiment above, so the question doens't really make sense anymore.
There's a bug in pip which causes random stuff to be installed if "extra" package isn't found. So, pip install 'azure[common]' shouldn't have worked at all. It's an error that led me to believe there was such a package.
There's an inconsistency between how setuptools and pip install packages from wheels. setuptools installs (or seems to) only install one package from a wheel, while pip will install everything, and if there are more than one package, then it will install more. So, pip was installing azure.common by mistake, but there is no way to intentionally install just that package. At the minimum, you will also get azure.profiles plus a fake package azure_common, which doesn't really contain anything.
Given all this new info, I reformulated the question here: How to make setuptools install a wheel containing multiple packages?
Azure does not provide the common extra dependency. pip install azure[common] shows the warning about it.
I see more and more commands like this:
$ pip install "splinter[django]"
What do these square brackets do?
The syntax that you are using is:
pip install "project[extra]"
In your case, you are installing the splinter package which has the added support for django. The square brackets ([]) are not specific syntax, just convention. Really, you are installing the package named: "splinter[django]".
An explanation from #chetner:
The command pip install splinter django would install two packages named splinter and django. splinter[django], on the other hand, installs a variant of the splinter package which contains support for django. Note that it has nothing to do with the django package itself, but is just a string defined by the splinter package for a particular feature set that gets enabled.
Brackets [optional] in PIP signify optional dependencies
Just in case another developer comes along looking to implement this pattern in their own Python package deployment, here's further explanation of the brackets [] in pip.
For Example: Apache Airflow
To install airflow from pip we use this command:
pip install 'apache-airflow'
You can install optional components of airflow with:
pip install 'apache-airflow[aws]'
# [optional] -----------^
When we search pypi for apache-airflow note that the optional packages do not show up:
pip search 'apache-airflow'
apache-airflow (1.10.9) - Programmatically author, schedule and monitor data pipelines
pylint-airflow (0.1.0a1) - A Pylint plugin to lint Apache Airflow code.
swe-airflow-tools (0.0.3) - Tools for Apache Airflow Application
airflow (0.6) - Placeholder for the old Airflow package
...
Implementation via setup.py
You can see how this was accomplished in the setup.py script
On the left in setup.py - extras_require is defined.
On the right are the correlated installation commands for these optional sub-packages.
Pretty sure these are setuptools extras:
https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies
Sometimes a project has “recommended” dependencies, that are not required for all uses of the project. For example, a project might offer optional PDF output if ReportLab is installed, and reStructuredText support if docutils is installed. These optional features are called “extras” ...
Maybe worthwhile to know that this optional package syntax admits multiple extras (separated by comma within the brackets) as in:
python -m pip install SomePackage[PDF,EPUB] # multiple extras
As per the pip manual
TLDR
The square bracket contains the 'extra' option's information defined in setup.py that pip will use to install additional dependencies.
pip install "splinter[django]"
To be specific, the above line will install first the 'splinter' package, then install the extra dependencies the 'splinter' project requires with the 'django' option specified in a setup.py of 'splinter' project.
Explanation
pip install "splinter[django]"
pip install "splinter" "Django>=2.0.6" "lxml>=4.2.4" "cssselect"
As of splinter==0.16.0, with python==3.9.2, the above two commands are equivalent.
Both pip install will result in the following packages given a clean virtual enviroment.
The reason why the two pip install commands achieve same is because this is literally what has been run in the background based on the setup.py of the splinter package
The '[django]' is the 'extra' option for the 'splinter' package. Pip will look into the setup.py of splinter package, and find what needs to be installed with the '[django]' option specified. In this case, it is these 3 packages: ["Django>=2.0.6", "lxml>=4.2.4", "cssselect"]
This is exactly the list from the setup.py file for the project in question:
"django": ["Django>=1.7.11;python_version<'3.0'", "Django>=2.0.6;python_version>'3.3'", "lxml>=2.3.6", "cssselect", "six"],
I see more and more commands like this:
$ pip install "splinter[django]"
What do these square brackets do?
The syntax that you are using is:
pip install "project[extra]"
In your case, you are installing the splinter package which has the added support for django. The square brackets ([]) are not specific syntax, just convention. Really, you are installing the package named: "splinter[django]".
An explanation from #chetner:
The command pip install splinter django would install two packages named splinter and django. splinter[django], on the other hand, installs a variant of the splinter package which contains support for django. Note that it has nothing to do with the django package itself, but is just a string defined by the splinter package for a particular feature set that gets enabled.
Brackets [optional] in PIP signify optional dependencies
Just in case another developer comes along looking to implement this pattern in their own Python package deployment, here's further explanation of the brackets [] in pip.
For Example: Apache Airflow
To install airflow from pip we use this command:
pip install 'apache-airflow'
You can install optional components of airflow with:
pip install 'apache-airflow[aws]'
# [optional] -----------^
When we search pypi for apache-airflow note that the optional packages do not show up:
pip search 'apache-airflow'
apache-airflow (1.10.9) - Programmatically author, schedule and monitor data pipelines
pylint-airflow (0.1.0a1) - A Pylint plugin to lint Apache Airflow code.
swe-airflow-tools (0.0.3) - Tools for Apache Airflow Application
airflow (0.6) - Placeholder for the old Airflow package
...
Implementation via setup.py
You can see how this was accomplished in the setup.py script
On the left in setup.py - extras_require is defined.
On the right are the correlated installation commands for these optional sub-packages.
Pretty sure these are setuptools extras:
https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies
Sometimes a project has “recommended” dependencies, that are not required for all uses of the project. For example, a project might offer optional PDF output if ReportLab is installed, and reStructuredText support if docutils is installed. These optional features are called “extras” ...
Maybe worthwhile to know that this optional package syntax admits multiple extras (separated by comma within the brackets) as in:
python -m pip install SomePackage[PDF,EPUB] # multiple extras
As per the pip manual
TLDR
The square bracket contains the 'extra' option's information defined in setup.py that pip will use to install additional dependencies.
pip install "splinter[django]"
To be specific, the above line will install first the 'splinter' package, then install the extra dependencies the 'splinter' project requires with the 'django' option specified in a setup.py of 'splinter' project.
Explanation
pip install "splinter[django]"
pip install "splinter" "Django>=2.0.6" "lxml>=4.2.4" "cssselect"
As of splinter==0.16.0, with python==3.9.2, the above two commands are equivalent.
Both pip install will result in the following packages given a clean virtual enviroment.
The reason why the two pip install commands achieve same is because this is literally what has been run in the background based on the setup.py of the splinter package
The '[django]' is the 'extra' option for the 'splinter' package. Pip will look into the setup.py of splinter package, and find what needs to be installed with the '[django]' option specified. In this case, it is these 3 packages: ["Django>=2.0.6", "lxml>=4.2.4", "cssselect"]
This is exactly the list from the setup.py file for the project in question:
"django": ["Django>=1.7.11;python_version<'3.0'", "Django>=2.0.6;python_version>'3.3'", "lxml>=2.3.6", "cssselect", "six"],
We've made a library which depends on other libraries. But there are necessary (e.g. for server batch processing) and optional dependencies (e.g. for clients with GUI).
Is something like this possible:
pip install mylib.tar.gz # automatically downloads and installs with the minimal set of dependencies
pip install mylib.tar.gz --install-option="complete" # automatically installs with all dependencies
I've found the extra_require flag, but how can I tell pip to use them? The setup.py looks like this:
from setuptools import setup
# ...
# Hard library depencencies:
requires = [
"numpy>=1.4.1",
"scipy>=0.7.2",
"traits>=3.4.0"
]
# Soft library dependencies:
recommended = {
"mpl": ["matplotlib>=0.99.3"],
"bn": ["bottleneck>=0.6"]
}
# ...
# Installer parameters:
setup(
name = "mylib",
#...
install_requires = requires,
extras_require = recommended
)
You can install the packages in extras_require by appending the name of the recommended dependency in square brackets (i.e. [mpl] or [bn] in your case) to the package name in pip.
So to install 'mylib' with the additional requirements, you would call pip like this:
pip install 'mylib[mpl]'
pip install 'mylib[bn]'
This will first download and install the extra dependencies, and then mylib's core dependencies.
This is anologous to how you declare those dependencies with setuptools: http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies (see the install_requires value in the third example)
So pip is actually quite picky about installing libraries with extra requirements
pip install -e ".[extra,requirements]" # works with file paths
pip install "package[extra,requirements]" # works when downloading packages
pip install ".[extra,requirments]" # DOES NOT WORK
I think this is down to how the RequirementsSpec parser works, and pip does some extra magic with the -e flag. Anyhow after much head banging, here's a mildly ugly workaround
pip install "file:///path/to/your/python_code#egg=SomeName[extra,requirements]"
The egg=SomeName part is basically ignored, but pip correctly picks up the extra requirements
Caveats
Tested with pip 1.5.6 so make sure you're using a current version of pip.
As far as I can tell, the file:/// syntax is undocumented in pip, so I'm not sure if it'll change in the future. It looks a bit like the VCS Support syntax but I was a bit surprised it worked.
You could also get around this by running your own pypi server, but that's a bit out of scope.
I've just started working with setuptools and virtualenv. My package requires the latest python-gearman that is only available from GitHub. The python-gearman version that's on PyPI is an old one. The Github source is setuptools-compatible, i.e. has setup.py, etc. Is there a way to make setuptools download and install the new version instead of looking for it on PyPI and installing the old one?
FYI, the new python-gearman is http://github.com/mtai/python-gearman
The key is to tell easy_install where the package can be downloaded. In this particular case, it can be found at the url http://github.com/mtai/python-gearman/tarball/master. However, that link by itself won't work, because easy_install can't tell just by looking at the URL what it's going to get.
By changing it to http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta instead, easy_install will be able to identify the package name and its version.
The final step is to add the URL to your package's dependency_links, e.g.:
setup(
...
dependency_links = ['http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta']
)
Now, when YOUR package is being installed, easy_install will discover that there is a "gearman 2.0.0beta" available for download from that URL, and happily pick it over the one on PyPI, if you specify "gearman>=2.0.0beta" in your dependencies..
(Normally, the way this sort of thing is done is to include a link on one's PyPI page to the downloadable source; in this case, if the author of the gearman package had included a link like the above, you'd be already set. Typically, people mark the development version with 'myproject-dev' and then people use a requirement of 'myproject>=somever,==dev', so that if there isn't a package of somever or higher, easy_install will try to check out or download the release.)
You'll need to specify --process-dependency-links when using pip. Note that dependency links processing has been deprecated and will be removed in a future release.
You can use the pip install protocol+location[#tag][#egg=Dependency] format to install directly from source using pip.
Git
pip install git+https://github.com/username/repo.git
pip install git+https://github.com/username/repo.git#MyTag
pip install git+https://github.com/username/repo.git#MyTag#egg=ProjectName
Mercurial
pip install hg+https://hg.myproject.org/MyProject/
SVN
pip install svn+svn://svn.myproject.org/svn/MyProject
Bzr
pip install bzr+http://bzr.myproject.org/MyProject/trunk
The following protocols are supported: [+git, +svn, +hg, +bzr]
Versions
#tag lets you specify a specific version/tag to check out.
#egg=name lets you specify what the project is as a dependency for others.
The order must always be #tag#egg=name.
Private Repositories
You can also install from private repositories by changing the protocol to SSH (ssh://) and adding an appropriate user (git#):
git+ssh://git#github.com/username/my_private_repo
You can also install from private repositories with a username / password.
git+https://<username>:<password>#github.com/<user>/<repo>.git
Github provides the ability to create personal OAuth tokens which can be cycled
git+https://<oauth token>:x-oauth-basic#github.com/<user>/<repo>.git
requirements.txt
requirements.txt is used to specify project dependencies:
requirements.txt
package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git
These are not installed automatically with the package and must be installed with the command pip -r requirements.txt.
Including requirements files
Requirements files can include other requirements files:
requirements-docs.txt
sphinx
-r requirements-dev.txt
requirements-dev.txt
some-dev-tool
-r requirements.txt
requirements.txt
package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git
setup.py
Requirements files can install dependencies specified in setup.py with the following command:
-e .
setup.py can also install from repositories using the same syntax as above, but using the dependency_links value as mentioned in this answer.
References:
https://pip.pypa.io/en/latest/user_guide.html#installing-packages
https://pip.pypa.io/en/latest/reference/pip_install.html
As I just had to do the same thing, I found another way to do this as pip's --process-dependency-links are scheduled to be removed in pip 19.0 according to this comment.
pip 18.1 includes the following feature
Allow PEP 508 URL requirements to be used as dependencies.
From the description of PEP 508, the syntax for such URL dependencies looks like:
A minimal URL based lookup:
pip # https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
So in your setup.py it would look like
setup(
...
install_requires = [
...
'python-gearman # https://github.com/mtai/python-gearman/archive/master.zip'
...
]
)
Notice, the link is an archive file and could also be a specific release or branch of a repository as described in this answer. Also, see that answer for working with other repository hosts.
To the best of my knowledge, the easiest way to update the dependency is by using pip install -I . when installing your package from its directory.
Vanilla setuptools does not support downloading directly from a git repository but you can use one of the Download Source links from that page, like:
easy_install http://github.com/mtai/python-gearman/tarball/master