PEP508 allows specifying a URL for a dependency, in particular a VCS. This is most useful for private packages that are not on pypi. If I have a package whose setup.py looks like:
from setuptools import setup
setup(name='foo',
install_requires=['bar # git+ssh://git#github.com/me/bar#1.2.3']
)
Then when I say pip install foo, it will download and install bar from the github repo. But if I later want to install a new version of foo, (pip install --upgrade foo), which has an updated bar dependency (e.g. tag 2.3.4), pip says that the dependency is already satisfied.
Is there a way to encode version information or something that will force pip to recognize that the dependency is NOT being met? I know I can give pip the --upgrade-strategy eager option, but that would affect ALL dependencies recursively and is too heavy-handed.
This related question PEP508: why either version requirement or URL but not both? asks about not being able to specify a version, but doesn't answer why pip doesn't get the URL when asked to upgrade.
Related
Python3 Pip error + Poetry Packaging
I am working in a python library that I am trying to publish to TestPypi. So far, there have been no issues with publishing my Poetry builds.
For context, as a beginner, I come from these websites :
https://python-poetry.org/docs/
https://packaging.python.org/en/latest/tutorials/packaging-projects/
The only issue that has arose is that dependencies listed in my pyproject.toml are not accounted for when installing the package with pip install.
I have attempted at updating setuptools and pip but I have done so to no avail.
My goal is to have clean dependency installation without the versioning errors.
This is the main solution I have tried.
pyproject.toml
I hid my real names.
[tool.poetry]
name = "package-name"
version = "0.1.0"
description = "<desc>"
authors = ["<myname> <myemail>"]
license = "MIT"
[tool.poetry.dependencies]
python = "^3.10"
beautifulsoup4 = {version = "4.11.1", allow-prereleases = true}
recurring-ical-events = {version = "1.0.2b0", allow-prereleases = true}
requests = {version = "2.28.0", allow-prereleases = true}
rich = {version = "12.4.4", allow-prereleases = true}
[tool.poetry.dev-dependencies]
black = {version = "22.3.0", allow-prereleases = true}
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
As the installer iterates through a dependency, it will return this error depending on whichever one is ordered first. (Throughout my monkey-patch-like attempts at fixing this, I was able to change the order of installation by modifying the strictness of each dependency version)
the error pip returns
ERROR: Could not find a version that satisfies the requirement requests==2.28.1 (from homeworkpy) (from versions: 2.5.4.1)
ERROR: No matching distribution found for requests==2.28.1
I have tried changing the strictness of the versions. (I removed the ^)
Switching to Poetry as a manager was also an attempt. My previous attempts were manual.
I have verified that the builds are corresponding to the correct builds previously published.
For extra info: I am building on a Github Codespace in which I run on 18.04.1-Ubuntu
Would anyone have any knowledge to spare of an issue like this? I am quite new to packaging and building, and I have had some success in most parts except for dependencies.
Main Error
TLDR; Pip tries to resolve dependencies with TestPypi, but they are in another index (Pypi). Workarounds at end of answer.
The fact that I am publishing to TestPypi is the reason this has happened. I will explain why what I did made this error appear, and then I will show how you, from the future, may solve this.
Difference between Pypi and TestPypi
Pypi is the Python Package Index. It's a giant index of Python packages one may install from with pip install.
TestPypi is the Python Package Index designated for testing and publishing without touching the real Package Index. It can be useful in times when learning how to publish a package. The main difference is that it is a completely separate repository. Therefore, what's on TestPypi may not be exactly what's on Pypi.
My research was limited, so if I confused anyone, the main difference is that they are two different Package Indexes. One was made for testing purposes.
I published my package to TestPypi and set my pip install to install from that repository. Not Pypi, but TestPypi.
Why dependency resolution failed
When I defined my project's dependencies, I defined them based off of their Pypi presences. Most dependencies are present in Pypi. Not TestPypi. This meant that when I asked for my package from TestPypi, pip only looked at TestPypi, and the pip installer workflow fell out to a pattern like this:
0.5. Set fetching repository to TestPypi and Not Pypi.
Pull package from TestPypi
Install and examine dependencies
Find first dependency (e.g. Beautifulsoup4)
Pull dependency from TestPypi
Successfully install Beautifulsoup4
-. This is because beautifulsoup4 is actually present in the TestPypi.
Move on to another dependency (e.g. rich)
Fail to pull from TestPypi
-. Rich is not present in TestPypi.
Return dependency not found.
Why some dependencies oddly worked
As you see in workflow step 5., the beautifulsoup4 package was found on the TestPypi. (Someone had put it up there).
image to TestPypi page with beautifulsoup4
However, as you see in step 7., Rich is not found on the TestPypi index. This issue occurs because I set my repoistiroy to install from TestPypi because my that is where my package was held. This caused pip to use TestPypi. for every single dependency as well.
How I got around it.
I got around it by using TestPypi to verify accurate build artifact publishing, and then I jumped to Normal Pypi to test installation and dependency installation.
Workarounds
Install from TestPypi
python3 -m pip install -i https://test.pypi.org/simple/ <package name>
Install from Pypi (by default)
python3 -m pip install <package name>
Install package from TestPypi but dependencies from Pypi
The Python Docs explains this very well.
If you want to allow pip to also download packages from PyPI, you can specify --extra-index-url to point to PyPI. This is useful when the package you’re testing has dependencies:
python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ your-package
Related to this question: What do square brackets mean in pip install?
I have a setup.py file that needs to install azure[common] package. However, if I try:
setup(
...
install_requires=['azure[common]'],
...
)
This results in an error:
pkg_resources.UnknownExtra: azure 4.0.0 has no such extra feature 'common'
But, if I do:
pip install 'azure[common]', then it works.
There were a lot of bugs and unexpected behavior involved in the experiment above, so the question doens't really make sense anymore.
There's a bug in pip which causes random stuff to be installed if "extra" package isn't found. So, pip install 'azure[common]' shouldn't have worked at all. It's an error that led me to believe there was such a package.
There's an inconsistency between how setuptools and pip install packages from wheels. setuptools installs (or seems to) only install one package from a wheel, while pip will install everything, and if there are more than one package, then it will install more. So, pip was installing azure.common by mistake, but there is no way to intentionally install just that package. At the minimum, you will also get azure.profiles plus a fake package azure_common, which doesn't really contain anything.
Given all this new info, I reformulated the question here: How to make setuptools install a wheel containing multiple packages?
Azure does not provide the common extra dependency. pip install azure[common] shows the warning about it.
The installation information page of PyCryptodome says the following under the "Windows (pre-compiled)" section:
Install PyCryptodome as a wheel:
pip install pycryptodomex
To make sure everything works fine, run the test suite:
python -m Cryptodome.SelfTest
There are several problems with this though:
Contrary to what these instructions say, this will not install PyCryptoDome as a wheel, but it will rather download it and try to build it, resulting in an error if you don't have the correct build environment installed for the C components included in this package (and the entire mess related to this is the biggest benefit of using a wheel instead to begin with).
Even if I instead download the correct wheel file from PyCryptoDome's PyPi page, I must (as far as I know?) instead use a command-line as follows to install it:
pip install c:\some\path\name-of-wheel-file.whl
This in turn makes it install under the default "Crypto" package instead of the "Cryptodome" package explicitly mentioned in the instructions (and therefore colliding in a breaking fashion with any pre-existing installations of the PyCrypto package).
So, my question is:
Is there any way to install a wheel file under a different package name than the default one?
PyCryptodome does not seem to provide any specific wheel files for installing under this alternative package name, so if this is impossible, I have a big problem (because I already have PyCrypto installed). :-(
PS.
Some more context regarding the need for the alternative package name can be provided by the following quote from the same installation page that is linked above:
PyCryptodome can be used as:
1.
a drop-in replacement for the old PyCrypto library. You install it with:
pip install pycryptodome
In this case, all modules are installed under the Crypto package. You can test everything is right with:
python -m Crypto.SelfTest
One must avoid having both PyCrypto and PyCryptodome installed at the same time, as they will interfere with each other.
This option is therefore recommended only when you are sure that the whole application is deployed in a virtualenv.
2.
a library independent of the old PyCrypto. You install it with:
pip install pycryptodomex
You can test everything is right with:
python -m Cryptodome.SelfTest
In this case, all modules are installed under the Cryptodome package. PyCrypto and PyCryptodome can coexist.
So, again, all I want is to install it as described under alternative 2 in this quote, from a wheel file, but the problem is that the provided wheel files seem to only default to the package name described under alternative 1 in this quote (i.e. "Crypto").
As far as I know this is not possible. The only way to achieve this by recompiling the wheel yourself after you modified its name in the setup.py.
I am trying to use pip to add the package pyhdf in python3.
I'm working in a virtualenv and have the prereq packages there:
% pip list
numpy (1.9.2)
pip (7.0.3)
requests (2.7.0)
setuptools (17.0)
wheel (0.24.0)
If I ask pip to get the latest/default version of pyhdf, I believe it searches this index page
https://pypi.python.org/simple/pyhdf/
This appears to trip up pip, as it tries to FTP the egg for v.0.7.x (not the latest) from a server that is not presently responding:
ftp://nordet.qc.dfo-mpo.gc.ca/pub/soft/pyhdf/pyhdf-0.7-1.tar.gz
I've been through the whole mess that the 'requests' package doesn't accept FTP URLs like this one, and that pip now strongly discourages getting external/unverified packages even though they are listed at PyPI. I settled on this workaround, getting the latest build directly from the authors' site:
wget http://hdfeos.org/software/pyhdf/pyhdf-0.9.0.tar.gz
To build the python package from a downloaded egg, you need to get the source headers for libhdf before running setup.py (via pip or manually.) The following I did outside the venv, though they might work within one too
sudo apt-get install libhdf4-0
sudo apt-get install libhdf4-dev libhdf4-doc
Finally, back in the venv, the actual pip install syntax that worked for me to install pyhdf from the tar.gz that I had downloaded:
pip install --global-option=build_ext --global-option="-I/usr/include/hdf" --no-index --find-links=file:pyhdf-0.9.0.tar.gz pyhdf
after which pip list yields:
numpy (1.9.2)
pip (7.0.3)
pyhdf (0.9.0)
requests (2.7.0)
setuptools (17.0)
wheel (0.24.0)
Yay!
Since I'm posting my workaround that got me pyhdf okay, to make this an actual question I will ask:
Is there an official proper way that users can do to contact either PyPI admins or the authors of the pyhdf package to report that the "best match" link at the project page is:
no longer the latest release, and
points to an FTP server that is apparently offline (at least currently)
making pip install pyhdf basically infeasible without much extra manual intervention (which if permanent, ought to be documented by the maintainers)
I've just started working with setuptools and virtualenv. My package requires the latest python-gearman that is only available from GitHub. The python-gearman version that's on PyPI is an old one. The Github source is setuptools-compatible, i.e. has setup.py, etc. Is there a way to make setuptools download and install the new version instead of looking for it on PyPI and installing the old one?
FYI, the new python-gearman is http://github.com/mtai/python-gearman
The key is to tell easy_install where the package can be downloaded. In this particular case, it can be found at the url http://github.com/mtai/python-gearman/tarball/master. However, that link by itself won't work, because easy_install can't tell just by looking at the URL what it's going to get.
By changing it to http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta instead, easy_install will be able to identify the package name and its version.
The final step is to add the URL to your package's dependency_links, e.g.:
setup(
...
dependency_links = ['http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta']
)
Now, when YOUR package is being installed, easy_install will discover that there is a "gearman 2.0.0beta" available for download from that URL, and happily pick it over the one on PyPI, if you specify "gearman>=2.0.0beta" in your dependencies..
(Normally, the way this sort of thing is done is to include a link on one's PyPI page to the downloadable source; in this case, if the author of the gearman package had included a link like the above, you'd be already set. Typically, people mark the development version with 'myproject-dev' and then people use a requirement of 'myproject>=somever,==dev', so that if there isn't a package of somever or higher, easy_install will try to check out or download the release.)
You'll need to specify --process-dependency-links when using pip. Note that dependency links processing has been deprecated and will be removed in a future release.
You can use the pip install protocol+location[#tag][#egg=Dependency] format to install directly from source using pip.
Git
pip install git+https://github.com/username/repo.git
pip install git+https://github.com/username/repo.git#MyTag
pip install git+https://github.com/username/repo.git#MyTag#egg=ProjectName
Mercurial
pip install hg+https://hg.myproject.org/MyProject/
SVN
pip install svn+svn://svn.myproject.org/svn/MyProject
Bzr
pip install bzr+http://bzr.myproject.org/MyProject/trunk
The following protocols are supported: [+git, +svn, +hg, +bzr]
Versions
#tag lets you specify a specific version/tag to check out.
#egg=name lets you specify what the project is as a dependency for others.
The order must always be #tag#egg=name.
Private Repositories
You can also install from private repositories by changing the protocol to SSH (ssh://) and adding an appropriate user (git#):
git+ssh://git#github.com/username/my_private_repo
You can also install from private repositories with a username / password.
git+https://<username>:<password>#github.com/<user>/<repo>.git
Github provides the ability to create personal OAuth tokens which can be cycled
git+https://<oauth token>:x-oauth-basic#github.com/<user>/<repo>.git
requirements.txt
requirements.txt is used to specify project dependencies:
requirements.txt
package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git
These are not installed automatically with the package and must be installed with the command pip -r requirements.txt.
Including requirements files
Requirements files can include other requirements files:
requirements-docs.txt
sphinx
-r requirements-dev.txt
requirements-dev.txt
some-dev-tool
-r requirements.txt
requirements.txt
package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git
setup.py
Requirements files can install dependencies specified in setup.py with the following command:
-e .
setup.py can also install from repositories using the same syntax as above, but using the dependency_links value as mentioned in this answer.
References:
https://pip.pypa.io/en/latest/user_guide.html#installing-packages
https://pip.pypa.io/en/latest/reference/pip_install.html
As I just had to do the same thing, I found another way to do this as pip's --process-dependency-links are scheduled to be removed in pip 19.0 according to this comment.
pip 18.1 includes the following feature
Allow PEP 508 URL requirements to be used as dependencies.
From the description of PEP 508, the syntax for such URL dependencies looks like:
A minimal URL based lookup:
pip # https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
So in your setup.py it would look like
setup(
...
install_requires = [
...
'python-gearman # https://github.com/mtai/python-gearman/archive/master.zip'
...
]
)
Notice, the link is an archive file and could also be a specific release or branch of a repository as described in this answer. Also, see that answer for working with other repository hosts.
To the best of my knowledge, the easiest way to update the dependency is by using pip install -I . when installing your package from its directory.
Vanilla setuptools does not support downloading directly from a git repository but you can use one of the Download Source links from that page, like:
easy_install http://github.com/mtai/python-gearman/tarball/master