This question is a consequence of the principal
solution of requirement specifation:
python django pip update problems: how to get compatible versions?
I try to synchronize the python requirements between a server
and the local development systems. Both are Ubuntu 16.04.,
so this should be a problem.
The server gives me a requirement file (after pip-installing django as described in the other quewtion).
But pip-installing the requirements on the local machine gives some errors,
like this:
Could not find a version that satisfies the requirement python-
apt==1.1.0b1 (from -r requirements.txt (line 3)) (from versions: 0.0.0,
0.7.8) No matching distribution found for python-apt==1.1.0b1 (from -r
requirements.txt
Is there a way to ensure, that requirements can be met on similar systems
without getting such errors caused by version incompatiblities
(or of cause alterantively solve the incompatibility problems)?
Do I need virtual environments to solve this?
Or even pipenv?
Or is there a simple straightforward way to have two systems with
compatible python and package environments?
Problem
This particular module (python-apt) is only available on PyPi with version 0.7.8. However, this release appears to have been a mistake!
One of the developers & Debian package maintainers for python-apt has stated the following:
Aargh, not this whole PyPI thing again. Nobody ever officially
uploaded python-apt there. It is tightly coupled with APT, and
not supposed to be distributed in any fashion other than via
Debian packages.
There is no, and has never been any support for PyPi. And I can
say that I have absolutely no interest in duplicating work there.
Source: Debian "Deity" Mailing List 2016-11-22 msg#00094
You can install python-apt from apt, we do not provide python-apt on pip. I recently got control over the pypi entry and need to do something with it. I'm not keen on providing python-apt outside of the distro, though (python-apt and apt versions x.y need to match), so I'd rather just get rid of it, so people stop with questions about outdated versions.
Source: python-apt#1883451
So at least for this dependency, it appears that we're out of luck when resolving dependencies python-natively via pip + PyPi. Luckily, the upstream project is hosted on the salsa.debian.org GitLab instance, and pip supports git+ SCM urls now, among other options.
Solutions:
Generally, there are many solutions for resolving such a dependency. The concerns you are trying to address are:
Where is this package being provided from?
System OS package manager
Arbitrary Upstream Release URL
Development SCM Repo
Forked Git Repo with bugfixes
etc...
What Version will you be installing?
Compatibility concerns:
Must be >= 2.0.0
Forked repo / feature branch with some bugfix
API Compatibility: Not greater than 2.x / Semantic Versioning
Development:
Just use latest & greatest ( bleeding edge )
Use a specific version for my platform or system (e.g.: system OS package, my local forked version in a development directory)
QA / Testing:
Test against a specific version
Test against latest (e.g.: nightly builds)
Test against a version provided by a system OS release
How should the dependency be resolved?
"abstract" dependency on pip-module-name + version constraint
Allows for some flexibility later on with where to gather (URL / PyPi / Artifactory) and what version is allowed to satisfy the constraint.
User can always override these if needed by specifying arguments to pip, installing using a specific virtualenv, etc...
"concrete" dependency on a specific URL + package-name + version
Extreme case: Locking to a specific URL + version + sha1 / sha256 / sha** with checksum verification to ensure exact location & file integrity.
Less flexible, but most assuredly locked to an exact and precise version & source.
Are you developing an "application" or a "library" / Python module?
Do the dependencies need to be installed via pip using setup.py install_requires = [...] style resolution? (library)
Do the dependencies need to be installed by the application installer via pip install -r requirements.txt? (application)
How will your project be released?
Who will be installing the package, and how will their system be allowed to resolve the dependency?
Is this going to be released on PyPi as a library, or elsewhere as an application? (some rules of thumb follow)
Generally speaking:
A library tends to want to have wide open ended version specifiers
An application wants very specific dependencies to ensure stability (lots of dependencies means lots of generally untested version permutations!)
Use setup.py to specify the library dependencies for PyPi
Use requirements.txt to specify an application's dependencies
Is this going to be packaged as a native OS package? (e.g.: .deb, .rpm, .apk, etc...)
The native package manager has dependency resolution too... perhaps use this to ensure native compatibility!
What other OS platforms will your package support & how will those platforms resolve the dependency?
So, most generally we can see that there are various concerns all relating to the desired specificity of where and how these dependencies get resolved and installed. There is no "one size fits all" solution here... There are Pros and Cons, and only many different solutions that fall somewhere on the spectrum of:
more specific <---------------------> less specific
reliable compatibility reliable installability
less testing permutations more (possibly un-vetted) testing permutations
limited platform support more platform support (when more permutations are well tested)
dependable known configurations less dependable known configurations
less platform tolerant more tolerant and agnostic of platforms
more OS native less OS native
Latest & Greatest from GitLab Upstream Repo
One solution to the python-apt package issue is to use this git+ URL feature in requirements.txt. This is great for development against the upstream version of python-apt from GitLab. To further isolate installation from the system OS provided version of python-apt, a virtualenv or pip install --user may be desired. For example:
requirements.txt:
--index-url https://pypi.python.org/simple/
-e git+https://salsa.debian.org/apt-team/python-apt.git#egg=python-apt
-e .
This can be used by an example project with setup.py containing:
[...SNIP...] # Boilerplate stuff here
setup(
#[...SNIP...] # Other setup() args here
platforms=['linux'],
# Reference:
# - https://github.com/pypa/interoperability-peps/pull/30/files#r184839487
# sudo apt install python3-apt apt-rdepends apt
# os_requires=[
# ['python3-apt', type='packagename', target='run', os='ubuntu'],
# ['apt-rdepends', type='packagename', target='run', os='ubuntu'],
# ['apt', type='packagename', target='run', os='ubuntu']
# ['libapt-pkg-dev', type='packagename', target='build', os='ubuntu']
# ]
# Build-deps for apt-python via git SCM: sudo apt install libapt-pkg-dev
python_requires='>=3.5',
install_requires=[
'python-apt (>= 2.0)',
# rest of your dependencies here
#[... SNIP ...]
],
package_dir={'': 'lib'},
scripts=_glob('bin/*'),
#[...SNIP...]
)
Note: os_requires isn't actually supported yet, but is proposed for a PEP. This might help in the future for external dependencies on packages. It would help in situations like this where a python module is not distributed via PyPi / pip, but instead is only provided via apt / .deb packages on the OS.
Set up your virtualenv or use pip3 install --user if you wish, then continue.
Using the git+ requirements feature results in the following when running pip3 install -r requirements.txt:
$ pip3 install -r requirements.txt
Looking in indexes: https://pypi.python.org/simple/
Obtaining file:///../example-project (from -r requirements.txt (line 4))
Obtaining python-apt from git+https://salsa.debian.org/apt-team/python-apt.git#egg=python-apt (from -r requirements.txt (line 3))
Updating ./example-project-venv/src/python-apt clone
Running command git fetch -q --tags
Running command git reset --hard -q c97d4159beae2f9cd42d55d3dff9c37f5c69aa44
ERROR: example-project 0.0.1 has requirement python-apt>=2.0, but you'll have python-apt 0.0.0 which is incompatible.
Installing collected packages: python-apt, example-project
Running setup.py develop for python-apt
Running setup.py develop for example-project
Successfully installed example-project python-apt
Note: You'll probably want to install runtime & build / setup.py dependencies for python-apt first:
# Runtime deps (e.g.: Ubuntu 20.04 needs python3-apt, <20.04 needs python-apt):
sudo apt install python3-apt apt
# python-apt pip install deps (also for setup.py / development)
sudo apt install libapt-pkg-dev
Alternative: dependency_links (Note: may be deprecated)
If you are developing a library type module, and also wish to use GitLab as source for python-apt you may want to consider using dependency_links in setup.py to provide the git+ or http(s) tarball release URL rather than in requirements.txt. This is helpful to distinguish an "application" python project from a "library" Python module project. It all depends on what your project's install process looks like. (e.g.: do you want to pip install -r requirements.txt, or just pip install example-module, or python[3] setup.py {sdist,bdist,bdist_rpm, etc...}. It could also be helpful to specify a custom URL for a forked version of python-apt. However, this method is likely to be deprecated sometime soon (if not partially already in new versions of pip). You may want to consider other options for future-proofing your dependency specifications such as PEP 508 or pip install --find-links ... instead.
Also, the "application" vs "library" distinction can be important here, as well as the concepts of "abstract" vs. "concrete" dependencies. A quick summary might be:
This split between abstract and concrete is an important one. It was
what allows the PyPI mirroring infrastructure to work. It is what
allows a company to host their own private package index. It is even
what enables you to fork a library to fix a bug or add a feature and
use your own fork. Because an abstract dependency is a name and an
optional version specifier you can install it from PyPI or from
Crate.io, or from your own filesystem. You can fork a library, change
the code, and as long as it has the right name and version specifier
that library will happily go on using it.
Setuptools has a feature similar to the Go example. It’s called
dependency links and it looks like this:
setup(
# ...
dependency_links = [
"http://packages.example.com/snapshots/",
"http://example2.com/p/bar-1.0.tar.gz",
], ) ```
This “feature” of setuptools removes the abstractness of its
dependencies and hardcodes an exact url from which you can fetch the
dependency from. Now very similarly to Go if we want to modify
packages, or simply fetch them from a different server we’ll need to
go in and edit each package in the dependency chain in order to update
the dependency_links.
Source: caremad.io Blog Post: setup.py vs requirements.txt
For this python-apt example, we might use something like this to lock a "concrete dependency" on v2.0.0:
setup(
# [...SNIP...]
dependency_links = [
"https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz#egg=python-apt"
],
# [...SNIP...]
) `
Note: This "mis-feature" was briefly removed, and then brought back given some usefulness in specifying private package dependency URLs. However, currently pip --process-dependency-links flag has been deprecated, so its' usefulness is probably limited to older versions of Python 2 + pip.
Alternative: PEP 508 Syntax
Newer versions of pip now have URL support for PEP 508 syntax. This is probably the most future-proof method for specifying concrete and abstract dependencies with a complex grammar (See PEP 508 for details). Packages can now be specified in many ways, including custom URLs.
For example, to lock python-apt to v2.0.0 with optional sha256 checksum:
setup(
# [...SNIP...]
install_requires=[
'python-apt#https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz#sha256=1ddbd3eb7cbc1ded7e0e8a2dd75219f0c59c7e062c6e6bfd5c8ff6f656c59a4e',
# [...SNIP...]
],
# [...SNIP...]
)
requirements.txt:
--index-url https://pypi.python.org/simple/
-e .
Then, pip install -r requirements.txt still works without any extra flags:
$ ./example-project-venv/bin/python3 ./example-project-venv/bin/pip3 install -r requirements.txt
Looking in indexes: https://pypi.python.org/simple/
Obtaining file://./src/pub/example-project (from -r requirements.txt (line 4))
Requirement already satisfied: graph-tools>=1.5 in ./example-project-venv/lib/python3.8/site-packages (from example-project==0.0.1->-r requirements.txt (line 4)) (1.5)
Collecting python-apt# https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz#sha256=1ddbd3eb7cbc1ded7e0e8a2dd75219f0c59c7e062c6e6bfd5c8ff6f656c59a4e
Using cached https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz (458 kB)
Building wheels for collected packages: python-apt
Building wheel for python-apt (setup.py) ... done
Created wheel for python-apt: filename=python_apt-0.0.0-cp38-cp38-linux_x86_64.whl size=2040980 sha256=79eeb0d1bb9e3c9785acb68f164a3f72a5777539137d180e9ded7558d2547a49
Stored in directory: ~/.cache/pip/wheels/c4/09/b5/36fc8c9a1ebe8786620db922f1495da200dce187ee7c618993
Successfully built python-apt
Installing collected packages: python-apt, example-project
Attempting uninstall: example-project
Found existing installation: example-project 0.0.1
Uninstalling example-project-0.0.1:
Successfully uninstalled example-project-0.0.1
Running setup.py develop for example-project
Successfully installed example-project python-apt-0.0.0
Alternative: pip install --find-links ...
Another alternative method for installing a "concrete dependency" locked to a specific version is to pass --find-links to pip install with a released tarball file. This method might be helpful for explicitly installing a specific version given a release URL. For example, using python-apt v2.0.0:
$ ./example-project-venv/bin/python3 ./example-project-venv/bin/pip3 install --find-links 'https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz' -r requirements.txt
Looking in indexes: https://pypi.python.org/simple/
Looking in links: https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz
Obtaining file://./example-project (from -r requirements.txt (line 4))
Requirement already satisfied: graph-tools>=1.5 in ./example-project-venv/lib/python3.8/site-packages (from example-project==0.0.1->-r requirements.txt (line 4)) (1.5)
Collecting python-apt>=2.0
Downloading https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz (458 kB)
|████████████████████████████████| 458 kB 614 kB/s
WARNING: Requested python-apt>=2.0 from https://salsa.debian.org/apt-team/python-apt/-/archive/2.0.0/python-apt-2.0.0.tar.gz (from example-project==0.0.1->-r requirements.txt (line 4)), but installing version 0.0.0
Building wheels for collected packages: python-apt
Building wheel for python-apt (setup.py) ... done
Created wheel for python-apt: filename=python_apt-0.0.0-cp38-cp38-linux_x86_64.whl size=2040783 sha256=d0a8f88c04f202e948b9855837140517d9b2bd3cef72e626221614552a476780
Stored in directory: ~/.cache/pip/wheels/8a/07/e9/b3c3328bac08c030a5b1e754e01e327b62fd26f9baedf07c15
Successfully built python-apt
ERROR: example-project 0.0.1 has requirement python-apt>=2.0, but you'll have python-apt 0.0.0 which is incompatible.
Installing collected packages: python-apt, example-project
Attempting uninstall: python-apt
Found existing installation: python-apt 0.0.0
Uninstalling python-apt-0.0.0:
Successfully uninstalled python-apt-0.0.0
Attempting uninstall: example-project
Found existing installation: example-project 0.0.1
Uninstalling example-project-0.0.1:
Successfully uninstalled example-project-0.0.1
Running setup.py develop for example-project
Successfully installed example-project python-apt-0.0.0
Base System Debian Package
On Debian & Ubuntu, there are two .deb packages you'll see on the various distribution versions: python3-apt and python-apt (for Python2).
These packages are managed by the APT package manager, and thus are installed in the system location: /usr/lib/python3/dist-packages or /usr/lib/python2.7/dist-packages for Python3 & Python2.7 respectively.
This dist-packages path, and other Python packaging conventions are explained well in this post:
The system has installed Python packages in the global dist-packages directory of each Python version and created symbolic links:
/usr/lib/python2.7/dist-packages/numpy
/usr/lib/python3/dist-packages/numpy
ls -ls /usr/include/numpy
#-> ../lib/python2.7/dist-packages/numpy/core/include/numpy
ls -l /usr/include/python2.7/numpy
#->../../lib/python2.7/dist-packages/numpy/core/include/numpy
ls -l /usr/include/python3.5/numpy
#-> ../../lib/python3/dist-packages/numpy/core/include/numpy
Note the good use of dist-packages instead of site-packages which should be reserved for the system Python.
So, if you're looking to use the base-OS system level version of python3-apt then you'd want to make sure that this path is on your sys.path or PYTHONPATH so import apt will work. Whereas, if you wanted to use the site-packages location, or a virtualenv location... those would have to exist on sys.path / PYTHONPATH instead.
Unfortunately, as mentioned before, there is not yet an official way to state a dependency on an OS package providing a certain version of a python module. However, as long as you're managing your python runtime environment's import path(s), you should be able to use the correct version from the OS package that lives in the dist-packages directory.
Indeed, python-apt's latest version is 0.7.8 https://pypi.org/project/python-apt/
If you're 100% sure it's the same package, try to change it in your requirements.txt file.
On the other hand, you could try seeing where is that python-apt on your local.
import apt
print(apt.__file__) # or print(apt.__path__)
And then, go there and check manually what is that python-apt package exactly.
Related
Python3 Pip error + Poetry Packaging
I am working in a python library that I am trying to publish to TestPypi. So far, there have been no issues with publishing my Poetry builds.
For context, as a beginner, I come from these websites :
https://python-poetry.org/docs/
https://packaging.python.org/en/latest/tutorials/packaging-projects/
The only issue that has arose is that dependencies listed in my pyproject.toml are not accounted for when installing the package with pip install.
I have attempted at updating setuptools and pip but I have done so to no avail.
My goal is to have clean dependency installation without the versioning errors.
This is the main solution I have tried.
pyproject.toml
I hid my real names.
[tool.poetry]
name = "package-name"
version = "0.1.0"
description = "<desc>"
authors = ["<myname> <myemail>"]
license = "MIT"
[tool.poetry.dependencies]
python = "^3.10"
beautifulsoup4 = {version = "4.11.1", allow-prereleases = true}
recurring-ical-events = {version = "1.0.2b0", allow-prereleases = true}
requests = {version = "2.28.0", allow-prereleases = true}
rich = {version = "12.4.4", allow-prereleases = true}
[tool.poetry.dev-dependencies]
black = {version = "22.3.0", allow-prereleases = true}
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
As the installer iterates through a dependency, it will return this error depending on whichever one is ordered first. (Throughout my monkey-patch-like attempts at fixing this, I was able to change the order of installation by modifying the strictness of each dependency version)
the error pip returns
ERROR: Could not find a version that satisfies the requirement requests==2.28.1 (from homeworkpy) (from versions: 2.5.4.1)
ERROR: No matching distribution found for requests==2.28.1
I have tried changing the strictness of the versions. (I removed the ^)
Switching to Poetry as a manager was also an attempt. My previous attempts were manual.
I have verified that the builds are corresponding to the correct builds previously published.
For extra info: I am building on a Github Codespace in which I run on 18.04.1-Ubuntu
Would anyone have any knowledge to spare of an issue like this? I am quite new to packaging and building, and I have had some success in most parts except for dependencies.
Main Error
TLDR; Pip tries to resolve dependencies with TestPypi, but they are in another index (Pypi). Workarounds at end of answer.
The fact that I am publishing to TestPypi is the reason this has happened. I will explain why what I did made this error appear, and then I will show how you, from the future, may solve this.
Difference between Pypi and TestPypi
Pypi is the Python Package Index. It's a giant index of Python packages one may install from with pip install.
TestPypi is the Python Package Index designated for testing and publishing without touching the real Package Index. It can be useful in times when learning how to publish a package. The main difference is that it is a completely separate repository. Therefore, what's on TestPypi may not be exactly what's on Pypi.
My research was limited, so if I confused anyone, the main difference is that they are two different Package Indexes. One was made for testing purposes.
I published my package to TestPypi and set my pip install to install from that repository. Not Pypi, but TestPypi.
Why dependency resolution failed
When I defined my project's dependencies, I defined them based off of their Pypi presences. Most dependencies are present in Pypi. Not TestPypi. This meant that when I asked for my package from TestPypi, pip only looked at TestPypi, and the pip installer workflow fell out to a pattern like this:
0.5. Set fetching repository to TestPypi and Not Pypi.
Pull package from TestPypi
Install and examine dependencies
Find first dependency (e.g. Beautifulsoup4)
Pull dependency from TestPypi
Successfully install Beautifulsoup4
-. This is because beautifulsoup4 is actually present in the TestPypi.
Move on to another dependency (e.g. rich)
Fail to pull from TestPypi
-. Rich is not present in TestPypi.
Return dependency not found.
Why some dependencies oddly worked
As you see in workflow step 5., the beautifulsoup4 package was found on the TestPypi. (Someone had put it up there).
image to TestPypi page with beautifulsoup4
However, as you see in step 7., Rich is not found on the TestPypi index. This issue occurs because I set my repoistiroy to install from TestPypi because my that is where my package was held. This caused pip to use TestPypi. for every single dependency as well.
How I got around it.
I got around it by using TestPypi to verify accurate build artifact publishing, and then I jumped to Normal Pypi to test installation and dependency installation.
Workarounds
Install from TestPypi
python3 -m pip install -i https://test.pypi.org/simple/ <package name>
Install from Pypi (by default)
python3 -m pip install <package name>
Install package from TestPypi but dependencies from Pypi
The Python Docs explains this very well.
If you want to allow pip to also download packages from PyPI, you can specify --extra-index-url to point to PyPI. This is useful when the package you’re testing has dependencies:
python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ your-package
I am developing two python packages, pkg_a and pkg_b. pkg_a is a requirement for pkg_b, and so the setup.py for pkg_b looks like this:
from setuptools import setup
inst_reqs = [
'pkg_a # git+ssh://git#bitbucket.org/vlad/pkg_a.git',
]
setup(
name="pkg_b",
version="0.0.0",
packages=['pkg_b'],
install_requires=inst_reqs,
)
Since I am developing both packages simultaneously, pkg_a is already installed in editable mode (pip install -e .).
When pip installing pkg_b, why is the existing installation of pkg_a removed? It looks like pip will systematically cone the specified repo, uninstall the existing pkg_a and reinstall it from the cloned repo:
Successfully built pkg_a
Installing collected packages: pkg_a, pkg_b
Attempting uninstall: pkg_a
Found existing installation: pkg_a 0.0.0
Uninstalling pkg_a-0.0.0:
Successfully uninstalled pkg_a-0.0.0
Running setup.py develop for pkg_b
Successfully installed pkg_a-0.0.0 pkg_b
I'm guessing this has to with versioning but I don't know how to fix this. Any tips?
That is how VCS dependencies are handled. You should specify a fixed reference (tag, commit id):
'pkg_a # git+ssh://git#bitbucket.org/vlad/pkg_a.git#da39a3ee5e6b4b0d3255bfef95601890afd80709'
See: https://pip.pypa.io/en/stable/reference/pip_install/#git
If you do not specific a fixed reference (non-moving tag, or commit ID), then pip has to clone every time since the content of the repository might have changed since the last installation.
(To be honest, even after this change, it might still be that pip will re-clone at each installation, I do not remember the exact behavior off the top of my head.)
See similar question: pip install upgrade fail to upgrade private dependency
This question already has answers here:
Pip install from pypi works, but from testpypi fails (cannot find requirements)
(2 answers)
Closed 2 years ago.
TL;DR Even though I've specified dependencies with install_requires in setup.py, the install through pip fails because some dependencies can't be found.
I've developed a package which I intend to distribute via PyPi. I've created a built distribution wheel and uploaded it to testPyPI to see if everything is working with the upload and if the package can be installed from a user perspective.
However, when I try to pip install the package inside a vanilla python 2.7 environment, the installation process fails while installing the dependencies.
My package depends on these packages (which I added to the setup.py file accordingly):
...
install_requires=['numpy','gdal','h5py','beautifulsoup4','requests','tables','progress'],
...
So when I run pip install, everything looks normal for a moment, until I receive this error:
Could not find a version that satisfies the requirement progress (from #NAME#) (from versions: )
No matching distribution found for progress (from #NAME#)
When I remove the progress dependency (I could live without it), same thing happens for pytables:
Could not find a version that satisfies the requirement tables (from #NAME#) (from versions: )
No matching distribution found for tables (from #NAME#)
If I run pip install tables and pip install progress manually beforehand, everything works as expected.
So how can I assure that if someone downloads my package, all missing dependencies are installed with it?
Related bonus question:
Can I include a wheel file in my package (maybe through MANIFEST.in) and install it as dependency if the module is not available? If so, how?
And I think I've found the answer to my question myself.
When installing a package from testPyPI, the dependencies are also installed from there. And it seems, that while there are many packages available, pytables and progress are apparently missing. This caused the installation to fail.
Naturally, manually installing with pip install gets the package from the "normal" PyPi, which of course works. This obviously added to my confusion.
Here's a look at the output from pip install when installing the package from the testPyPi:
Downloading https://test-files.pythonhosted.org/packages/4f/96/b3329750a04fcfc316f15f658daf6d81acc3ac61e3db390abe8954574c18/nump
y-1.9.3.tar.gz (4.0MB)
while installing the wheel directly, it looks slightly different:
Downloading https://files.pythonhosted.org/packages/2e/91/504e434d3b95d943caab926f33dee5691768fbb622bc290a0fa6df77e1d8/numpy-1.1
4.2-cp27-none-win32.whl (9.8MB)
Additionally, running
pip install --index-url https://test.pypi.org/simple/ tables
produces the same error as described in my question.
I'm building a small app that uses PyQt and tought it'd be nice to declare that dependency in setup.py.
However, according to this blog (first hit on google for pyqt setuptools) says it can't be done, and the last paragraph here doens't try to do it either.
Ideas? Perhaps I should switch to PySide which is on PyPi?
Update:
The obvious install_requires = [ 'pyqt >= 0.7' ] in setup.py gives me:
D:\3rd\BuildBotIcon\my-buildboticon\python>setup.py test
running test
install_dir .
Checking .pth file support in .
C:\Python26-32\pythonw.exe -E -c pass
Searching for pyqt>=4.7
Reading http://pypi.python.org/simple/pyqt/
Reading http://www.riverbankcomputing.com/software/pyqt/
Reading http://www.riverbankcomputing.com/software/pyqt/download
No local packages or download links found for pyqt>=4.7
error: Could not find suitable distribution for Requirement.parse('pyqt>=4.7')
Right, the PyQT packages are not using distutils / setup.py for it's installation, so they can't be installed with easy_install or pip. You need to download and install it manually.
That also means you should not put it in your requires metadata, as easy_install and pip then will try to install it and fail.
I don't know if PySide is any good, but is also has not setup.py, and also refuse to install with easy_install/pip, so not a good option. :)
Another option is to repackage PyQt with distutils, but that may be a lot of work.
While you can pip install pyqt5 thanks to the now available wheels (as suggested by #mfitzp), it cannot be required from setup.py via install_requires. The reason is that setuptools doesn't know how to install wheels which pip knows how to, and PyQT5 is only available as wheels on PyPI (there is no source distribution, i.e. no tar.gz file). See this email and that bug report for details.
While the accepted answer was originally correct Python Wheels now provide a means to install C extension packages such as PyQt5 without the need for compilation from source.
PyPi currently has .whl files for PyQt5 on Python3 for multiple platforms, including MacOS X, Linux (any), Win32 and Win64. For example, this is the output when pip-installing PyQt5 on Python3 on a Mac:
mfitzp#MacBook-Air ~ $ pip3 install pyqt5
Collecting pyqt5
Downloading PyQt5-5.6-cp35-cp35m-macosx_10_6_intel.whl (73.2MB)
100% |████████████████████████████████| 73.2MB 2.5kB/s
Collecting sip (from pyqt5)
Downloading sip-4.18-cp35-cp35m-macosx_10_6_intel.whl (46kB)
100% |████████████████████████████████| 49kB 1.8MB/s
Installing collected packages: sip, pyqt5
Successfully installed pyqt5-5.6 sip-4.18
If you are targeting Python3+PyQt5 then you should have no problem specifying PyQt5 as a normal dependency in setup.py.
Setuptools >= 38.2.0 now knows how to install wheels. The trivial answer, therefore, is to install a recent version of setuptools and require that your enlightened userbase does so as well. To enforce usage of setuptools >= 38.2.0 at installation time, see this relevant answer elsewhere.
Since setuptools 38.2.0 was released over a year ago, all prior answers to this question are horrifyingly obsolete, blatantly wrong, and less than useful.
I've just started working with setuptools and virtualenv. My package requires the latest python-gearman that is only available from GitHub. The python-gearman version that's on PyPI is an old one. The Github source is setuptools-compatible, i.e. has setup.py, etc. Is there a way to make setuptools download and install the new version instead of looking for it on PyPI and installing the old one?
FYI, the new python-gearman is http://github.com/mtai/python-gearman
The key is to tell easy_install where the package can be downloaded. In this particular case, it can be found at the url http://github.com/mtai/python-gearman/tarball/master. However, that link by itself won't work, because easy_install can't tell just by looking at the URL what it's going to get.
By changing it to http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta instead, easy_install will be able to identify the package name and its version.
The final step is to add the URL to your package's dependency_links, e.g.:
setup(
...
dependency_links = ['http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta']
)
Now, when YOUR package is being installed, easy_install will discover that there is a "gearman 2.0.0beta" available for download from that URL, and happily pick it over the one on PyPI, if you specify "gearman>=2.0.0beta" in your dependencies..
(Normally, the way this sort of thing is done is to include a link on one's PyPI page to the downloadable source; in this case, if the author of the gearman package had included a link like the above, you'd be already set. Typically, people mark the development version with 'myproject-dev' and then people use a requirement of 'myproject>=somever,==dev', so that if there isn't a package of somever or higher, easy_install will try to check out or download the release.)
You'll need to specify --process-dependency-links when using pip. Note that dependency links processing has been deprecated and will be removed in a future release.
You can use the pip install protocol+location[#tag][#egg=Dependency] format to install directly from source using pip.
Git
pip install git+https://github.com/username/repo.git
pip install git+https://github.com/username/repo.git#MyTag
pip install git+https://github.com/username/repo.git#MyTag#egg=ProjectName
Mercurial
pip install hg+https://hg.myproject.org/MyProject/
SVN
pip install svn+svn://svn.myproject.org/svn/MyProject
Bzr
pip install bzr+http://bzr.myproject.org/MyProject/trunk
The following protocols are supported: [+git, +svn, +hg, +bzr]
Versions
#tag lets you specify a specific version/tag to check out.
#egg=name lets you specify what the project is as a dependency for others.
The order must always be #tag#egg=name.
Private Repositories
You can also install from private repositories by changing the protocol to SSH (ssh://) and adding an appropriate user (git#):
git+ssh://git#github.com/username/my_private_repo
You can also install from private repositories with a username / password.
git+https://<username>:<password>#github.com/<user>/<repo>.git
Github provides the ability to create personal OAuth tokens which can be cycled
git+https://<oauth token>:x-oauth-basic#github.com/<user>/<repo>.git
requirements.txt
requirements.txt is used to specify project dependencies:
requirements.txt
package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git
These are not installed automatically with the package and must be installed with the command pip -r requirements.txt.
Including requirements files
Requirements files can include other requirements files:
requirements-docs.txt
sphinx
-r requirements-dev.txt
requirements-dev.txt
some-dev-tool
-r requirements.txt
requirements.txt
package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git
setup.py
Requirements files can install dependencies specified in setup.py with the following command:
-e .
setup.py can also install from repositories using the same syntax as above, but using the dependency_links value as mentioned in this answer.
References:
https://pip.pypa.io/en/latest/user_guide.html#installing-packages
https://pip.pypa.io/en/latest/reference/pip_install.html
As I just had to do the same thing, I found another way to do this as pip's --process-dependency-links are scheduled to be removed in pip 19.0 according to this comment.
pip 18.1 includes the following feature
Allow PEP 508 URL requirements to be used as dependencies.
From the description of PEP 508, the syntax for such URL dependencies looks like:
A minimal URL based lookup:
pip # https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
So in your setup.py it would look like
setup(
...
install_requires = [
...
'python-gearman # https://github.com/mtai/python-gearman/archive/master.zip'
...
]
)
Notice, the link is an archive file and could also be a specific release or branch of a repository as described in this answer. Also, see that answer for working with other repository hosts.
To the best of my knowledge, the easiest way to update the dependency is by using pip install -I . when installing your package from its directory.
Vanilla setuptools does not support downloading directly from a git repository but you can use one of the Download Source links from that page, like:
easy_install http://github.com/mtai/python-gearman/tarball/master