This python wheel website says, only 300 of the top 360 packages use wheel. I further analysed the Python ecosystem and found that about 2961 packages out of top 5000 use wheel, and others don't.
My questions are:
If they don't use wheel, do they use egg?
Why don't they use wheel? Is that just the laziness of authors or something else, which stop them from using wheel.
I also found from this post that wheel stops install time scripts (correct me if I'm wrong here). So, isn't it the case that because of some wheel functionalities, those packages can't use wheel (because they might need some functionalities of setup.py file, during the installation, e.g. install time scripts).
If they don't use wheel, do they use egg?
They probably don't. Wheels are built distributions, the alternative is to provide a source distribution, so this is likely what these packages are publishing instead (source distributions have filenames that end in .zip or .tar.gz.
Why don't they use wheel? Is that just the laziness of authors or something else, which stop them from using wheel.
Unless the project can be built with pure-Python wheels, building wheels for a certain platform requires access to a similar build environment. It's possible that they either don't have a given build environment, or don't have enough users to justify the extra work. It's also possible that their package is trivial enough that it doesn't make much difference to install from source vs. from a built distribution.
I also found from this post that wheel stops install time scripts (correct me if I'm wrong here).
This is correct: wheels are built for a given platform, and thus don't do anything at install-time other than put the package in the path.
So, isn't it the case that because of some wheel functionalities, those packages can't use wheel (because they might need some functionalities of setup.py file, during the installation, e.g. install time scripts).
Not really, any package that can be installed can produce a wheel. There is the possibility that a given package is doing more than just installing at install-time (e.g., perhaps it is also downloading some large files or something from an external source) but patterns like this are generally discouraged.
Related
From this answer, I am able to get the dependencies of a package. However I am looking for a solution that does not require you to download and install the package.
Is this possible?
Edit: I meant getting the package dependencies from PyPI. I want to get the package dependencies from a PyPI package, but my only solution requires me to install the package, which is not what I want. Is it possible to get the package dependencies from PyPI?
As far as I know, in many cases PyPI won't be able to deliver reliable info about the dependencies for a distribution. For example, for source distributions the list of dependencies is somewhat dynamic, and the only way to get a definitive static list of dependencies is to build the distribution (run the setup.py). Meaning that in some cases there is no way around downloading and building the distributions locally (with the correct Python interpreter). No need to install them though.
Note:
I believe johnnydep is a tool that does exactly that. But it doesn't answer your question as you are probably looking for a library or an API.
The advantage of wheels over eggs is clear (see section why not egg? https://pypi.python.org/pypi/wheel).
However, it is not entirely clear to me what is the advantage of using wheels over tar.gz. I might be missing something obvious like "they are the same".
As I see it both can be installed directly using pip (even in Windows), have similar size and when packaging require a similar effort.
It sounds to me like the kind of questions you might get when justifying a packaging methodology.
EDIT:
Just found an example where tar.gz might be better than wheels. CherryPy (https://pypi.python.org/pypi/CherryPy) provides wheels for Python 3.x only, so if you want to have a local repository to serve CherryPy for Python 2.7 and 3.x dependencies, it seems to make more sense to store the tarball. Is this correct? (just to add a couple of "case-based" justification to the discussion)
This answered it for me (directly from the wheel PEP):
Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new
virtualenv. This system of conflating build-install is slow, hard to
maintain, and hinders innovation in both build systems and installers.
Wheel attempts to remedy these problems by providing a simpler
interface between the build system and the installer. The wheel binary
package format frees installers from having to know about the build
system, saves time by amortizing compile time over many installations,
and removes the need to install a build system in the target
environment.
https://www.python.org/dev/peps/pep-0427/#rationale
Note the tarballs we're speaking of are what are referred to as "sdists" above.
From Python Wheels
Advantages of wheels
• Faster installation for pure python and native C extension packages.
• Avoids arbitrary code execution for installation. (Avoids setup.py)
• Installation of a C extension does not require a compiler on Windows or OS X.
• Allows better caching for testing and continuous integration.
• Creates .pyc files as part of installation to ensure they match the python interpreter used.
• More consistent installs across platforms and machines.
Make sure wheel is installed.
python3 -m pip install wheel
I'm writing a source bundle (not a fully packaged module, but some scripts with dependencies) to be installed and executed inside a framework application (Specifically, Amazon SageMaker's TensorFlow serving container - running Python 3.5).
One of my dependencies is matplotlib, which in turn needs kiwisolver, which has C++ components.
It seems like my target container doesn't have wheel installed by default, because when I supply just a requirements.txt file I get the error described in "Why is python setup.py saying invalid command 'bdist_wheel' on Travis CI?".
I think I got it working by supplying a setup.py instead, with setup_requires=["wheel"] as advised in the answers to that Travis CI question.
My Python packaging-fu is weak, so my question is: Who should be specifying this dependency, because it seems like it shouldn't be me?
Should kiwisolver be advertising that it needs wheel?
Does a framework application/environment installing user code modules via requirements.txt have an implicit contract to make wheel available in the environment, for some reason in Python's packaging ethos?
Maybe it really is on me to know that, since I'm indirectly consuming a module like kiwisolver, my package requires wheel for setup and a straight pip install -r requirements.txt won't work?
Even better if somebody can explain whether this answer is changing with PEP 518 and the deprecation of setup_requires :S
Usually wheel could be considered a build-time dependency and not an install-time dependency. But actually, wheel is just a way of distributing Python projects (libraries or applications), so it usually isn't a mandatory dependency.
The one system building the library (kiwisolver) might have a need to have the wheel tool installed. But if I am not mistaken recent versions of pip have wheel already bundled in, so nowadays there is often no need to install it explicitly.
In many cases there are wheels already built available on PyPI. But sometimes there are no wheels compatible with the target system (Python interpreter version, operating system, CPU bitness). In your case here, kiwisolver has a wide range of wheels available but not for Python 3.5.
So it seems like the system you want to install kiwisolver on, is not compatible with any of the wheels available on PyPI. So pip has to build it locally. Usually pip tries to build a wheel first, but as far as I know it's not a deal-breaker if a wheel cannot be built then pip usually just continues and installs the project without going to the wheel intermediary step.
But still pip has to be able to build the library, which might require some C/C++ compilers or that other unusual conditions are met on the local system. Which is why distributing libraries as wheel is very comfortable, since the build step is already done.
So to sum it up, from my point of view, no one really has to declare wheel as a dependency or install wheel unless they actually want to build wheels. But wheel really is just an intermediary optional step. It's a way of distributing Python projects (libraries or applications). I don't see the absolute need for adding wheel to setuptools' setup_requires (which is deprecated or on close to it) nor to pyproject.toml's build-system.requires, it's more of a (very common, and quasi standard) convenience.
Now what would I do in your situation?
Before installing from the requirements.txt file that contains kiwisolver (directly or indirectly) either make sure that pip is up-to-date or explicitly install wheel, and:
Use a version of Python for which wheels are already available on PyPI.
If you want to stay on Python 3.5:
Make sure the target system is able to build kiwisolver itself (maybe it requires a C/C++ compiler plus some other native libraries).
The installation of the virtual environment of my python application takes too much time during deployment due to a large amount of dependencies. In order to minize that time, I want to include the dependencies residing in the virtual environment in git, so that they are already there on deployment.
The main issue with that is that dependencies with C code need to be rebuild because of architecture differences between machines.
Is there a way to rebuild all dependencies that need compilation in my virtual environment?
wheel format is what you need
Popular example is lxml, which when installing from source on Linux takes about 3 minutes to get downloaded, compiled and installed.
Using wheel format and installing from local wheel file for lxml installs within fraction of a second.
For detailed instructions how I use it see corrected link to Detailed SO answer how to configure pip incl. instructions how to take advantage of wheels
For more information:
pythonwheels page listing already available wheels.
wheel ReadTheDocs
using pip to build a wheel
Some notes:
pure python packages can be distributed in wheel format regardless of target platform (apart from being possibly python version dependent).
compiled python packages shall be build on the same platform, where you are going to install them. There might be some cross-compile options, but I do not have real experience with that.
some do consider wheel of "package format of the future", others claim, that it is supposed to be build on your own side and use your own wheels. The later case is for lxml not being provided as a wheel - see launchpad issue related to lxml in wheel format. Consider adding there yourself as an affected person, if you care.
Once you manage using wheels the first time, you will love it.
I'm preparing my first package distribution for PyPI using setuptools etc. I've gotten the source distribution (.tar.gz) working well and now I'm wondering whether I should provide any other formats, like .zip and .egg.
On a random walk through PyPI I noticed some projects that provided eggs for different Python versions along with the source tarball. And I noticed one or two that provided the source distribution in .zip form.
What's the best practice for which formats to upload to PyPI? And do the various installation programs (easy_install, pip) have a preference for which one they use?
Current best practice is to add source distribution (sdist) and wheel (bdist_wheel). Look at what numpy does, for example.
If you have things to be built, you might save your users major headaches by providing a wheel. Compare the installation of numpy with wheel and without wheel (e.g. by installing a version for which they didn't create a wheel for your python version): without the wheel, you need a Fortran compiler and Blas libraries installed and usable. Without it, you don't even have to know that numpy uses Fortran.
If you have a pure Python package, the wheel format still has the advantage that it can be installed simply by copying the code to the correct location. I'm not sure how big this advantage is, though
See also: https://packaging.python.org/guides/distributing-packages-using-setuptools/#wheels
If you build no C-extensions, just ship the source tarball in tar.gz format. It can be installed on all platforms. A binary egg (installable by easy_install, not pip) is only neccessary if you build C-extensions and your software runs on Windows as people seldomly have a C-compiler on their machine.