Develop Reference

How to build whl package for pandas?

Hi I have a built up Python 2.7 environment with Ubuntu 19.10.
I would like to build a whl package for pandas.
I pip installed the pandas but do not know how to pack it into whl package.
May I ask what I should do to pack it.
Thanks

You cannot pack back an installed wheel. Either you download a ready-made wheel with pip download or build from sources: python setup.py bdist_wheel (need to download the sources first).

what does pip install actually do?

Super newb question here..
What does pip install actually do?
Assuming the pypi package is a tarball...
Does it just download the tar.gz, unpack it and run setup.py?
Does it add the downloaded package to the site_packages folder?
I want to create a pip installable pkg using pypiserver so my colleagues can download my pkg in a painless way, but am a little unsure exactly what to include beyond the actual .py scripts.
Any guidance would be appreciated

The tar.gz file is a source archive whereas the .whl file is a built
distribution. Newer pip versions preferentially install built
distributions, but will fall back to source archives if needed. You
should always upload a source archive and provide built archives for
the platforms your project is compatible with. In this case, our
example package is compatible with Python on any platform so only one
built distribution is needed.
See: https://packaging.python.org/tutorials/packaging-projects/
You would typically not manually create the source archive and wheel, but use setuptools and wheel to do so for you.
Nowadays, many packages are wheels. While installing these wheels using pip, pip will:
[...] unpack the archive in your current site packages directory and install any console scripts contained in the wheel.
See: https://wheel.readthedocs.io/en/stable/user_guide.html#installing-wheels

in virtualenv, pip installing locally from source fails

I am trying to install a library in a virtualenv instance with pip. The library version I want (wxPython 3.0.2)
is not available on PyPi; it is only available for download from SourceForge. Thus, I have the source tarball downloaded on my machine and I am trying to install it in such a way that it will play nicely with virtualenv.
(I am on a Windows computer, running Python 2.7.)
I have tried the following:
doing a direct install: pip install wxPython-src-3.0.2.0.tar.bz2
extracting the files from the tarball to wxPython-src-3.0.2.0, then installing from the extracted directory: pip install wxPython-src-3.0.2.0
extracting the files from the tarball, then navigating into the extracted folder to the nested wxPython directory, which holds the setup.py file, and then installing from there: pip install wxPython
The last attempt seems the most promising, but I get the following traceback:
Processing \wxpython-src-3.0.2.0\wxpython
Complete output from command python setup.py egg_info:
Setuptools must be installed to build an egg
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\__MY_USERNAME__\appdata\local\temp\pip-req-build-q0pxlt\
This is also strange, because it suggests I don't have setuptools even though I can run pip list and see version 40.6.3 installed.
Any help appreciated.

Why not install a precompiled version? There are a lot of .exe files at SF. You probably need wxPython3.0-win64-3.0.2.0-py27.exe.
Also take a look at Christoph Gohlke's collection.
If you still insist on installing from sources please bear in mind that wxPython 3 is so old it predates pip. Forget about pip.
First, you need to install wxWidgets as wxPython is just a Python wrapper for wxWidgets C++ library. Extract wxPython-src-3.0.2.0.tar.bz2 and follow instructions in wxPython-src-3.0.2.0/docs/msw/install.txt.
After compiling and installing wxWidgets compile wxPython. See wxPython-src-3.0.2.0/wxPython/docs/BUILD.txt.

My eventual solution was the easy way out: installing my package (wxPython) locally as #phd suggested, and opting for local package access via either virtualenv --system-site-packages env or deleting the "no-global-site-packages.txt" file in an existing environment folder.
Not what I expected to do, but it works so no complaints.

Is it possible to require PyQt from setuptools setup.py?

I'm building a small app that uses PyQt and tought it'd be nice to declare that dependency in setup.py.
However, according to this blog (first hit on google for pyqt setuptools) says it can't be done, and the last paragraph here doens't try to do it either.
Ideas? Perhaps I should switch to PySide which is on PyPi?
Update:
The obvious install_requires = [ 'pyqt >= 0.7' ] in setup.py gives me:
D:\3rd\BuildBotIcon\my-buildboticon\python>setup.py test
running test
install_dir .
Checking .pth file support in .
C:\Python26-32\pythonw.exe -E -c pass
Searching for pyqt>=4.7
Reading http://pypi.python.org/simple/pyqt/
Reading http://www.riverbankcomputing.com/software/pyqt/
Reading http://www.riverbankcomputing.com/software/pyqt/download
No local packages or download links found for pyqt>=4.7
error: Could not find suitable distribution for Requirement.parse('pyqt>=4.7')

Right, the PyQT packages are not using distutils / setup.py for it's installation, so they can't be installed with easy_install or pip. You need to download and install it manually.
That also means you should not put it in your requires metadata, as easy_install and pip then will try to install it and fail.
I don't know if PySide is any good, but is also has not setup.py, and also refuse to install with easy_install/pip, so not a good option. :)
Another option is to repackage PyQt with distutils, but that may be a lot of work.

While you can pip install pyqt5 thanks to the now available wheels (as suggested by #mfitzp), it cannot be required from setup.py via install_requires. The reason is that setuptools doesn't know how to install wheels which pip knows how to, and PyQT5 is only available as wheels on PyPI (there is no source distribution, i.e. no tar.gz file). See this email and that bug report for details.

While the accepted answer was originally correct Python Wheels now provide a means to install C extension packages such as PyQt5 without the need for compilation from source.
PyPi currently has .whl files for PyQt5 on Python3 for multiple platforms, including MacOS X, Linux (any), Win32 and Win64. For example, this is the output when pip-installing PyQt5 on Python3 on a Mac:
mfitzp#MacBook-Air ~ $ pip3 install pyqt5
Collecting pyqt5
Downloading PyQt5-5.6-cp35-cp35m-macosx_10_6_intel.whl (73.2MB)
100% |████████████████████████████████| 73.2MB 2.5kB/s
Collecting sip (from pyqt5)
Downloading sip-4.18-cp35-cp35m-macosx_10_6_intel.whl (46kB)
100% |████████████████████████████████| 49kB 1.8MB/s
Installing collected packages: sip, pyqt5
Successfully installed pyqt5-5.6 sip-4.18
If you are targeting Python3+PyQt5 then you should have no problem specifying PyQt5 as a normal dependency in setup.py.

Setuptools >= 38.2.0 now knows how to install wheels. The trivial answer, therefore, is to install a recent version of setuptools and require that your enlightened userbase does so as well. To enforce usage of setuptools >= 38.2.0 at installation time, see this relevant answer elsewhere.
Since setuptools 38.2.0 was released over a year ago, all prior answers to this question are horrifyingly obsolete, blatantly wrong, and less than useful.

Python packages installation in Windows

I recently began learning Python, and I am a bit confused about how packages are distributed and installed.
I understand that the official way of installing packages is distutils: you download the source tarball, unpack it, and run: python setup.py install, then the module will automagically install itself
I also know about setuptools which comes with easy_install helper script. It uses eggs for distribution, and from what I understand, is built on top of distutils and does the same thing as above, plus it takes care of any dependencies required, all fetched from PyPi
Then there is also pip, which I'm still not sure how it differ from the others.
Finally, as I am on a windows machine, a lot of packages also offers binary builds through a windows installer, especially the ones that requires compiling C/Fortran code, which otherwise would be a nightmare to manually compile on windows (assumes you have MSVC or MinGW/Cygwin dev environment with all necessary libraries setup.. nonetheless try to build numpy or scipy yourself and you will understand!)
So can someone help me make sense of all this, and explain the differences, pros/cons of each method. I'd like to know how each keeps track of packages (Windows Registry, config files, ..). In particular, how would you manage all your third-party libraries (be able to list installed packages, disable/uninstall, etc..)

I use pip, and not on Windows, so I can't provide comparison with the Windows-installer option, just some information about pip:
Pip is built on top of setuptools, and requires it to be installed.
Pip is a replacement (improvement) for setuptools' easy_install. It does everything easy_install does, plus a lot more (make sure all desired distributions can be downloaded before actually installing any of them to avoid broken installs, list installed distributions and versions, uninstall, search PyPI, install from a requirements file listing multiple distributions and versions...).
Pip currently does not support installing any form of precompiled or binary distributions, so any distributions with extensions requiring compilation can only be installed if you have the appropriate compiler available. Supporting installation from Windows binary installers is on the roadmap, but it's not clear when it will happen.
Until recently, pip's Windows support was flaky and untested. Thanks to a lot of work from Dave Abrahams, pip trunk now passes all its tests on Windows (and there's a continuous integration server helping us ensure it stays that way), but a release has not yet been made including that work. So more reliable Windows support should be coming with the next release.
All the standard Python package installation mechanisms store all metadata about installed distributions in a file or files next to the actual installed package(s). Distutils uses a distribution_name-X.X-pyX.X.egg-info file, pip uses a similarly-named directory with multiple metadata files in it. Easy_install puts all the installed Python code for a distribution inside its own zipfile or directory, and places an EGG-INFO directory inside that directory with metadata in it. If you import a Python package from the interactive prompt, check the value of package.__file__; you should find the metadata for that package's distribution nearby.
Info about installed distributions is only stored in any kind of global registry by OS-specific packaging tools such as Windows installers, Apt, or RPM. The standard Python packaging tools don't modify or pay attention to these listings.
Pip (or, in my opinion, any Python packaging tool) is best used with virtualenv, which allows you to create isolated per-project Python mini-environments into which you can install packages without affecting your overall system. Every new virtualenv automatically comes with pip installed in it.
A couple other projects you may want to be aware of as well (yes, there's more!):
distribute is a fork of setuptools which has some additional bugfixes and features.
distutils2 is intended to be the "next generation" of Python packaging. It is (hopefully) adopting the best features of distutils/setuptools/distribute/pip. It is being developed independently and is not ready for use yet, but eventually should replace distutils in the Python standard library and become the de facto Python packaging solution.
Hope all that helped clarify something! Good luck.

I use windows and python. It is somewhat frustrating, because pip doesn't always work to install things. Python is moving to pip, so I still use it. Pip is nice, because you can uninstall items and use
pip freeze > requirements.txt
pip install -r requirements.txt
Another reason I like pip is for virtual environments like venv with python 3.4. I have found venv a lot easier to use on windows than virtualenv.
If you cannot install a package you have to find the binary for it. http://www.lfd.uci.edu/~gohlke/pythonlibs/
I have found these binaries to be very useful.
Pip is trying to make something called a wheel for binary installations.
pip install wheel
wheel convert path\to\binary.exe
pip install converted_wheel.whl
You will also have to do this for any required libraries that do not install and are required for that package.