I'm learning how to use distutils, and there's something I don't understand, and I wish someone could explain this to me.
I already am successful in creating tar.gz packages that can be installed with
pip install mypackage.tar.gz
I did this with setup.py, with a simple script that runs the function setuptools.setup(), which I call using python3 setup.py sdist.
What I wanna learn now: How to include building extensions in this.
My problem: I couldn't find any comprehensive text that explains how a pip install of a package that has a build_ext class can get it to build, then install.
If we look at this example, for the famous tool cx_freeze package, we see:
There's an inherited build_ext class
There's a method build_extension()
in setup(), the variable cmdclass, which contains a dict that contains the class build_ext
My question: What gets cx_freeze to build the extension and then install it? Is having a cmdclass defined with build_ext enough to achieve this?
After many tests, I learned that this is related to how pip works, not how setup.py works. It turns out that after writing your setup.py file and using pip to install, this is what happens:
pip creates a temporary file, in Linux it's in /tmp, and on Windows it's in the temp dir of the user.
pip downloads/extracts the package to that temp directory (whether from a tar.gz or from an online source or from a repository)
pip runs the following operations in order:
setup.py install
setup.py build
setup.py install_lib
setup.py build_py
setup.py build_ext
And all this depends on whether you have stuff defined in the cmdclass parameter of setup(). For example, build_ext will run only if you have build_ext defined in cmdclass AND you have ext_modules defined in the parameters of your setup() call. So ext_modules, for example, is expected to be a list of Extension(), which contains all the information about every extension. The function in the class build_extension(self,ext) will be executed on every element of that list.
All these classes that go to cmdclass (I use build and build_py) have a method called run() that you should override to put in your custom building procedure.
After all that is done, pip installs (or copies) the packages defined in setup() (which are basically directories in that temp) to your Python directory, which is the end of the installation, and deletes temp files.
There's more details to all this, but I guess this is good for a starter. I saw none of this explained anywhere comprehensively. So I hope this helps and saves people the empirical testing I had to do to learn this.
Related
I am attempting to change how the Python bindings for a "large" C++ software project (i.e. the bindings are a minor part) are shipped so that the bindings are no longer installed as bare, metadata-free .so files in site-packages. The bindings' .so files are already built as part of the rest of the software's mess of cmake instructions, and I'm adding instructions and the proper module structure (i.e. <module>/__init__.py) so that setuptools can be in charge of the install step. In setup.py, I have the prebuilt .so files included as package_data, as was done in the related question Distributing pre-built libraries with python modules. And when cmake &&
make && make install gets around to invoking python setup.py build and python setup.py install --root=${CMAKE_INSTALL_PREFIX}, everything technically works. Hooray.
The problem is that python setup.py install ... drops everything under {PREFIX}/usr/lib/pythonN.M/site-packages even though I've included binaries built for 64-bit arch in the package data. I'm struggling to figure out how to get setuptools to nicely install to, e.g. for Linux, /usr/lib64. I could possibly add some cmake logic to figure out what the libdir should be and pass it in as --install-lib to setup.py install, but it seems like the right thing to do is somehow make setuptools aware, during the install step, that the contents of the package are platform-specific and to set the install location accordingly.
I assume this is largely because I'm including the bindings as package data rather than building them inside the setup script as extension modules. Is there some way to tell setuptools that a package without explicit ext_modules is platform-specific so that python setup.py install places files under /usr/lib64/... instead of /usr/lib/...?
I am trying to package my Python project, which comes with a configuration dotfile that I want copied into the user's home directory on installation. The quick guide to packaging says that this can be done using the data_files argument to setuptools.setup. So this is what I have:
data_files = [(os.path.expanduser("~"), [".my_config"])]
This appears to work fine if I use python setup.py install, but when I upload my package to PyPI and run pip install the dotfile isn't copied.
FWIW, I've put the dotfile in the MANIFEST.in and also tried including the package_data argument to setup. None of these steps appear to make a difference. If I pip install and poke around the site-packages directory, just the source files are here.
How can I achieve what I'm looking for?
This is an issue I had once to experience myself. Its root is that when you are building a wheel file, all the absolute paths specified in data_files will be relativized to the target site-packages directory, see this issue on github. This influences installations performed by pip install as it will build a wheel out of any source package (.tar.gz, .tar.bz2 or .zip) and install the resulting wheel:
$ pip install spam-0.1.tar.gz
Processing ./spam-0.1.tar.gz
Building wheels for collected packages: spam
Running setup.py bdist_wheel for spam ... done
Stored in directory: /Users/hoefling/Library/Caches/pip/wheels/d0/95/be/bc79f1d589d90d67139481a3e706bcc54578fdbf891aef75c0
Successfully built spam
Installing collected packages: spam
Successfully installed spam-0.1
Checking installed files yields:
$ pip show -f spam
Name: spam
Version: 0.1
...
Location: /Users/hoefling/.virtualenvs/stackoverflow/lib/python3.6/site-packages
Requires:
Files:
Users/hoefling/.my_config
spam-0.1.dist-info/DESCRIPTION.rst
spam-0.1.dist-info/INSTALLER
spam-0.1.dist-info/METADATA
spam-0.1.dist-info/RECORD
spam-0.1.dist-info/WHEEL
spam-0.1.dist-info/metadata.json
spam-0.1.dist-info/top_level.txt
Note the path meant to be absolute is relative to the Location dir. In the example, .my_config would be placed under /Users/hoefling/.virtualenvs/stackoverflow/lib/python3.6/site-packages/Users/hoefling/.my_config.
It gets even better because these built wheels are cached on your disk, so next time you reinstall the package and the built wheel still exists in pip's cache, it will be used for the installation and you won't even see any mentions of building a wheel in the terminal log.
There is no real solution to avoid this. The most decent workaround I found is to prohibit "binary" packages when installing to enforce the execution of package's setup.py on installation:
$ pip install spam-0.1.tar.gz --no-binary=spam
Processing ./spam-0.1.tar.gz
Skipping bdist_wheel for spam, due to binaries being disabled for it.
Installing collected packages: spam
Running setup.py install for spam ... done
Successfully installed spam-0.1
The file is now placed correctly:
$ pip show -f spam
Name: spam
Version: 0.1
...
Location: /Users/hoefling/.virtualenvs/stackoverflow/lib/python3.6/site-packages
Requires:
Files:
../../../../../.my_config
spam-0.1-py3.6.egg-info/PKG-INFO
spam-0.1-py3.6.egg-info/SOURCES.txt
spam-0.1-py3.6.egg-info/dependency_links.txt
spam-0.1-py3.6.egg-info/top_level.txt
Unfortunately, the user must be separately informed about calling pip install with the extra key (via readme, webpage FAQ or similar) as there is no possibility to prohibit building the wheel in package metadata.
As the result, I do not include files with absolute paths anymore. Instead, I install them with the python sources in the site-packages dir. In the python code, I have to add additional logic for the existence checks and file copying if necessary:
# program entrypoint
if __name__ == '__main__':
config = os.path.join(os.path.expanduser('~'), '.my_config')
if not os.path.exists(config):
shutil.copyfile('.my_config', config)
main.run()
Besides what #hoefling said, I suggest you not using data_files at all! Because it's really unpredictable where the files will be copied to. You could test this by giving the directory something like '', '/', or '/anything/you/want'.
I suggest you use package_data instead, which just copies the files under the distributed package root on installation. Then you can copy that to anywhere you want at run time.
For more on package_data, refer to Python Doc https://docs.python.org/2/distutils/setupscript.html#installing-package-data
We want to exclude the Python source code from the package we create. But after configuring setup.py, I failed excluding the py files. I have been using python setup.py bdist_wheel as command. Is there any way to exclude to source code from Python package? Basically we do not want to expose the source codes.
The wheel plugin for setuptools that handles bdist_wheel does not have the --exclude_source_files which is only supported by bdist_egg. The egg packages are however deprecated and not supported by pip for example. What you can do however:
pip3 install wheel
python3 setup.py bdist_egg --exclude-source-files
wheel convert dist/mypackage-1.0-py3.6.egg
The wheel utility takes a source-stripped egg and converts it into whl package.
This came up for us putting Python libs on an embedded system with very minimal available memory.
It can be achieved with:
./setup.py bdist_egg --exclude_source_files
Works for both distutils and setuptools.
Well, we also encountered this issue (around 2 years ago) and didn't find a sensible automation process for it. So I wrote my own.
You're welcomed to it: setup.py template
My setup.py script is simple:
from distutils.core import setup
setup(name='my-awesome-app',
version='1.0',
scripts=['my-awesome-app.py'],
)
And the file structure is:
my-awesome-app/
my-awesome-app.py
setup.py
In theory I am only including my-awesome-app.py in the distribution. In practice setup.py ends up in the RPM too.
I don't see a point of including setup.py there, is there a way to force distutils to leave this file out?
I am using python 2.7, I build my RPM by running python setup.py bdist_rpm.
Thanks for help :)
setup.py is required because when the package is installed in your environment, the following command is run:
$ python setup.py install
Running python setup.py bdist_rpm only creates a distribution package that you can give to others. setup.py is still required to do the installation.
You can always create the spec file manually and leave out the setup.py.
For example and more details see:
https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file
I am running into a bizarre problem where I try to install my python package (using setuptools) on a Linux system and it tries to install pywin32 as a dependency. This only happens if I run it from a specific directory. If I move the whole tree to a different directory, it installs fine.
My setup.py does reference pywin32 for win32 platforms, but even when I remove this it makes no difference. I've also removed all the other requirements without any change in behavior. My setup.py has nothing in it other than setuptools initialization and a call to setup().
I have a virtualenv active and have tried switching to a new clean virtualenv without a change in behavior. I see the following when I run setup.py develop:
python setup.py develop
running develop
running egg_info
deleting foo.egg-info/requires.txt
writing foo.egg-info/PKG-INFO
writing top-level names to foo.egg-info/top_level.txt
writing dependency_links to foo.egg-info/dependency_links.txt
reading manifest file 'foo.egg-info/SOURCES.txt'
writing manifest file 'foo.egg-info/SOURCES.txt'
running build_ext
Creating /home/user/foo-env2/lib/python2.7/site-packages/foo.egg-link (link to .)
foo 1.0 is already the active version in easy-install.pth
Installed /home/user/magicdirectory/magic
Processing dependencies for foo==1.0
Searching for pywin32
Reading http://pypi.python.org/simple/pywin32/
Reading http://sf.net/projects/pywin32
Now, I may have accidentally tried to install a win32 egg of this package at some point, but that was in the original virtual environment and I can't figure out what it could have done to cause this.
How do I debug this? Where else could setuptools be searching for dependencies from?
UPDATE: It appears that it's not the name of the directory that matters, but where it is. If I move the package directory up one level, it will install fine. The directory in question here has been deleted and recreated multiple times so it doesn't appear to be related to anything inside the directory.
Try removing the *.egg-info in the dist develop directory. If that doesn't work, try starting with a fresh, untouched checkout of your code without any of the other setuptools/dist artifacts and see if that reproduces the problem. If it doesn't then you know it's one of the setuptools/dist artifacts that just needs to be cleaned up.