I'm working on a buildout script that needs to install a distutils package on remote server.
On PyPi there are 2 recipes for doing this
collective.recipe.distutils 0.1 and zerokspot.recipe.distutils 0.1.1.
The later module a derivative of the former, and is a little more convenient then the first, but the both suffer from the same problem, which I will describe now.
When bootstrap.py is executed, it downloads zc.buildout package and puts it into buildout's eggs directory. This gives ./bin/buildout access to zc.buildout code, but /usr/local/python does not know anything about zc.buildout at this point.
Buildout attepts to install the package by running 'python setup.py install' inside of a subprocess. This produces an ImportError because zc.buildout is not installed for /usr/local/python.
So, I have several solutions.
Install zc.buildout using easy_install on the remote server. I don't like this option at all, it makes a special case for a module that is very insignificant.
Modify zerokspot.recipe.distutils to put try block around 'import zc.buildout' this way, it will install even if zc.buildout is not installed. It's an ok solution, but somewhat hackish.
Replace subprocess with code that will install the package using distutils api or setuptools api. This would be the best solution in my opinion.
The question is how would i do #3?
Thank you,
Taras
PS: I solved the problem by creating another package that does not have dependancy on zc.buildout. My package is called taras.recipe.distutils and it's available on pypi.
You can call a command line program within your Python program using the subprocess module:
import subprocess
subprocess.call('python setup.py install')
However, how much control do you have over the environment that this install will be run? If it is a package that you are distributing, you will likely have issues no matter what solution people propose. How will you handle cases of needing root access (e.g. sudo python setup.py install)?
You may consider looking into Paver since it provides an API that is in some ways an extension of setuptools.
zerokspot.recipe.distutils is fundamentally broken in that it adds a dependency on zc.buildout in it's setup.py, as follows:
setup.py imports get_version from zerokspot.recipe.distutils
All of zerokspot.recipe.distutils is defined in it's __init__.py, including get_version
__init__.py in zerokspot.recipe.distutils imports zc.buildout
Why the author defines get_version is a mystery to me; best practice keeps a simple version string in setup.py itself and lets setuptools deal with dev versions (through setup.cfg), and distutils for version metadata extraction.
Generally it is not a good idea to import the whole package in setup.py as that would require all the package dependencies to be present at install time. Obviously the author of the package has zc.buildout installed as a site-wide package and didn't notice his oversight.
Your best bet is to fork the package on github, remove the get_version dependency, and propose the change to the original author while you use your fork instead.
Are you sure you don't want to just generate a bdist?
Related
This weekend I've been reading up on conda and the python packaging user guide because I have a simple pure python project that depends on numpy. It seemed to me that distributing/installing this project via conda was better than pip due to this dependency.
One thing on which I'm still not clear: conda will install a python package from a recipe in build.sh, but it seems like build.sh just ends up calling python setup.py install for most python packages.
So even if I want to distribute/install my python package with conda, I still end up depending on setuptools (or distutils) for the actual installation, correct? I was unable to find a conda utility analogous to setuptools; am I missing something?
FWIW, I posted this question on the conda issue tracker.
Thanks!
Typically you will still be using distutils (or setuptools if the library requires it) to install things, yes. It is not technically required. The build.sh can be anything. If you wanted to, you could just copy the code into site-packages. Using setup.py install is recommended, though, as libraries will already have setup.py working, it will install metadata that can be read by pip, and it will compile any extension modules and install any data files.
A lot of software packages require users of the system to install a set of dependencies first before they can use the software. (This is a general question not specific to python, but I'll speak in context of python.). Can't we make the python module install its own dependencies before executing the code if the dependencies are not already installed. This should be doable with the help of system or subprocess calls but I rarely see people doing this.
For example lets say lib is a python library that needs to be used in the python file main.py:
import os
try:
import lib
except:
os.system('pip install pdir')
import lib
# Can make use of lib now
Is there anything potentially wrong with this approach? Could doing something like this cause problems for big projects?
Note: The advantage here is that a user using the file does not have to install the dependencies separately, he can simply run python main.py. And the second thing that I realize is that such approach makes sense only when virtualenv is being used.
You should never do this - pip doesn't have dependency resolution so there's no guarantee that you'll get a certain version. Dependencies should be installed using setup.py, requirements.txt or a different approach.
You also shouldn't need user permissions or sudo to install the packages just for running the code. The user should be aware of the packages that are needed for installing your package as they may come from PyPI or the OS's package system or an internal company PyPI mirror - and silently installing dependencies is not a good idea in that case.
You could always consider using a more advanced print statement to inform the user.
try:
import ConfigParser
except ImportError as err:
print '\n'.join([i + ''.join(str(err).split(' ')[-1:]) for i in ['$ pip install ', '$ easy_install ']])
You can recommend them installing the missing package this way.
$ pip install ConfigParser
$ easy_install ConfigParser
Additionally you could consider having a ImportError be reason to recommend the user to run their ./setup.py or ./INSTALL instead of just the missing module.
I've created a Python module on Github that uses Nose for unit testing and Sphinx for generating documentation. I have two questions:
Should I include Sphinx and/or Nose in my module's dependencies in setup.py (install_requires), as they are not required for basic module functionality, only if you want to build the documentation/run tests yourself?
Should I include Sphinx and/or Nose in my module's requirements.txt on Github, for the same reasons but users that download my project from Github might be more likely to build docs/run tests?
This is my first Python module, so a bit of best practices/standards advice would be appreciated.
If nose and/or sphinx are not required for the basic functionality of your package then don't include them in setup.py. There's no point in forcing users to install packages that they might not ever use. If they eventually want to help you develop your package they can install the requisite packages themselves.
requirements.txt files should also not include development-required packages, though there's some wiggle room there.
For example, over at pandas we use requirements files for our Travis-CI builds. You can check them out here.
One thing we are considering is building our documentation on Travis-CI, as sometimes a failed doc build catches bugs that the test suite doesn't. In that case we would put sphinx in the requirements file of the Python version we use to build the documentation.
Don't include those nice-to-haves in your setup.py. You can write a requirements file for developers if you like; users won't need one. For example, call one file reqs.development:
-e . # include the package defined by setup.py in editable (development) mode
nose
sphinx
Users can pip install yourmodule or pip install https://your/tarball, developers can fork, clone and pip install -r reqs.development.
I have read the documentation but I don't understand.
Why do I have to use distutils to install python modules ?
Why do I just can't save the modules in python path ?
You don't have to use distutils. You can install modules manually, just like you can compile a C++ library manually (compile every implementation file, then link the .obj files) or install an application manually (compile, put into its own directory, add a shortcut for launching). It just gets tedious and error-prone, as every repetive task done manually.
Moreover, the manual steps I listed for the examples are pretty optimistic - often, you want to do more. For example, PyQt adds the .ui-to-.py-compiler to the path so you can invoke it via command line.
So you end up with a stack of work that could be automated. This alone is a good argument.
Also, the devs would have to write installing instructions. With distutils etc, you only have to specify what your project consists of (and fancy extras if and only if you need it) - for example, you don't need to tell it to put everything in a new folder in site-packages, because it already knows this.
So in the end, it's easier for developers and for users.
what python modules ? for installing python package if they exist in pypi you should do :
pip install <name_of_package>
if not, you should download them .tar.gz or what so ever and see if you find a setup.py and run it like this :
python setup.py install
or if you want to install it in development mode (you can change in package and see the result without installing it again ) :
python setup.py develop
this is the usual way to distribute python package (the setup.py); and this setup.py is the one that call disutils.
to summarize this distutils is a python package that help developer create a python package installer that will build and install a given package by just running the command setup.py install.
so basically what disutils does (i will sit only important stuff):
it search dependencies of the package (install dependencies automatically).
it copy the package modules in site-packages or just create a sym link if it's in develop mode
you can create an egg of you package.
it can also run test over your package.
you can use it to upload your package to pypi.
if you want more detail see this http://docs.python.org/library/distutils.html
You don't have to use distutils to get your own modules working on your own machine; saving them in your python path is sufficient.
When you decide to publish your modules for other people to use, distutils provides a standard way for them to install your modules on their machines. (The "dist" in "distutils" means distribution, as in distributing your software to others.)
This is somewhat related to this question. Let's say I have a package that I want to deploy via rpm because I need to do some file copying on post-install and I have some non-python dependencies I want to declare. But let's also say I have some python dependencies that are easily available in PyPI. It seems like if I just package as an egg, an unzip followed by python setup.py install will automatically take care of my python dependencies, at the expense of losing any post-install functionality and non-python dependencies.
Is there any recommended way of doing this? I suppose I could specify this in a pre-install script, but then I'm getting into information duplication and not really using setuptools for much of anything.
(My current setup involves passing install_requires = ['dependency_name'] to setup, which works for python setup.py bdist_egg and unzip my_package.egg; python my_package/setup.py install, but not for python setup.py bdist_rpm --post-install post-install.sh and rpm --install my_package.rpm.)
I think it would be best if your python dependencies were available as RPMs also, and declared as dependencies in the RPM. If they aren't available elsewhere, create them yourself, and put them in your yum repository.
Running PyPI installations as a side effect of RPM installation is evil, as it won't support proper uninstallation (i.e. uninstalling your RPM will remove your package, but leave the dependencies behind, with no proper removal procedure).