I am trying to deploy my python virtualenv as centos rpm.
Following steps I have taken.
Created virtual environment with required dependencies.
One of requirement is pyOpenSSL
Built rpm package.
Now when installed on fresh centos instance, I am getting error as
'No module named OpenSSL'.
Is there any different procedure for pyOpenSSL module or do we need to install openssl-devel and openssl explicitly on new machine.
Is there any different procedure for pyOpenSSL module or do we need to install openssl-devel and openssl explicitely on new machine.
First, I'm pretty sure pyOpenSSL only requires openssl-devel as a build dependency, not a runtime dependency. So, as long as you're distributing a pre-built copy (whether by tarring up a portable virtualenv, using a wheel file in a custom repo, or otherwise), openssl-devel shouldn't be a problem, only openssl.
But openssl is a problem. pyOpenSSL is a pure-Python wrapper; all it does is dlopen the .so file and call a bunch of functions out of it. Without the .so, it can't do anything useful at all.
I'm a bit surprised that any (non-embedded) linux distro doesn't come with OpenSSL. If CentOS comes with openssl pre-installed, then you already have all your dependencies, and there's nothing to do here.
But, given that, as far as I know, OpenSSL still isn't part of LSB, this is at least a conceivable problem, so let's deal with it.
As a brief sidebar: If you switched to a different OpenSSL wrapper (there are at least three major alternatives) that's written in C (or SWIG or SIP) rather than Python and links OpenSSL directly rather than by cffi, everything would be a lot simpler. For example, I believe with M2Crypto, all you have to do is create a static-only build of OpenSSL, then build M2Crypto with --openssl=/path/to/static/openssl, and you're done.
But if you're set on pyOpenSSL, is there any way around this? Yes, but it's not going to be easy.
The good news is that pyOpenSSL isn't a wrapper around OpenSSL itself, but the lower-level cryptography module. And cryptography uses cffi rather than ctypes, which means it can be distributed with a pre-built helper module, which can be statically linked against OpenSSL.
Unfortunately, while cryptography does have built-in support for pre-building its helpers with statically linked OpenSSL on Windows, last time I checked, it did not have that support on other platforms. That means you're either going to need to add that support yourself, or trick it in some way.
If you don't know much about this stuff, you're almost certainly going to need step-by-step help with this. Read the docs on cffi, and both the Building cryptography on Linux docs and the bundled INSTALL file, and make sure you understand static linking and GNU ld, and then go to the cryptography-dev mailing list. Or maybe go to the bug tracker and search for or create an issue for static linking OpenSSL on linux.
But if you know what you're doing, here's the basic idea:
To build in static linking support, look in the OpenSSL binding and binding utilities source to see what it does on Windows and do the equivalent on Linux.
To work around it, what you want to do is build OpenSSL locally, copy it inside the cryptography source tree, trick cryptography into link to libopenssl.so (and friends) with relative rather than absolute paths, and modify its setup.py to copy the .so files into the package itself.
If you want to get really hacky, instead of modifying setup.py, just copy the .so files into the package manually after installation. (This will obviously only work if you're using a distribution mechanism based on bundling up the virtualenv, rather than rebuilding it.)
Another option is to use an executable packager to automatically bundle any dependent non-system libraries into the executable. Unfortunately, the only one I know of that does that kind of thing on linux is pyInstaller, and last I checked it didn't know how to handle cryptography/pyOpenSSL, so again you'd have to figure it out and extend pyInstaller yourself. If you want to go this way, take a look at the other recipes there. Also, I believe py2app can handle pyOpenSSL; while it's Mac-specific, looking at what it does may help you do the same on linux.
Related
The advantage of wheels over eggs is clear (see section why not egg? https://pypi.python.org/pypi/wheel).
However, it is not entirely clear to me what is the advantage of using wheels over tar.gz. I might be missing something obvious like "they are the same".
As I see it both can be installed directly using pip (even in Windows), have similar size and when packaging require a similar effort.
It sounds to me like the kind of questions you might get when justifying a packaging methodology.
EDIT:
Just found an example where tar.gz might be better than wheels. CherryPy (https://pypi.python.org/pypi/CherryPy) provides wheels for Python 3.x only, so if you want to have a local repository to serve CherryPy for Python 2.7 and 3.x dependencies, it seems to make more sense to store the tarball. Is this correct? (just to add a couple of "case-based" justification to the discussion)
This answered it for me (directly from the wheel PEP):
Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new
virtualenv. This system of conflating build-install is slow, hard to
maintain, and hinders innovation in both build systems and installers.
Wheel attempts to remedy these problems by providing a simpler
interface between the build system and the installer. The wheel binary
package format frees installers from having to know about the build
system, saves time by amortizing compile time over many installations,
and removes the need to install a build system in the target
environment.
https://www.python.org/dev/peps/pep-0427/#rationale
Note the tarballs we're speaking of are what are referred to as "sdists" above.
From Python Wheels
Advantages of wheels
• Faster installation for pure python and native C extension packages.
• Avoids arbitrary code execution for installation. (Avoids setup.py)
• Installation of a C extension does not require a compiler on Windows or OS X.
• Allows better caching for testing and continuous integration.
• Creates .pyc files as part of installation to ensure they match the python interpreter used.
• More consistent installs across platforms and machines.
Make sure wheel is installed.
python3 -m pip install wheel
Our command line utility is written with Python.
On Linux/OS X this is usually not a problem since both come with Python 2.x pre installed. However on Windows, Python isn't installed by default.
Additional problem is that few of our dependencies require compiling which yet again is not a trivial problem for Windows users since it requires tinkering with MSVC/Cygwin/etc'.
Up until now we solved this issue by using Pyinstaller to create "frozen" Python package with pre-installed dependencies. This worked well, however made our utility non extendable - we cannot add additional Python modules by using utilities such as pip for example. Since our CLI depends on this ability in order to add additional usability, this limitation became a blocker for us and we would like to rethink our approach.
Searching around, I found how Rhodecode solve this. Basicly their installer brings Python and everything else (including pre-compiled dependencies).
This seem as a good idea for us, the only limitation I see here is that their installer actually installs Python from .msi which puts stuff in Windows Registry. So there can be only one Python of version X.Y installed on Windows (from .msi)
For a server application this might be reasonable since server application tends to act like it's the only thing installed on the PC, but for command line utility, this is completely unacceptable.
Looking around I found few projects that claim to make Python portable - for example Portable Python. However I'm not sure how "portable" it really is, especially after issues like this.
So questions are:
Is it possible to install same Python version multiple times on Windows without creating collisions between the instances?
Would you choose other workaround to solving this problem (please no "smart" solutions such as: drop Windows support/don't use Python)
Thanks!
Frankly I would stick with PyInstaller or something similar. That will always provide you with the correct version of Python whether or not the target machine has Python installed. It also protects you from clobbering a previously installed version of Python.
If you need to add plugins, then you should build that into your app. There are a few projects that might help you with that. Here are a couple of examples:
http://yapsy.sourceforge.net/
https://pypi.python.org/pypi/Plugins/
You might also take a look at how Django or Flask handles extensions. For example, you can add an extension to Flask to allow it to work with SQLAlchemy. You should be able to do something similar with your own application. The pip utility won't work with a frozen application after all. Alternatively, have you looked at conda? It might work for your purposes.
This is an answer for my second question, sadly I still haven't figured out a better solution for number 1.
For now here's how we changed our approach for creating setup file:
We package our code and its dependencies as set of pre-built Python wheels. It's relatively easy to create pre-built wheels on Windows since the release of Visual C++ compiler for Python2.7.
We package Python setup MSI together with Pip, Setuptools and Virtualenv wheels.
Before install starts we check whether Python, Pip and Virtualenv are already installed (by looking in the registry and \Scripts), if it's not, we install it from the packaged wheels. Pip wheel is installed by using get-pip.py script which we bundle as well.
We create separate Virtualenv and install our wheels into it.
Uninstall is done by removing the virtualenv folder. Python, Pip and Virtualenv install are left behind.
Installer created by Inno Setup.
I'm still not fully satisfied with this solution since some components are globally installed and might collide with what user had already installed before (older/newer version of python, pip, setuptools or virtualenv). This creates potential for unpredictable bugs during install or runtime. Or the possibility for user to upgrade one of the components in the future and somehow break the application.
Also, uninstall is dirty and leaves stuff behind.
I'm an author of a pure Python library that aims to be also convenient to use from a command line. For Windows users it would be nice just installing the package from an .exe or .msi package.
However I cannot get the installer to install package dependencies (especially the dependency on setuptools itself, so that running the software fails with an import error on pkg_resources). I don't believe that providing an easy .exe installer makes much sense, if the user then needs to manually install setuptools and other libraries on top. I'd rather tell them how to add easy_install to their PATH and go through this way (http://stackoverflow.com/questions/1449494/how-do-i-install-python-packages-on-windows).
I've build .exe packages in the past, but don't remember if that ever worked the way I'd preferred it to.
It is quite common to distribute packages that have dependencies, especially those as you have, but I understand your wish to make installation as simple as possible.
Have a look at deployment bootstrapper, a tool dedicated to solving the problem of delivering software including its prerequisites.
Regardless of what packaging method you eventually choose, maintain your sanity by staying away from including MSIs in other MSI in any way. That just does not work because of transactional installation requirements and locking of the Windows Installer database.
I recently began learning Python, and I am a bit confused about how packages are distributed and installed.
I understand that the official way of installing packages is distutils: you download the source tarball, unpack it, and run: python setup.py install, then the module will automagically install itself
I also know about setuptools which comes with easy_install helper script. It uses eggs for distribution, and from what I understand, is built on top of distutils and does the same thing as above, plus it takes care of any dependencies required, all fetched from PyPi
Then there is also pip, which I'm still not sure how it differ from the others.
Finally, as I am on a windows machine, a lot of packages also offers binary builds through a windows installer, especially the ones that requires compiling C/Fortran code, which otherwise would be a nightmare to manually compile on windows (assumes you have MSVC or MinGW/Cygwin dev environment with all necessary libraries setup.. nonetheless try to build numpy or scipy yourself and you will understand!)
So can someone help me make sense of all this, and explain the differences, pros/cons of each method. I'd like to know how each keeps track of packages (Windows Registry, config files, ..). In particular, how would you manage all your third-party libraries (be able to list installed packages, disable/uninstall, etc..)
I use pip, and not on Windows, so I can't provide comparison with the Windows-installer option, just some information about pip:
Pip is built on top of setuptools, and requires it to be installed.
Pip is a replacement (improvement) for setuptools' easy_install. It does everything easy_install does, plus a lot more (make sure all desired distributions can be downloaded before actually installing any of them to avoid broken installs, list installed distributions and versions, uninstall, search PyPI, install from a requirements file listing multiple distributions and versions...).
Pip currently does not support installing any form of precompiled or binary distributions, so any distributions with extensions requiring compilation can only be installed if you have the appropriate compiler available. Supporting installation from Windows binary installers is on the roadmap, but it's not clear when it will happen.
Until recently, pip's Windows support was flaky and untested. Thanks to a lot of work from Dave Abrahams, pip trunk now passes all its tests on Windows (and there's a continuous integration server helping us ensure it stays that way), but a release has not yet been made including that work. So more reliable Windows support should be coming with the next release.
All the standard Python package installation mechanisms store all metadata about installed distributions in a file or files next to the actual installed package(s). Distutils uses a distribution_name-X.X-pyX.X.egg-info file, pip uses a similarly-named directory with multiple metadata files in it. Easy_install puts all the installed Python code for a distribution inside its own zipfile or directory, and places an EGG-INFO directory inside that directory with metadata in it. If you import a Python package from the interactive prompt, check the value of package.__file__; you should find the metadata for that package's distribution nearby.
Info about installed distributions is only stored in any kind of global registry by OS-specific packaging tools such as Windows installers, Apt, or RPM. The standard Python packaging tools don't modify or pay attention to these listings.
Pip (or, in my opinion, any Python packaging tool) is best used with virtualenv, which allows you to create isolated per-project Python mini-environments into which you can install packages without affecting your overall system. Every new virtualenv automatically comes with pip installed in it.
A couple other projects you may want to be aware of as well (yes, there's more!):
distribute is a fork of setuptools which has some additional bugfixes and features.
distutils2 is intended to be the "next generation" of Python packaging. It is (hopefully) adopting the best features of distutils/setuptools/distribute/pip. It is being developed independently and is not ready for use yet, but eventually should replace distutils in the Python standard library and become the de facto Python packaging solution.
Hope all that helped clarify something! Good luck.
I use windows and python. It is somewhat frustrating, because pip doesn't always work to install things. Python is moving to pip, so I still use it. Pip is nice, because you can uninstall items and use
pip freeze > requirements.txt
pip install -r requirements.txt
Another reason I like pip is for virtual environments like venv with python 3.4. I have found venv a lot easier to use on windows than virtualenv.
If you cannot install a package you have to find the binary for it. http://www.lfd.uci.edu/~gohlke/pythonlibs/
I have found these binaries to be very useful.
Pip is trying to make something called a wheel for binary installations.
pip install wheel
wheel convert path\to\binary.exe
pip install converted_wheel.whl
You will also have to do this for any required libraries that do not install and are required for that package.
I'm just learning the art of writing a setup.py file for my project. I see there's lots of talk about setuptools, which is supposed to be superior to distutils. There's one thing though that I fail to understand, and I didn't see it addressed in any tutorial I've read about this: What if setuptools isn't installed? I understand it's not part of the standard library, so how can you assume the person who wants to install your program will have it installed?
The standard way to distribute packages with setuptools includes an ez_setup.py script which will automatically download and install setuptools itself - on Windows I believe it will actually install an executable for easy_install. You can get this from the standard setuptools/easy_install distribution.
In most librarys I ever installed for python, a warning apears "You have to install setuptools". You could do it as well I think, you could add a link so the user don't have to search the internet for it.
I have used setuptools to compile many python scripts that I have written into windows EXEs. However, it has always been my understanding (from experience) that the computer running the compiled EXE does not need to have setup tools installed.
Hope that helps
You can't assume it's installed. There are ways around that, you can fall back to distutils (but then why have setuptools in the first place) or you can install setuptools in setup.py (but I think that's evil).
Use setuptools only if you need it.
When it comes to setuptools vs distrubute, they are compatible, and choosing one over the other is mainly up to the user. The setup.py is identical.
I would say it depends on what kind of user you are addressing.
If they are simply users and not Python programmers, or if they are basic programmers, using setuptools might be a little bit too much at first. For those the distutils is perfect.
For clients, I would definitely stick to distutils.
For more enthusiast programmers the setuptools would be fine.
Somehow, it also depends on how you want to distribute updates, and how often. For example, do the users have an access to the Internet without a nasty proxy setup by their company that would block setuptools? - We do have one and it's an extra step to configure and make it work on every workstation.
You can download Windows EXE installers and a Linux RPM from here
http://pypi.python.org/pypi/setuptools
Then, once you have setuptools in place you can use the easy_install command to both download and install new packages. Because easy_install, also automatically downloads and installs dependencies, you might want to set up virtualenv before you actually use it. That way you can decide whether or not you want to install a bunch of packages into your system's default Python install.
Yes, this means that your users will have to have setuptools installed in order for them to use it. Of course, you could take the setuptools installers, rename them, and package them up with like NSIS and distribute that to your users. The fact is, that you have to install something, so if you don't want to put your application in the installer, you can package up setuptools instead.