I am very confused, how to handle python's modules. There are multiple ways of installing packages. I am currently using three ways.
1) Packages of the linux distribution
Currently, many of the most popular modules/packets, like ipython, it is possible to install them via the distribution package manager. This leaves me with a system wide access of the package. I don't have to do anything about my PATHs, user access rights. It just work and until now, it was my most favorite method.
2) Pip or conda
As started to use packages, which are not that famous, they don't have a distribution package. So i have to obtain them elsewhere. Until now, if i couldn't find it in my distribution, i could just use pip to install any other necessary package. Conda is another option though.
My Question:
What is the "best" approach. I mean all possibilities do exactly the same, in the view of the user. I would like to use one, so i don't get confused, however as i stated, not everything is available everywhere. I am forced to use all of them right now. This is very annoying, especially in terms of update, conflicts, user rights and access, path variables, not even mentioning python 2.7 and 3.x "war".
So how do you do it? How do you maintain a system (i.e. a desktop) with so many different package managers. Do you stick do pip/conda only, installing everything for yourself(i.e. your home directory)?
The Python documentation recommends pip for installing python models:
pip is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers.
For a more complete overview, you can check out the documentation on installing modules.
In terms of dealing with the conflicts you've mentioned, you should be using virtual environments, either with pyenv or virtualenv. Virtual environments allow you to use different modules or versions of modules for different projects. Using virtual environments also allows you to replicate that environment elsewhere, for instance, on a server.
Related
I am using jedi and more specifically deoplete-jedi in neovim and I wonder if I should install it in every project as a dependency or if I can let jedi reside in the same python environment as neovim uses (and set the setting to tell deoplete-jedi where to look)
It seems wasteful to have to install it in ever project but then again IDK how it would find my project environment from within the neovim environment either.
If by the word "project"you mean Python virtual environments then yes, you have to install every program and every library that you use to every virtualenv separately. flake8, pytest, jedi, whatever. Python virtual environments are intended to protect one set of libraries from the other so that you could install different sets of libraries and even different versions of libraries. The price is that you have to duplicate programs/libraries that are used often.
There is a way to connect a virtualenv to the globally installed packages but IMO that brings more harm than good.
Our command line utility is written with Python.
On Linux/OS X this is usually not a problem since both come with Python 2.x pre installed. However on Windows, Python isn't installed by default.
Additional problem is that few of our dependencies require compiling which yet again is not a trivial problem for Windows users since it requires tinkering with MSVC/Cygwin/etc'.
Up until now we solved this issue by using Pyinstaller to create "frozen" Python package with pre-installed dependencies. This worked well, however made our utility non extendable - we cannot add additional Python modules by using utilities such as pip for example. Since our CLI depends on this ability in order to add additional usability, this limitation became a blocker for us and we would like to rethink our approach.
Searching around, I found how Rhodecode solve this. Basicly their installer brings Python and everything else (including pre-compiled dependencies).
This seem as a good idea for us, the only limitation I see here is that their installer actually installs Python from .msi which puts stuff in Windows Registry. So there can be only one Python of version X.Y installed on Windows (from .msi)
For a server application this might be reasonable since server application tends to act like it's the only thing installed on the PC, but for command line utility, this is completely unacceptable.
Looking around I found few projects that claim to make Python portable - for example Portable Python. However I'm not sure how "portable" it really is, especially after issues like this.
So questions are:
Is it possible to install same Python version multiple times on Windows without creating collisions between the instances?
Would you choose other workaround to solving this problem (please no "smart" solutions such as: drop Windows support/don't use Python)
Thanks!
Frankly I would stick with PyInstaller or something similar. That will always provide you with the correct version of Python whether or not the target machine has Python installed. It also protects you from clobbering a previously installed version of Python.
If you need to add plugins, then you should build that into your app. There are a few projects that might help you with that. Here are a couple of examples:
http://yapsy.sourceforge.net/
https://pypi.python.org/pypi/Plugins/
You might also take a look at how Django or Flask handles extensions. For example, you can add an extension to Flask to allow it to work with SQLAlchemy. You should be able to do something similar with your own application. The pip utility won't work with a frozen application after all. Alternatively, have you looked at conda? It might work for your purposes.
This is an answer for my second question, sadly I still haven't figured out a better solution for number 1.
For now here's how we changed our approach for creating setup file:
We package our code and its dependencies as set of pre-built Python wheels. It's relatively easy to create pre-built wheels on Windows since the release of Visual C++ compiler for Python2.7.
We package Python setup MSI together with Pip, Setuptools and Virtualenv wheels.
Before install starts we check whether Python, Pip and Virtualenv are already installed (by looking in the registry and \Scripts), if it's not, we install it from the packaged wheels. Pip wheel is installed by using get-pip.py script which we bundle as well.
We create separate Virtualenv and install our wheels into it.
Uninstall is done by removing the virtualenv folder. Python, Pip and Virtualenv install are left behind.
Installer created by Inno Setup.
I'm still not fully satisfied with this solution since some components are globally installed and might collide with what user had already installed before (older/newer version of python, pip, setuptools or virtualenv). This creates potential for unpredictable bugs during install or runtime. Or the possibility for user to upgrade one of the components in the future and somehow break the application.
Also, uninstall is dirty and leaves stuff behind.
I have a virtualenv running python 2.7.7. It has a pretty extensive set of libraries which support a pretty complicated set of proprietary modules. In other words, the virtualenv needs to maintain its integrity. That is of course the whole point of virtualenv.
Recently, I encountered a number of problems that are very easily solved by using Anaconda. I tried it out in a test environment and it worked quite well. Now I'm tasked with incorporating this new configuration into production. It isn't clear to me how to incorporate Anaconda into a virtualenv, or whether this is even a good idea. In fact, it almost seems to me like I should use the anaconda install as the new source and desconstruct the old virtualenv... merging the libraries it held into the conda.
Does anyone have a recommendation as to the best approach? If merging the environments is called for, can anyone point to an explanation of how to go about it?
It doesn't really make sense to merge Anaconda and a virtualenv, as Anaconda is a completely independent installation of Python. You can do it, typically by setting your PYTHONPATH, but things have a good chance of breaking when you do this sort of thing, and I would recommend against it.
If there are libraries in your virtualenv, you can use them with Anaconda by making conda packages for them. They may already have conda packages (search with conda search and search https://binstar.org/). Otherwise, you can build a package using a conda recipe. See http://conda.pydata.org/docs/build.html and https://github.com/conda/conda-recipes for some example recipes.
I downloaded numpy1.9.0 but there is no EXE file in the package, how should I install it?
Numpy must be compiled for your system. You'll have to use a compiler to compile it if you want to use it from a download of the source code.
For compatibility with your system and other related or dependant packages, I recommend using a prepackaged distribution. (I get most of the Python data stack on Ubuntu with sudo apt-get install python-statsmodels.) For people with Windows who come to our Python meetup group and who have trouble installing things, we typically recommend getting the Python data stack distribution, Anaconda, from Continuum Analytics which will include Numpy. This link should take you to the versions for Python 3.4.
I would much rather you get everything you're likely to need at once as opposed to installing piecemeal different packages from various sources that may not work well together. If you just install Numpy, you're likely to want other packages that depend on it later, which is likely to give you problems if you're not using a package manager. I am unaware of a package manager for Windows, and it's probably out of scope to look for one.
(Disclaimer, Continuum Analytics sometimes sponsors and provides speakers for our meetup group, but I'm just a volunteer, and otherwise unconnnected to them.)
There are many versions of Python, and it becomes difficult to manage them all.
Often I need to install one module into 3 different versions of Python.
Is there a tool which can simplify things?
I'm on Windows.
Thanks.
Are you using virtualenv? If not, you definitely want to check that out: http://pypi.python.org/pypi/virtualenv
It helps you by managing and switching between several virtual Python environments, with different versions of Python if you want to.
There are loads of tutorials of how to set it up, all over dem interweb.
What Legogris said: use virtualenv.
I just answered a question on pip, virtualenv, and virtualenvwrapper applicable here. I highly recommend this combination of tools for maintaining isolated python environments.
As a further point, I strong recommend using the no-site-packages option so that each virtualenv has all its requirements in one place.
Because some modules contain binary code which is linked agains a specific Python version, it will not be possible to instal a module only once. You will always have to install it for each installed version. But if you use pip, you should have a look at pip: dealing with multiple Python versions? Just create a batch file which calls pip for each installed version. That should at least simplify your life.
I'm not aware of any Python facility for doing that, that's really the OS's job. Debian/Ubuntu, for example, has support for installing multiple versions of Python and installing libraries into each version. I doubt there's any such support in Windows.