How can I check is some package is installed in my system. My system is Linux, but even better if it could works in other OSs. I mean OS specific package (like could be *.rpm or *.deb).
Is there any python module or script that could do it?
To find out whether you've installed a .deb, .rpm, etc. package, you need to use the appropriate tools for your packaging system.
APT has a Python wrapper named python-apt in Debian, or just apt at PyPI.
RPM has a whole slew of Python tools—in fact, most of Redhat's installer ecosystem is built on Python, and you should already have the rpm module installed. Read Programming RPM with Python (or, better, search for a newer version…) before looking for a high-level wrapper, so you understand what you're actually doing; it's only a couple lines of code even with the low-level interface.
As far as I know, nobody has wrapped these up in a universal tool for every packaging format and database that any linux distro has ever used (and, even if they had, that wouldn't do you much good on linux systems that don't use a packaging system). But if you just want to handle a handful of popular systems, python-apt and either Redhat's own tools or search PyPI for RPM, and that will cover almost everything you care about.
Alternatively, pkg-config is the closest thing to a universal notion of "packages installed on this system". Every linux system will have it (and most other non-Windows systems), but not every package registers with pkg-config. Still, if this is what you're looking for, pkgconfig is the Python answer.
The word "package" has a half-dozen similar but incompatible meanings, but the fact that you said "package or module" implies you specifically want to know about Python packages and modules, as in the things you can import.
In which case, the way to test it is to import them.
Manually, do this:
$ python
>>> import foo
ImportError: No module named foo
Well, foo isn't installed.
Programmatically:
try:
import foo
except ImportError:
# do whatever you wanted if foo is missing
Note that this doesn't actually tell you foo is missing, just that it couldn't be imported. In a simple "test whether you have this" script, that's generally what you want to actually check for. But what if you really want to check "is installed (even if broken)"?
In recent Python (I think 3.4+), the ImportError will have additional information in it that you can access—name for the name you were trying to import, path if it was found, etc. However, this is one of those cases where EAFP may not be better than LBYL. You can use importlib to search for the module without trying to import it, like this:
spec = importlib.util.find_spec('foo')
What if you're using an older Python? There are similar features going back to 3.2, but not quite as nice, and if you're using 2.7, there's really nothing worth using, because the import machinery wasn't exposed very well.
For that case (and many, many other cool things related to package installation), use setuptools—which isn't in the stdlib, but a huge number of third-party packages depend on it (until recently it was the cornerstone of Python package installation, even if unofficially):
pkg_resources.get_distribution('foo')
However, that looks for a distutils/setuptools/PyPI package, not a Python module or package. There's a lot of overlap there, but they're not exactly the same thing. For a simple example, when you pip install more-itertools, you get the more-itertools PyPI package, which installs the more_itertools Python package into your site-packages.
Related
How should a file myModule.cpython-35m-x86_64-linux-gnu.so be imported in python? Is it possible?
I tried the regular way:
import myModule
and the interpreter says:
`ModuleNotFoundError: No module named 'myModule'`
This is a software that I can't install in the cluster that I am working at so I just extracted the .deb package and it does not have a wheel file or structure to install.
It is problematic to use a C-extension built for one Python version in another Python version. Normally (at least for Python3) there is a mechanism in place to differentiate C-extensions for different Python versions, so they can co-exist in the same directory.
In your example, the suffix is cpython-35m-x86_64-linux-gnu so this C-extension will be picked up by a CPython3.5 on a x86_64 Linux. If you try to import this extension with another Python-version or on another plattform, the module isn't visible and ModuleNotFoundError is raised.
It is possible to see, which suffixes are accepted by the current Python version, e.g. via:
>>> import _imp
>>>_imp.extension_suffixes()
['.cpython-36m-x86_64-linux-gnu.so', '.abi3.so', '.so']
A possibility is to use the stable C-API which could be used with multiple Python versions without recompilation. Cython start to support it in version 3.0 (see this PR), see also this SO-post about setuptools and stable C-API.
One might want to be clever and rename the extension to simple .so, so it can be picked up by the Finder - this can/does work for some Python-version combinations on some platforms for some extension - yet this approach cannot be sustained in the long run and is not the right thing to do.
The right thing to do, is to build the C-extension for/with the right Python-version on the right OS/platform or to use the right wheel (or use stable C-API).
In general, a C-extension built for a python-version (let's say PythonA.B) cannot be used by another Python version (let's say PythonC.D), because those extensions/modules are linked against a special Python-library and the needed functionality might no longer/not yet be present in the library of another version.
This different to *.py-files and more similar to *.pyc-files which cannot be used with a different version.
While PEP-3147 regulates the suffices of *.pyc-files, PEP-3149 does the same for the C-extensions. PEP-3149 is however not the state-of-the-art, as some of the problems where fixed only in Python3.5, the whole discussion can be found here.
I'm using setuptools to install a Python module that I'm working on. In addition to numpy, scipy, ..., whose presence I can assure with install_requires = [...], my module also depends on a Python module - let's call it specialmodule - that is a Python interface to a program that is neither an egg, nor a single .py-file or a VCS repo (so Dependencies that aren’t in PyPI is not applicable). The program is written in C++ and has a Python interface, and can either be built from source after cloning from git, or obtained as a tar archive.
Is there a way to use setuptools to check the existence of this module (which is in PYTHONPATH), and if it can not be found, display some message to the user that the module is missing (and if possible, also some instructions on how to get it)?
Edit: Also, if there is a more elegant way to do this with a different approach than with setuptools, I'd be glad to hear! But I would really like to check directly on installation, not during runtime of my module.
I'm in desperate need of a cross platform framework as I have vast numbers of .NET products that I'm trying to port to Linux. I have started to work with Python/pyQt and the standard library and all was going well until I try to import non-standard libraries. I'm hearing about pip and easy_install and I'm completely confused about this.
My products need to ship with everything required to execute them, so in the .NET world I simply package my DLLs (or licensed DLLs) with my product.
As a test bed I'm trying to import this library called requests: https://github.com/kennethreitz/requests
I've got an __init__.py file and the library source in my program directory but it isn't working. Please tell me that there is a simple way to include libraries without needing any kind of extra package installer.
I would suggest you start by familiarizing yourself with python packages (see the distutils docs. Pip is simply a manager that install packages directly from the internet repository, so that you don't need to manually go and download them. So for, example, as stated under "Installing" on the requests homepage, you simply run pip install requests in a terminal, without manually downloading anything.
Packaging your product is a different story, and the way you do it depends on the target system. On windows, the easiest might be to create an installer using NSIS which will install all dependencies. You might also want to use cx-freeze to pull all the dependencies (including the python interpreter) into a single package.
On linux, many of the dependencies will already be including in most distributions. so you should just list them as requirements when creating your package (e.g. deb for ubuntu). Other dependencies might not be included in the distro's repo, but you can still list them as requirements in setup.py.
I can't really comment on Mac, since I've never used python on one, but I think that it would be similar to the linux approach.
I have some small Python programs which depend on several big libraries, such as:
NumPy & SciPy
matplotlib
PyQt
OpenCV
PIL
I'd like to make it easier to install these programs for Windows users. Currently I have two options:
either create huge executable bundles with PyInstaller, py2exe or similar tool,
or write step-by-step manual installation instructions.
Executable bundles are way too big. I always feel like there is some magic happening, which may or may not work the next time I use a different library or a new library version. I dislike wasted space too. Manual installation is too easy to do wrong, there are too many steps: download this particular interpreter version, download numpy, scipy, pyqt, pil binaries, make sure they all are built for the same python version and the same platform, install one after another, download and unpack OpenCV, copy its .pyd file deep inside Python installation, setup environment variables and file asssociations... You see, few users will have the patience and self-confidence to do all this.
What I'd like to do: distribute only a small Python source and, probably, an installation script, which fetches and installs all the missing dependencies (correct versions, correct platform, installs them in the right order). That's a trivial task with any Linux package manager, but I just don't know which tools can accomplish it on Windows.
Are there simple tools which can generate Windows installers from a list of URLs of dependencies1?
1 As you may have noticed, most of the libraries I listed are not installable with pip/easy_install, but require to run their own installers and modify some files and environment variables.
npackd exists http://code.google.com/p/windows-package-manager/ It could be done through here or use distribute (python 3.x) or setuptools (python 2.x) with easy_install, possibly pip (don't know it's windows compatibility). But I would choose npackd because PyQt and it's unusual setup for pip/easy_install (doesn't play with them nicely, using a configure.py instead of setup.py). Though you would have to create your own repo for npackd to use for some of them. I forget what is contributed in total for python libs with it.
AFAIK there is no tool (and I'd assume you googled), so you must make one yourself.
Fetching the proper library versions seems simple enough -- using python's ftplib you can fetch the proper installers for every library. How would you know which version is compatible with the user's python? You can store different lists of download URLs, each for a different python version (this method came off the top of my head and there is probably a better way; not that it matters much if it's simple and it works).
After you figure out how to make each installer run, you can py2exe your installer script, and even use it to fetch the program itself.
EDIT
Some Considerations
There are a couple of things that popped into my mind just as I posted:
First, some pseudocode (how I would approach it, anyway)
#first, we check modules
try:
import numpy
except ImportError:
#flag numpy for installation
#lather, rinse repeat for all dependencies
#next we check version compatibility -- note that if a library version you need
#is not backwards-compatible, you're in DLL hell, and there is little we can do.
<insert version-checking code here>
#once you have your unavailable dependencies, you install them
import ftplib
<all your file-downloading here>
#now you install. sorry I can't help you here.
There are a few things you can do to make your utility reusable --
put all URL lists, minimum version numbers, required library names etc in config files
Write a script which helps you set up an installer
Py2exe the installer-maker-script
Sell it
Even better, release it under GPL so we can all feast upon fruits of your labours.
I have a similar need as you, but in addition I need the packaged application to work on several platforms. I'm currently exploring the currently available solutions, here are a few interesting ones:
Use SnakeBasket, which wraps around Pip and add a recursive dependency resolution plus a heuristic to choose the right version when there are conflicts.
Package all dependencies as an egg, but not your sourcecode which will still be editable: https://stackoverflow.com/a/528064/1121352
Package all dependencies in a zip file and directly import the modules on the fly: Cross-platform alternative to py2exe or http://davidf.sjsoft.com/mirrors/mcmillan-inc/install1.html
Using buildout: http://www.buildout.org/en/latest/install.html
Using virtualenv with virtualenv-tools (instead of "relocate")
If your main problem when freezing your code using PyInstaller or similar is that you end up with a big single file, you can customize the process so that you get several files, one for each dependency, instead of one big executable.
I will update here if I find something that fills my bill.
With PHP you have the phpinfo() which lists installed modules and then from there look up what they do.
Is there a way to see what packages/modules are installed to import?
Type help() in the interpreter
then
To get a list of available modules, keywords, or topics, type "modules",
"keywords", or "topics". Each module also comes with a one-line summary
of what it does; to list the modules whose summaries contain a given word
such as "spam", type "modules spam".
help> modules
If you use ipython, which is an improved interactive Python shell (aka "REPL"), you can type import (note the space at the end) followed by a press of the [TAB] key to get a list of importable modules.
As noted in this SO post, you will have to reset its hash of modules after installing (certain?) new ones. You likely don't need to worry about this yet.
If you don't use ipython, and you haven't tried it, it might be worth checking out. It's a lot better than the basic Python shell, or pretty much any other REPL I've used.
ipython Installation
If you're running linux, there is most likely an ipython package that you can install through your system management tools. Others will want to follow these instructions.
If your installation route requires you to use easy_install, you may want to consider instead using pip. pip is a bit smarter than easy_install and does a better job of keeping track of file locations. This is very helpful if you end up wanting to uninstall ipython.
Listing packages
Note that the above tip only lists modules. For a list which also includes packages —which contain modules— you can do from + [TAB]. An explanation of the difference between packages and modules can be found in the Modules chapter of the helpful official Python tutorial.
#rtfm
As an added note, if you are very new to python, your time may be better spent browsing the standard library documentation than by just selecting modules based on their name. Python's core documentation is well-written and well-organized. The organizational groups —File and Directory Access, Data Types, etc.— used in the library documentation's table of contents are not readily apparent from the module/package names, and are not really used elsewhere, but serve as a valuable learning aid.
This was very useful. Here is a script version of this:
# To list all installed packages just execfile THIS file
# execfile('list_all_pkgs.py')
for dist in __import__('pkg_resources').working_set:
print dist.project_name.replace('Python', '')
You can list available modules like so:
python -c "for dist in __import__('pkg_resources').working_set:print dist.project_name.replace('Python', '')"
As aaronasterling says, all .py or .pyc files on sys.path is a module because it can be imported. There are scripts that can let you find what external module is installed in site-packages.
Yolk is a Python command-line tool and library for obtaining information about packages installed by setuptools, easy_install and distutils and it can also query pypi packages.
http://tools.assembla.com/yolk/
You may use the pip module:
from pip._internal.operations.freeze import freeze
for line in freeze():
print(line.split('=='))