I installed libsvm in Ubuntu. All I want to do is call from svmutil import * in Python without it giving me an error. However, I see no good documentation on how to configure libsvm to work with python. "Adding libsvm to a path" tells me nothing.
And I don't want to use a package manager. What do I need to do to make libsvm found when running python outside of the original libsvm install folder?
(ykp) yannik#yannik-Inspiron-7580:~/Projects/libsvm$ ls
COPYRIGHT java python svm.o svm-toy
description-pak libsvm.so.2 README svm-predict svm-train
doc-pak Makefile svm.cpp svm-predict.c svm-train.c
FAQ.html Makefile.win svm.def svm-scale tools
heart_scale matlab svm.h svm-scale.c windows
(ykp) yannik#yannik-Inspiron-7580:~/Projects/libsvm$ ls python
commonutil.py Makefile README svmutil.py
description-pak __pycache__ svm.py
I tried doing sudo ln -s libsvm.so.2 /usr/local/lib and adding ~/Projects/libsvm to my PYTHONPATH with no success. Please help!
If ~/Projects/libsvm/python/ is where the module you want to import exists, then you have to add ~/Projects/libsvm/python/ to your PYTHONPATH not ~/Projects/libsvm/. The PYTHONPATH doesn't just automatically recursively search down for python files if that is what you were expecting.
How can I (hermetically) include python as an (executable) input to my genrule?
Conceptually, I'm aware of the following approaches:
include a python interpreter in the source repo at third_party/python
have Bazel fetch python ala rules-go
There also seem to be a couple methods for doing so:
py_runtime
py_toolchain (doesn't seem to be available yet)
py_distribution_package (doesn't seem to be available yet)
genrules.tools
genrules.toolchains
Example:
I have a python library:
myproject
- setup.py
- mylibrary
- __init__.py
- myfunction.py
I'd like to produce a pip install-able distribution with Bazel so that I can do:
pip install <path/to/distribution>
python
>>> from mylibrary.myfunction import helloworld
>>> helloworld()
I added a (empty) WORKSPACE file at myproject/WORKSPACE and followed the example in this medium article:
# myproject/BUILD
genrule(
name = "mylibrary_sdist",
srcs = glob(["**/*.py"]),
outs = ["mylibrary.tar.gz"],
cmd = "python setup.py sdist && mv dist/mylibrary*.tar.gz $(location mylibrary.tar.gz)"
)
This works (ie. produces bazel-genfiles/mylibrary.tar.gz), but I'm shelling out to python.
How can I (hermetically) include python as an (executable) input to my genrule?
Unless I'm misunderstanding your question, this should just be a matter of passing the actual path to your target python executable. I would check the python executable into the third_party dir and instead of invoking your genrule using simply python I'd pass the relative path to that executable relative to WORKSPACE.
I currently have a PyInstaller packed Elf file and I'm looking to unpack it into the original .py file(s). I have been using PyInstaller Extractor but it appears to be telling the archive is not a PyInstaller archive.
Here is an example of what I've been doing:
$ cat main.py
#! /usr/bin/python3
print ("Hello %s" % ("World"))
I pack it in the file dist/main/main with the command:
pyinstaller main.py
Which outputs the file:
$ file dist/main/main
dist/main/main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=373ec5dee826653796e927ac3d65c9a8ec7db9da, stripped
Now, when I want to unpack it:
$ python pyinstxtractor.py dist/main/main
[*] Processing dist/main/main
[*] Error : Unsupported pyinstaller version or not a pyinstaller archive
I don't understand why the file cannot be unpacked while I've been looking through many posts telling that this should be possible and I'm beginning to doubt it.
Is the unpacking of the ELF file actually possible?
Am I doing it the right away?
According to the Github page, this script is applicable only for Windows binaries. There is an archive_viewer.py script distributed with pyinstaller itself that allows to view binary contents and extract it. If you get a .pyz file after extraction, use archive_viewer.py on it again. IIRC, after all you will get .pyc files, which have to be decompiled.
On my system (Manjaro Linux) I've found this script at /lib/python3.6/site-packages/PyInstaller/utils/cliutils
It is also available as pyi-archive_viewer (at /usr/bin/pyi-archive_viewer) after installing to global interpreter.
Using pyi-archive_viewer CLI seems to be the supported solution, i.e. to print only the module names, recursively, and quit instead of prompting:
$ pyi-archive_viewer --log --recursive --brief build/PYZ-00.pyz
['__future__',
'_aix_support',
---SNIP---
'zipfile',
'zipimport']
But if you don't want to parse or unsafely eval() the CLI output, it seems to work to use the library directly:
from PyInstaller.utils.cliutils import archive_viewer
archive = archive_viewer.get_archive('build/PYZ-00.pyz')
output = []
archive_viewer.get_content(archive, recursive=True, brief=True, output=output)
# Now, output is ['__future__', '_aix_support', ---SNIP--- 'zipfile', 'zipimport']
This use of the library is undocumented, but it's essentially the same to what the CLI does given those flags.
Is there any way to get pip to print the config it will attempt to use? For debugging purposes it would be very nice to know that:
config.ini files are in the correct place and pip is finding them.
The precedence of the config settings is treated in the way one would expect from the docs
For 10.0.x and higher
There is new pip config command, to list current configuration values
pip config list
(As pointed by #wmaddox in comments) To get the list of where pip looks for config files
pip config list -v
Pre 10.0.x
You can start python console and do. (If you have virtaulenv don't forget to activate it first)
from pip import create_main_parser
parser = create_main_parser()
# print all config files that it will try to read
print(parser.files)
# reads parser files that are actually found and prints their names
print(parser.config.read(parser.files))
create_main_parser is function that creates parser which pip uses to read params from command line(optparse) and loading configs(configparser)
Possible file names for configurations are generated in get_config_files. Including PIP_CONFIG_FILE environment variable if it set.
parser.config is instance of RawConfigParser so all generated file names in get_config_files are passed to parser.config.read
.
Attempt to read and parse a list of filenames, returning a list of filenames which were successfully parsed. If filenames is a string, it is treated as a single filename. If a file named in filenames cannot be opened, that file will be ignored. This is designed so that you can specify a list of potential configuration file locations (for example, the current directory, the user’s home directory, and some system-wide directory), and all existing configuration files in the list will be read. If none of the named files exist, the ConfigParser instance will contain an empty dataset. An application which requires initial values to be loaded from a file should load the required file or files using read_file() before calling read() for any optional files:
From how I see it, your question can be interpreted in three ways:
What is the configuration of the pip executable?
There is a quite extensive documentation for the configurations supported by pip, see here: https://pip.pypa.io/en/stable/user_guide/#configuration
What is the configuration that pip uses when configuring and subsequently building code required by a Python module?
This is specified by the package that is being installed. The package maintainer is responsible for producing a configuration script. For example, Numpy has a Configuration class (https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py) that they use to configure their Cython build.
What are the current modules installed with pip so I can reproduce a specific environment configuration?
This is easy, pip freeze > requirements.txt. This will produce a file of all currently installed pip modules along with their exact versions. You can then do pip install -r requirements.txt to reproduce that exact environment configuration on another machine.
I hope this helps.
You can run pip in pdb. Here's an example inside ipython:
>>> import pip
>>> import pdb
>>> pdb.run("pip.main()", globals())
(Pdb) s
--Call--
> /usr/lib/python3.5/site-packages/pip/__init__.py(197)main()
-> def main(args=None):
(Pdb) b /usr/lib/python3.5/site-packages/pip/baseparser.py:146
Breakpoint 1 at /usr/lib/python3.5/site-packages/pip/baseparser.py:146
(Pdb) c
> /usr/lib/python3.5/site-packages/pip/baseparser.py(146)__init__()
-> if self.files:
(Pdb) p self.files
['/etc/xdg/pip/pip.conf', '/etc/pip.conf', '/home/andre/.pip/pip.conf', '/home/andre/.config/pip/pip.conf']
The only trick here was looking up the path of the baseparser (and knowing that the files are in there). If you don't know this already you can simply step through the program or read the source. This type of exploration should work for most Python programs.
How do I learn where the source file for a given Python module is installed? Is the method different on Windows than on Linux?
I'm trying to look for the source of the datetime module in particular, but I'm interested in a more general answer as well.
For a pure python module you can find the source by looking at themodule.__file__.
The datetime module, however, is written in C, and therefore datetime.__file__ points to a .so file (there is no datetime.__file__ on Windows), and therefore, you can't see the source.
If you download a python source tarball and extract it, the modules' code can be found in the Modules subdirectory.
For example, if you want to find the datetime code for python 2.6, you can look at
Python-2.6/Modules/datetimemodule.c
You can also find the latest version of this file on github on the web at
https://github.com/python/cpython/blob/main/Modules/_datetimemodule.c
Running python -v from the command line should tell you what is being imported and from where. This works for me on Windows and Mac OS X.
C:\>python -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# C:\Python24\lib\site.pyc has bad mtime
import site # from C:\Python24\lib\site.py
# wrote C:\Python24\lib\site.pyc
# C:\Python24\lib\os.pyc has bad mtime
import os # from C:\Python24\lib\os.py
# wrote C:\Python24\lib\os.pyc
import nt # builtin
# C:\Python24\lib\ntpath.pyc has bad mtime
...
I'm not sure what those bad mtime's are on my install!
I realize this answer is 4 years late, but the existing answers are misleading people.
The right way to do this is never __file__, or trying to walk through sys.path and search for yourself, etc. (unless you need to be backward compatible beyond 2.1).
It's the inspect module—in particular, getfile or getsourcefile.
Unless you want to learn and implement the rules (which are documented, but painful, for CPython 2.x, and not documented at all for other implementations, or 3.x) for mapping .pyc to .py files; dealing with .zip archives, eggs, and module packages; trying different ways to get the path to .so/.pyd files that don't support __file__; figuring out what Jython/IronPython/PyPy do; etc. In which case, go for it.
Meanwhile, every Python version's source from 2.0+ is available online at http://hg.python.org/cpython/file/X.Y/ (e.g., 2.7 or 3.3). So, once you discover that inspect.getfile(datetime) is a .so or .pyd file like /usr/local/lib/python2.7/lib-dynload/datetime.so, you can look it up inside the Modules directory. Strictly speaking, there's no way to be sure of which file defines which module, but nearly all of them are either foo.c or foomodule.c, so it shouldn't be hard to guess that datetimemodule.c is what you want.
If you're using pip to install your modules, just pip show $module the location is returned.
The sys.path list contains the list of directories which will be searched for modules at runtime:
python -v
>>> import sys
>>> sys.path
['', '/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', ... ]
from the standard library try imp.find_module
>>> import imp
>>> imp.find_module('fontTools')
(None, 'C:\\Python27\\lib\\site-packages\\FontTools\\fontTools', ('', '', 5))
>>> imp.find_module('datetime')
(None, 'datetime', ('', '', 6))
datetime is a builtin module, so there is no (Python) source file.
For modules coming from .py (or .pyc) files, you can use mymodule.__file__, e.g.
> import random
> random.__file__
'C:\\Python25\\lib\\random.pyc'
Here's a one-liner to get the filename for a module, suitable for shell aliasing:
echo 'import sys; t=__import__(sys.argv[1],fromlist=[\".\"]); print(t.__file__)' | python -
Set up as an alias:
alias getpmpath="echo 'import sys; t=__import__(sys.argv[1],fromlist=[\".\"]); print(t.__file__)' | python - "
To use:
$ getpmpath twisted
/usr/lib64/python2.6/site-packages/twisted/__init__.pyc
$ getpmpath twisted.web
/usr/lib64/python2.6/site-packages/twisted/web/__init__.pyc
In the python interpreter you could import the particular module and then type help(module). This gives details such as Name, File, Module Docs, Description et al.
Ex:
import os
help(os)
Help on module os:
NAME
os - OS routines for Mac, NT, or Posix depending on what system we're on.
FILE
/usr/lib/python2.6/os.py
MODULE DOCS
http://docs.python.org/library/os
DESCRIPTION
This exports:
- all functions from posix, nt, os2, or ce, e.g. unlink, stat, etc.
- os.path is one of the modules posixpath, or ntpath
- os.name is 'posix', 'nt', 'os2', 'ce' or 'riscos'
et al
On windows you can find the location of the python module as shown below:i.e find rest_framework module
New in Python 3.2, you can now use e.g. code_info() from the dis module:
http://docs.python.org/dev/whatsnew/3.2.html#dis
Check out this nifty "cdp" command to cd to the directory containing the source for the indicated Python module:
cdp () {
cd "$(python -c "import os.path as _, ${1}; \
print _.dirname(_.realpath(${1}.__file__[:-1]))"
)"
}
Just updating the answer in case anyone needs it now, I'm at Python 3.9 and using Pip to manage packages. Just use pip show, e.g.:
pip show numpy
It will give you all the details with the location of where pip is storing all your other packages.
On Ubuntu 12.04, for example numpy package for python2, can be found at:
/usr/lib/python2.7/dist-packages/numpy
Of course, this is not generic answer
Another way to check if you have multiple python versions installed, from the terminal.
$ python3 -m pip show pyperclip
Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
$ python -m pip show pyperclip
Location: /Users/umeshvuyyuru/Library/Python/2.7/lib/python/site-packages
Not all python modules are written in python. Datetime happens to be one of them that is not, and (on linux) is datetime.so.
You would have to download the source code to the python standard library to get at it.
For those who prefer a GUI solution: if you're using a gui such as Spyder (part of the Anaconda installation) you can just right-click the module name (such as "csv" in "import csv") and select "go to definition" - this will open the file, but also on the top you can see the exact file location ("C:....csv.py")
If you are not using interpreter then you can run the code below:
import site
print (site.getsitepackages())
Output:
['C:\\Users\\<your username>\\AppData\\Local\\Programs\\Python\\Python37', 'C:\\Users\\<your username>\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages']
The second element in Array will be your package location. In this case:
C:\Users\<your username>\AppData\Local\Programs\Python\Python37\lib\site-packages
In an IDE like Spyder, import the module and then run the module individually.
enter image description here
as written above
in python just use help(module)
ie
import fractions
help(fractions)
if your module, in the example fractions, is installed then it will tell you location and info about it, if its not installed it says module not available
if its not available it doesn't come by default with python in which case you can check where you found it for download info