Develop Reference

Python Pylearn2 package "ImportError: No module named pylearn2.utils"

I have recently tried to use pylearn2, a deep machin learning package for Python developed at University of Montreal.
I've just installed it and tried to run a simple example, but it did not work.
I have been using a pc with an Ubuntu 13.10 system, on which I found ipython installed.
I have installed Theano and later pylearn2, by following this webpage instructions:
http://deeplearning.net/software/pylearn2/
I have also modified the .bashrc file, as suggested
I thought that everything went well, and then I tried this Quick start example:
http://deeplearning.net/software/pylearn2/tutorial/index.html
I stopped at the first command:
python make_dataset.py
My terminal states:
Traceback (most recent call last): File "make_dataset.py", line 14,
in
Do you have any ideas on why it is not working?
Do you why these errors occur?
Thanks a lot
EDIT: the 14 line is the first non-commented line of the file. It states
from pylearn2.utils import serial

Without more information, I can only guess, but my first guess is…
You haven't actually installed pylearn2, because if you follow the linked docs to grab the git repo and add a PYLEARN2_DATA_PATH variable, nothing gets installed into site-packages (or dist-packages or anywhere else on sys.path).
This means that pylearn2 will only work when you start Python from within the top-level directory of the pylearn2 repo.
So, if you run a script like this:
$ cd /path/to/pylearn2
$ cd scripts/tutorials/grbm_smd/
$ python make_dataset.py
… it won't actually work.
It looks like there is a setup.py file in the repository. Does it work? I have no idea. Even though the docs don't mention using it, you might want to try. Either this:
$ pip install .
… or, if you don't have pip or it doesn't work on this package:
$ python setup.py install
Either way, of course, you may need sudo or a flag to install to your user site-packages instead of system, etc., as with any other Python package.
If that doesn't work, you might be able to just add /path/to/pylearn2 to your sys.path in some way. The most obvious way is by doing an export PYTHONPATH=/path/to/pylearn2:$PYTHONPATH in your ~/.bashrc.
Also, you will need to either source ~/.bashrc or create a new shell to get any effects of modifying the file.
If you're wondering why the instructions and the tutorial together don't give you enough information to make this work without a lot of hassle, I think that's covered in the very top of the documentation:
Pylearn2 is still undergoing rapid development. Don’t expect a clean road without bumps!
And the very fact that there is no PyPI download yet implies that this really is not ready for novices to use. If you don't know enough about using Python packages (and bash basics) to muddle through on your own, there's a good chance you won't be able to use this package.

Building Python and more on missing modules

I have another thread asking help on "missing zlib". With the nice help the problem has been resolved (almost).
Now I am interested in building Python myself (on Ubuntu 10.10).
A few important questions have caught my attention:
After building Python (say 2.7.1), do I need to rebuild Python if I have missing modules?
Is there a way to find out what modules will be missing prior to building Python? Say sqlite3. I have sqlite3 installed for the system default (Python 2.6.6), and I can import that into Python 2.6.6 shell. Now I use pythonbrew to build 2.7.1, and in the shell I cannot import sqlite3 because _sqlite3 is not available. I am sure there are a few more important one missing which I need for future development (such as Django..).
I am willing to learn how to build without using pythonbrew.
Please share with me your experience in building another version of Python, and how would you address the problem of missing modules? Is there a practical solution to building Python?
I have never bothered building one myself, so please bear with me. I am beginning to realize the importance of learning and building one myself! Thank you very much!
EDIT
First I thank you all of your inputs. They meant a lot. I did the building.
Python build finished, but the necessary bits to build these modules were not found:
_bsddb _curses _curses_panel
_tkinter bsddb185 bz2
dbm gdbm readline
sunaudiodev _sqlite3
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
I got sqlite3 and readline away by
sudo apt-get install libreadline6 libreadline6-dev
sudo apt-get install libsqlite3-dev
I tried to import them, but still "no named module xxxx".
At AskUbuntu I actually asked people how to get previous commands because I couldn't use that feature when I am in Python 2.7.1 shell. I believe it's due to readline.
Readline
I installed the Python-2.7.1 under this directory: /home/jwxie518/python27/
I looked into setup.py, I found the following lines:
# The sqlite interface
sqlite_setup_debug = False # verbose debug prints from this script?
# We hunt for #define SQLITE_VERSION "n.n.n"
# We need to find >= sqlite version 3.0.8
sqlite_incdir = sqlite_libdir = None
sqlite_inc_paths = [ '/usr/include',
'/usr/include/sqlite',
'/usr/include/sqlite3',
'/usr/local/include',
'/usr/local/include/sqlite',
'/usr/local/include/sqlite3',
]
All the paths listed above do not exist.
So I guess I have to install sqlite3 manually? I got another reference here (it's in Chinese, however)
# Download the latest and extract
# Go into the extracted directory
./configure --prefix=/home/jwxie518/python27/python
make && make install
# Then edit python-2.7 's setup.py before rebuild it
# Sample (add these two lines to the end....)
'~/share/software/python/sqlite-3.6.20/include',
'~/share/software/python/sqlite-3.6.20/include/sqlite3',
# Then rebuild python like how we did before
I went into my directory where I installed sqlite3. I found include/sqlite3.h only. So I went back and check /usr/include/. I can only find sqlite3.h too.
So what is going on here? Readline is also non-importable.
3RD EDIT
I started everything over, except I didn't reinstall sqlite3.
# Extract Python-2.7.1
# cd into Python-2.7.1
# ./configure
make >make.out 2>&1
less make.out
make.out is here: http://pastebin.com/raw.php?i=7k3BfxZQ
I still couldn't import sqlite3. So I went into setup.py and made changes:
# We hunt for #define SQLITE_VERSION "n.n.n"
# We need to find >= sqlite version 3.0.8
sqlite_incdir = sqlite_libdir = None
sqlite_inc_paths = [ '/usr/include',
'/usr/include/sqlite',
'/usr/include/sqlite3',
'/usr/local/include',
'/usr/local/include/sqlite',
'/usr/local/include/sqlite3',
'/home/jwxie518/python-mod/include/sqlite',
'/home/jwxie518/python-mod/include/sqlite3',
]
Then again, ran everything over (this time I also did make clean)
Output is here: http://pastebin.com/raw.php?i=8ZKgAcWn
According to the output, I don't think the custom path is included.... (for complete output please go to the link above and search for sqlite)
build/temp.linux-i686-2.7/home/jwxie518/Python-2.7.1/Modules/_sqlite/util.o
-L/usr/lib -L/usr/local/lib -Wl,-R/usr/lib -lsqlite3 -o build/lib.linux-i686-2.7/_sqlite3.so
I still cannot import sqlite3.
THanks!
Thank you very much, Michael Dillon, for helping me out. Your tutorial was neat and clear.
I solved the problem as soon as I realized whenever I tried Python-2.7.1, I was actually using the one installed by Pythonbrew.
The moral of the story is read all the errors. I neglected the errors generated by importing sqlite3. The one installed by Pythonbrew didn't have sqlite3 installed. The development package for sqlite3 was installed after Pythonbrew installed the Python-2.7.1
Thanks.

Here is how to build Python and fix any dependencies. I am assuming that you want this Python to be entirely separate from the Ubuntu release Python, so I am specifying the --prefix option to install it all in /home/python27 using the standard Python layout, i.e. site-packages instead of dist-packages.
1. Get the .tar.gz file into your own home directory.
2. tar zxvf Py*.tar.gz
3. cd Py*1
4. ./configure --prefix=/home/python27
5. make
6. make install
Step 5 is the important one. At the end, it will display a list of any modules that could not be built properly. Often you can fix this by installing an Ubuntu package, and rerunning make.
a. sudo apt-get install something-dev
b. make
It is pretty common to have a problem because you are missing the -dev addon to some module or other. But sometimes you should start over like this:
a. make clean
b. ./configure --prefix=/home/python27
c. make
Starting over never hurts if you are unsure. An important note about step 6. I am not using sudo on this command which means that you will need to have the /home/python27 directory already created with the appropriate ownership.
Don't hesitate to try out ./configure --help |less before building something because there may be interesting options that you could use. One time on a minimal distro I had to do --with-dbmliborder=gdbm:bdb in order to get gdbm working. When you run ./configure, the last few lines will tell you where it put the information that it learned. In the case of Python, Modules/Setup has been useful to figure out how to get a module to build.
Another useful thing is to make clean and then run make >make.out 2>&1 to capture all the output from the full make process. Then, after it is complete, use less or an editor to look for the details on a problem module such as _sqlite. For instance, check all the -I options that are passed to gcc. If the correct include directory is not on the list that would cause a problem. You can edit setup.py to change the list of include directories.
In the past it was more common to have library problems that would be fixed by logging out, logging in again, and running "sudo ldconfig" before doing a complete rebuild.

Why can't Python find shared objects that are in directories in sys.path?

I'm trying to import pycurl:
$ python -c "import pycurl"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: libcurl.so.4: cannot open shared object file: No such file or directory
Now, libcurl.so.4 is in /usr/local/lib. As you can see, this is in sys.path:
$ python -c "import sys; print(sys.path)"
['', '/usr/local/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
'/usr/local/lib/python25.zip', '/usr/local/lib/python2.5',
'/usr/local/lib/python2.5/plat-linux2', '/usr/local/lib/python2.5/lib-tk',
'/usr/local/lib/python2.5/lib-dynload',
'/usr/local/lib/python2.5/sitepackages', '/usr/local/lib',
'/usr/local/lib/python2.5/site-packages']
Any help will be greatly appreciated.

sys.path is only searched for Python modules. For dynamic linked libraries, the paths searched must be in LD_LIBRARY_PATH. Check if your LD_LIBRARY_PATH includes /usr/local/lib, and if it doesn't, add it and try again.
Some more information (source):
In Linux, the environment variable
LD_LIBRARY_PATH is a colon-separated
set of directories where libraries
should be searched for first, before
the standard set of directories; this
is useful when debugging a new library
or using a nonstandard library for
special purposes. The environment
variable LD_PRELOAD lists shared
libraries with functions that override
the standard set, just as
/etc/ld.so.preload does. These are
implemented by the loader
/lib/ld-linux.so. I should note that,
while LD_LIBRARY_PATH works on many
Unix-like systems, it doesn't work on
all; for example, this functionality
is available on HP-UX but as the
environment variable SHLIB_PATH, and
on AIX this functionality is through
the variable LIBPATH (with the same
syntax, a colon-separated list).
Update: to set LD_LIBRARY_PATH, use one of the following, ideally in your ~/.bashrc
or equivalent file:
export LD_LIBRARY_PATH=/usr/local/lib
or
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
Use the first form if it's empty (equivalent to the empty string, or not present at all), and the second form if it isn't. Note the use of export.

Ensure your libcurl.so module is in the system library path, which is distinct and separate from the python library path.
A "quick fix" is to add this path to a LD_LIBRARY_PATH variable. However, setting that system wide (or even account wide) is a BAD IDEA, as it is possible to set it in such a way that some programs will find a library it shouldn't, or even worse, open up security holes.
If your "locally installed libraries" are installed in, for example, /usr/local/lib, add this directory to /etc/ld.so.conf (it's a text file) and run ldconfig
The command will run a caching utility, but will also create all the necessary "symbolic links" required for the loader system to function. It is surprising that the make install for libcurl did not do this already, but it's possible it could not if /usr/local/lib is not in /etc/ld.so.conf already.
PS: it's possible that your /etc/ld.so.conf contains nothing but include ld.so.conf.d/*.conf. You can still add a directory path after it, or just create a new file inside the directory it's being included from. Dont forget to run ldconfig after it.
Be careful. Getting this wrong can screw up your system.
Additionally: make sure your python module is compiled against THAT version of libcurl. If you just copied some files over from another system, this wont always work. If in doubt, compile your modules on the system you intend to run them on.

You can also set LD_RUN_PATH to /usr/local/lib in your user environment when you compile pycurl in the first place. This will embed /usr/local/lib in the RPATH attribute of the C extension module .so so that it automatically knows where to find the library at run time without having to have LD_LIBRARY_PATH set at run time.

Had the exact same issue. I installed curl 7.19 to /opt/curl/ to make sure that I would not affect current curl on our production servers.
Once I linked libcurl.so.4 to /usr/lib:
sudo ln -s /opt/curl/lib/libcurl.so /usr/lib/libcurl.so.4
I still got the same error! Durf.
But running ldconfig make the linkage for me and that worked. No need to set the LD_RUN_PATH or LD_LIBRARY_PATH at all. Just needed to run ldconfig.

As a supplement to above answers - I'm just bumping into a similar problem, and working completely of the default installed python.
When I call the example of the shared object library I'm looking for with LD_LIBRARY_PATH, I get something like this:
$ LD_LIBRARY_PATH=/path/to/mysodir:$LD_LIBRARY_PATH python example-so-user.py
python: can't open file 'example-so-user.py': [Errno 2] No such file or directory
Notably, it doesn't even complain about the import - it complains about the source file!
But if I force loading of the object using LD_PRELOAD:
$ LD_PRELOAD=/path/to/mysodir/mypyobj.so python example-so-user.py
python: error while loading shared libraries: libtiff.so.5: cannot open shared object file: No such file or directory
... I immediately get a more meaningful error message - about a missing dependency!
Just thought I'd jot this down here - cheers!

I use python setup.py build_ext -R/usr/local/lib -I/usr/local/include/libcalg-1.0 and the compiled .so file is under the build folder.
you can type python setup.py --help build_ext to see the explanations of -R and -I

For me what works here is to using a version manager such as pyenv, which I strongly recommend to get your project environments and package versions well managed and separate from that of the operative system.
I had this same error after an OS update, but was easily fixed with pyenv install 3.7-dev (the version I use).

Installing python on 1and1 shared hosting

I'm trying to install python to a 1and1.com shared linux hosting account.
There is a nice guide at this address:
http://www.jacksinner.com/wordpress/?p=3
However I get stuck at step 6 which is: "make install". The error I get is as follows:
(uiserver):u58399657:~/bin/python > make install
Creating directory /~/bin/python/bin
/usr/bin/install: cannot create directory `/~’: Permission denied
Creating directory /~/bin/python/lib
/usr/bin/install: cannot create directory `/~’: Permission denied
make: *** [altbininstall] Error 1
I look forward to some suggestions.
UPDATE:
Here is an alternative version of the configure step to fix the above error, however this time I'm getting a different error:
(uiserver):u58399657:~ > cd Python-2.6.3
(uiserver):u58399657:~/Python-2.6.3 > ./configure -prefix=~/bin/python
configure: error: expected an absolute directory name for --prefix: ~/bin/python
(uiserver):u58399657:~/Python-2.6.3 >

The short version is, it looks like you've set the prefix to /~/bin/python instead of simply ~/bin/python. This is typically done with a --prefix=path argument to configure or some other similar script. Try fixing this and it should then work. I'd suggest actual commands, but it's been a while (hence my request to see what you've been typing.)
Because of the above mistake, it is trying to install to a subdirectory called ~ of the root directory (/), instead of your home directory (~).
EDIT: Looking at the linked tutorial, this step is incorrect:
./configure --prefix=/~/bin/python
It should instead read:
./configure --prefix=~/bin/python
Note, this is addressed in the very first comment to that post.
EDIT 2: It seems that whatever shell you are using isn't expanding the path properly. Try this instead:
./configure --prefix=$HOME/bin/python
Failing even that, run echo $HOME and substitute that for $HOME above. It should look something like --prefix=/home/mscharley/bin/python

You really should consider using the AS binary package from Activestate for this kind of thing. Download the .tar.gz file, unpack it, change to the python directory and run the install shell script. This installs a completely standalone version of python without touching any of the system stuff. You don't need root permissions and you don't need to mess around with make.
Of course, maybe you are a C/C++ developer, make is a familiar tool and you are experienced at building packages from source. But if any of those is not true then it is worth your while to try out the Activestate AS binary package.

I was facing same issue with 1and1 shared hosting (Your provided linked tutorial is not available now). I followed Installing Python modules on Hostgator shared hosting using VirtualEnv tutorial with only one change for 1and1. That is:
Instead of:
> python virtualenv-1.11.6/virtualenv.py /home1/yourusername/public_html/yourdomain.com/env --no-site-package
I used:
> python virtualenv-1.11.6/virtualenv.py /kunden/homepages/29/yourusername/htdocs/env --no-site-package
Rest of the instructions worked and I successfully installed VirtualEnv.
Example: 1and1 does not provide Requests module and pip cannot be used in shared hosting. This screenshot demonstrates that after installing VirtualEnv, pip command can be used and at the end >>> import requests successfully worked.

Unable to install python-setuptools: ./configure: No such file or directory

The question is related to the answer to "Unable to install Python without sudo access".
I need to install python-setuptools to install python modules.
I have extracted the installation package.
I get the following error when configuring
[~/wepapps/pythonModules/setuptools-0.6c9]# ./configure --prefix=/home/masi/.local
-bash: ./configure: No such file or directory
I did not find the solution at the program's homepage.
How can I resolve this error?

As Noah states, setuptools isn't an automake package so doesn't use ‘./configure’. Instead it's a pure-Python-style ‘setup.py’ (distutils) script.
You shouldn't normally need to play with .pydistutils.cfg, as long as you run it with the right version of Python. So if you haven't added the .local/bin folder to PATH, you'd have to say explicitly:
/home/masi/.local/bin/python setup.py install
AIUI this should Just Work.
I did not find the solution at the program's homepage.
Yeah, they want you to install it from a shell script egg which uses the default version of Python. Which you don't want.
(Another approach if you can't get setuptools to work is to skip it and install each module and dependency manually. Personally I have a bit of an aversion to setuptools/egg, as it contains far too much “clever” magic for my tastes and makes a mess of my filesystem. But I'm an old curmudgeon like that. Most Python modules can be obtained as simple Python files or plain old distutils scripts, but unfortunately there are some that demand eggs.)

You might want to check http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations.
EasyInstall is a python module with some shell scripts (or some shell scripts with a python module?) and does not use the unix make tool that gets configured with the "./configure" command. It looks like your best bet is to try editing ~/.pydistutils.cfg to include:
[install]
install_lib = /home/masi/.local/lib/python/site-packages/
install_scripts = /home/masi/.local/bin
You'll also presumably have made the ~/.local/bin/ folder part of your PATH so you can run the easy_install script. (I'm not sure exactly where the site-packages directory will be under .local, but it shouldn't be hard to find.)
Hope this helps.