Develop Reference

Can I zip PySpark dependencies containing some setuptools.Extension?

I am attempting to include the dateparser package for a PySpark (v2.4.3) shell session by a short little zip build process pip install -r requirements.txt -t some_target && cd some_target && zip -r ../deps.zip . && cd .., after which I would, for example, pyspark --py-files deps.zip. When importing dateparser, though, I get an indirect ModuleNotFoundError from the regex library, whining that "No module named 'regex._regex'" (stack trace says this is referenced in /mnt/tmp/spark-some/long/path/deps.zip/regex/_regex_core.py line 21, which is of course referenced much farther up the stack by dateparser).
I attempted adding a flag to the dateparser line in requirements.txt like dateparser --no-binary=regex, but the error persisted. A normal python shell is able to import without issue, and other packages in this zip seem to be importable in PySpark shell without issue. This has led me down a number of rabbit holes, but I think/hope I have finally found the culprit: namely, that regex._regex is not a normal .py file, but rather a .so. My knowledge of python build process is limited, but it seems that regex library's setup.py uses the setuptools.Extension class to compile some C files into this shared object. I have seen suggestions to modify LD_LIBRARY_PATH environment variable in order to make those shared objects discoverable to python, but a number of comments also suggested this was dangerous and not a viable long-term solution. The fact that a normal python interactive session has no issue with the import also has me skeptical, since the LD_LIBRARY_PATH variable doesn't even exist in os.environ within that interactive shell. I'm thence left wondering if --py-files is insufficient for including packages that compile these Extension objects (seems unlikely, since there are a lot of people doing crazier things than my simple use case), or if this actually stems from neglecting some other setting.
Merci mille fois for any and all help :)

The error appears to stem from the import statements not being able to recognize binary (.so) files within a zip archive, i.e., the dependencies.zip that I pass with the --py-files parameter. I first tried pulling out regex dependency and building a .whl to include in --py-files, to discover that my version of PySpark (v2.4.3) predates wheel support. I was, however, able to build an .egg based on the source code, then set PYTHON_EGG_CACHE and PYTHON_EGG_DIR env variables for spark.executorEnv and spark.driverEnv... Not sure if the last step would be necessary for others; it seems to have stemmed from weird permissions issues that may just apply to my user/group/use case.

Python Pylearn2 package "ImportError: No module named pylearn2.utils"

I have recently tried to use pylearn2, a deep machin learning package for Python developed at University of Montreal.
I've just installed it and tried to run a simple example, but it did not work.
I have been using a pc with an Ubuntu 13.10 system, on which I found ipython installed.
I have installed Theano and later pylearn2, by following this webpage instructions:
http://deeplearning.net/software/pylearn2/
I have also modified the .bashrc file, as suggested
I thought that everything went well, and then I tried this Quick start example:
http://deeplearning.net/software/pylearn2/tutorial/index.html
I stopped at the first command:
python make_dataset.py
My terminal states:
Traceback (most recent call last): File "make_dataset.py", line 14,
in
Do you have any ideas on why it is not working?
Do you why these errors occur?
Thanks a lot
EDIT: the 14 line is the first non-commented line of the file. It states
from pylearn2.utils import serial

Without more information, I can only guess, but my first guess is…
You haven't actually installed pylearn2, because if you follow the linked docs to grab the git repo and add a PYLEARN2_DATA_PATH variable, nothing gets installed into site-packages (or dist-packages or anywhere else on sys.path).
This means that pylearn2 will only work when you start Python from within the top-level directory of the pylearn2 repo.
So, if you run a script like this:
$ cd /path/to/pylearn2
$ cd scripts/tutorials/grbm_smd/
$ python make_dataset.py
… it won't actually work.
It looks like there is a setup.py file in the repository. Does it work? I have no idea. Even though the docs don't mention using it, you might want to try. Either this:
$ pip install .
… or, if you don't have pip or it doesn't work on this package:
$ python setup.py install
Either way, of course, you may need sudo or a flag to install to your user site-packages instead of system, etc., as with any other Python package.
If that doesn't work, you might be able to just add /path/to/pylearn2 to your sys.path in some way. The most obvious way is by doing an export PYTHONPATH=/path/to/pylearn2:$PYTHONPATH in your ~/.bashrc.
Also, you will need to either source ~/.bashrc or create a new shell to get any effects of modifying the file.
If you're wondering why the instructions and the tutorial together don't give you enough information to make this work without a lot of hassle, I think that's covered in the very top of the documentation:
Pylearn2 is still undergoing rapid development. Don’t expect a clean road without bumps!
And the very fact that there is no PyPI download yet implies that this really is not ready for novices to use. If you don't know enough about using Python packages (and bash basics) to muddle through on your own, there's a good chance you won't be able to use this package.

Python installation, failing to find bz2 module

OK, I have an old Debian VM. Package managers are useless. No, I'm not going to update the OS.
I have the bzip2 lib and development headers installed correctly on my system (those actually came from a package).
I start with absolutely NO Python on the system. I removed everything manually. I downloaded the Python 2.7.5 source, and configured with ./configure --prefix=/usr. It configures fine. I run make, and it compiles fine. I try ./python -c "import bz2; print bz2.__doc__", it works, and says:
The python bz2 module provides a comprehensive interface for
the bz2 compression library. It implements a complete file
interface, one shot (de)compression functions, and types for
sequential (de)compression.
I then run make test and the whole test suite progresses fine, and notably the "test_bz2" test passes.
I then run make install, which installs my new Python binary into /usr/bin/ like I wanted.
I try /usr/bin/python -c "import bz2; print bz2.__doc__", and it fails with:
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named bz2
I've tried a bunch of different things, including building Python as --enable-shared and not, no luck. I've tried at least 10 times (each time totally cleaning out everything, running make distclean, etc.). No luck.
I tried: PYTHONPATH="/usr/lib/python2.7"; export PYTHONPATH. Still no luck.
HOWEVER, if I delete the symlink that make install creates for /usr/bin/python, and instead do: ln -s /path/to/my/python/compile/python python, NOW it magically works.
So, what the heck? Why is this Python binary I'm getting created only able to find stuff when the binary exists in the compile directory, and not when it's put into normal production install location? What am I missing?
I am root during the entire process, from configure to make to make install to trying to test the Python import call.
I have started from scratch again (this time compiling with --enable-shared btw), and verified that not only in the compile directory is there build/lib.linux-x86_64-2.7/bz2.so, but once I run make install, that file is put into /usr/lib/python2.7/lib-dynload/bz2.so.
I've tried to do some reading on lib-dynload, but haven't been able to determine if there's something else a Python program (like default configuration for the CLI or whatever) would need to be able to tell it to pull module imports from lib-dynload, or if there's some other place or option to tell the make install where it should be putting it instead of dynload.
Still I have no explanation why the /path/to/compilation/python binary can find and load bz2.so fine, but the /usr/bin/python binary can't find (or load) /usr/lib/python2.7/lib-dynload/bz2.so.
I thought maybe it was something to do with the fact that the installation doesn't create like a /usr/lib/python symlink to point at /usr/lib/python2.7 directory. But I created the symlink and still no go.
I am still lost here.

It would appear that a sort of non-answer answer was arrived at accidentally via a long string of Twitter conversation(s).
I've filed another Stack Overflow question here to ask WHY what we found was the solution to this problem: https://stackoverflow.com/questions/17662091/python-installation-prefix-not-being-persisted-in-config
For posterity sake, right now the solution is that I have to set the PYTHONHOME environment variable to /usr, and everything starts working. The puzzling part is that the documentation says PYTHONHOME should default to {prefix}, which I was clearly setting as default during configure to /usr. So why should I have to manually set it?
Running python-config --prefix reveals that the {prefix} default is in fact /usr/bin, NOT /usr like I specified, which leads to me needing to override the default back to the default, bizarrely.

py2app built app displays `ERROR: pygame.macosx import FAILED` on other machines

Trying to build an app on the Mac using py2app. Got everything working fine on my machine, but when moving the app to another, it crashes and the console displays this error.
ERROR: pygame.macosx import FAILED
Anybody have a solution to this?

Found the problem and solution after many hours. Turns out other people have experienced similar problems and their articles were quite helpful:
http://b.atcg.us/blog/2010/04/13/py2app-hell-the-first.html
http://www.vijayp.ca/blog/?p=62
In case someone else runs into the issue, this particular problem was caused because the Python framework was not being bundled into the application. You can confirm this by right-clicking your app to view package contents, then proceed to Contents/Frameworks/. If Python.framework is not there, it should be.
Be sure to download Python -
My first issue was reliance on Apple's build in Python package. Don't use this. You need to install your own version of python. Go to http://www.python.org/download/releases/, find a version (I stuck with 2.6), download the gzip (not the mac package), and install with the following if you are running Snow Leopard:
./configure --enable-framework MACOSX_DEPLOYMENT_TARGET=10.6 --with-universal-archs=intel --enable-universalsdk=/
make
sudo make install
Adjust Paths, Install Packages - From here there you need to adjust your paths to ensure you are using your custom-installed version. From here, I reinstalled the following packages - this turned out to be a dependency nightmare so I'm including the version numbers as well:
py2app 0.5.2
macholib 1.3
modulegraph .8.0
If these packages actually worked, you should be able to build and run your app now. Unfortunately, they don't. I'll go into the errors and my hacked solutions in a bit, but there's some settings in the build file that need to be made first.
First the setup.py file should like a little somethin' like this:
setup.py
from setuptools import setup
APP = ['Game.py']
DATA_FILES = ['data']
OPTIONS = {
"argv_emulation": False,
"compressed" : True,
"optimize":2,
"iconfile":'data/game.icns',
}
setup(
app=APP,
data_files=DATA_FILES,
options={'py2app': OPTIONS},
)
then to be extra safe, I use a shell script to call this.
build.sh
## Remove previous builds. Start with clean slate.
rm -rf build dist
## Force python into 32 bit mode.
export VERSIONER_PYTHON_PREFER_32_BIT=yes
## Force build with custom installed python
/Library/Frameworks/Python.framework/Versions/2.6/bin/python setup.py py2app
Running build.sh should compile the app. If it does not compile, I have good news -- it's not your fault. Due to glitches in the libraries, you may run into some (or all) of the following:
Potential Problems
If the build script fails, scan the traceback for some of the following keywords:
pygame not found - basic path problem in py2app. Add...
sys.path.insert(0, os.path.join(os.getcwd(), 'lib', 'python2.6','lib-dynload')) ## Added to fix dynlib bug
after the import statements in boot_app.py in the py2app lib.
pythonNone - This appears to be a bug in the macho package where it cannot determine the version number of your python build. To solve this, I added the following lines to build_app.py in py2app.
## Add these two lines...
if not info["version"]:
info["version"] = "2.6"
## Before this line. (line 941 in method copy_python_framework() at time of writing)
pydir = 'python%s'%(info['version'])
No such file or directory...Python.framework/[lib|include] - py2app is simply looking for directories that exist deeper in the file system tree. Go to the Python.framework directory and symlink up the place...
cd /Library/Frameworks/Python.framework
sudo ln -s Versions/Current/include/ include
sudo ln -s Versions/Current/lib lib
That should do it! - These steps created a compiled app that worked on other intel machines.

Thank you for posting what you found!
I had a similar problem. I tried various combinations of what you suggested, and isolated the single issue for me to be the bug in boot_app.py which you identify above.
Once I added the one-line fix to boot_app.py which you identify above, everything worked, even using the pre-installed Apple build of python (version 2.6.1).
I should note that when I say "everything worked," I really mean building a py2app app for actual distribution, i.e. using the normal command:
python setup.py py2app
The "alias" mode. i.e.
python setup.py py2app -A
which the py2app documentation suggests for use during development, still does not work for me (with the same module not found error). But better the actual distribution build working than nothing at all! Again, thanks.

Unable to install python-setuptools: ./configure: No such file or directory

The question is related to the answer to "Unable to install Python without sudo access".
I need to install python-setuptools to install python modules.
I have extracted the installation package.
I get the following error when configuring
[~/wepapps/pythonModules/setuptools-0.6c9]# ./configure --prefix=/home/masi/.local
-bash: ./configure: No such file or directory
I did not find the solution at the program's homepage.
How can I resolve this error?

As Noah states, setuptools isn't an automake package so doesn't use ‘./configure’. Instead it's a pure-Python-style ‘setup.py’ (distutils) script.
You shouldn't normally need to play with .pydistutils.cfg, as long as you run it with the right version of Python. So if you haven't added the .local/bin folder to PATH, you'd have to say explicitly:
/home/masi/.local/bin/python setup.py install
AIUI this should Just Work.
I did not find the solution at the program's homepage.
Yeah, they want you to install it from a shell script egg which uses the default version of Python. Which you don't want.
(Another approach if you can't get setuptools to work is to skip it and install each module and dependency manually. Personally I have a bit of an aversion to setuptools/egg, as it contains far too much “clever” magic for my tastes and makes a mess of my filesystem. But I'm an old curmudgeon like that. Most Python modules can be obtained as simple Python files or plain old distutils scripts, but unfortunately there are some that demand eggs.)

You might want to check http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations.
EasyInstall is a python module with some shell scripts (or some shell scripts with a python module?) and does not use the unix make tool that gets configured with the "./configure" command. It looks like your best bet is to try editing ~/.pydistutils.cfg to include:
[install]
install_lib = /home/masi/.local/lib/python/site-packages/
install_scripts = /home/masi/.local/bin
You'll also presumably have made the ~/.local/bin/ folder part of your PATH so you can run the easy_install script. (I'm not sure exactly where the site-packages directory will be under .local, but it shouldn't be hard to find.)
Hope this helps.