I can't get scipy to function in 32 bit mode when compiled as a i386/x86_64 universal binary, and executed on my 64 bit 10.6.2 MacPro1,1.
My python setup
With the help of this answer, I built a 32/64 bit intel universal binary of python 2.6.4 with the intention of using the arch command to select between the architectures. (I managed to make some universal binaries of a few libraries I wanted using lipo.) That all works. I then installed scipy according to the instructions on hyperjeff's article, only with more up-to-date numpy (1.4.0) and skipping the bit about moving numpy aside briefly during the installation of scipy.
Now, everything except scipy seems to be working as far as I can tell, and I can indeed select between 32 and 64 bit mode using arch -i386 python and arch -x86_64 python.
The error
Scipy complains in 32 bit mode:
$ arch -x86_64 python -c "import scipy.interpolate; print 'success'"
success
$ arch -i386 python -c "import scipy.interpolate; print 'success'"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/interpolate/__init__.py", line 7, in <module>
from interpolate import *
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/interpolate/interpolate.py", line 13, in <module>
import scipy.special as spec
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/__init__.py", line 8, in <module>
from basic import *
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/basic.py", line 8, in <module>
from _cephes import *
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so, 2): Symbol not found: _aswfa_
Referenced from: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so
Expected in: flat namespace
in /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so
Attempt at tracking down the problem
It looks like scipy.interpolate imports something called _cephes, which looks for a symbol called _aswfa_ but can't find it in 32 bit mode. Browsing through scipy's source, I find an ASWFA subroutine in specfun.f. The only scipy product file with a similar name is specfun.so, but both that and _cephes.so appear to be universal binaries:
$ cd /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/
$ file _cephes.so specfun.so
_cephes.so: Mach-O universal binary with 2 architectures
_cephes.so (for architecture i386): Mach-O bundle i386
_cephes.so (for architecture x86_64): Mach-O 64-bit bundle x86_64
specfun.so: Mach-O universal binary with 2 architectures
specfun.so (for architecture i386): Mach-O bundle i386
specfun.so (for architecture x86_64): Mach-O 64-bit bundle x86_64
Ho hum. I'm stuck. Things I may try but haven't figured out how yet include compiling specfun.so myself manually, somehow.
I would imagine that scipy isn't broken for all 32 bit machines, so I guess something is wrong with the way I've installed it, but I can't figure out what.
I don't really expect a full answer given my fairly unique (?) setup, but if anyone has any clues that might point me in the right direction, they'd be greatly appreciated.
(edit) More details to address questions:
I'm using gfortran (GNU Fortran from GCC 4.2.1 Apple Inc. build 5646).
Python 2.6.4 was installed more-or-less like so:
cd /tmp
curl -O http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2
tar xf Python-2.6.4.tar.bz2
cd Python-2.6.4
# Now replace buggy pythonw.c file with one that supports the "arch" command:
curl http://bugs.python.org/file14949/pythonw.c | sed s/2.7/2.6/ > Mac/Tools/pythonw.c
./configure --enable-framework=/Library/Frameworks --enable-universalsdk=/ --with-universal-archs=intel
make -j4
sudo make frameworkinstall
Scipy 0.7.1 was installed pretty much as described as here, but it boils down to a simple sudo python setup.py install.
It would indeed appear that the symbol is undefined in the i386 architecture if you look at the _cephes library with nm, as suggested by David Cournapeau:
$ nm -arch x86_64 /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so | grep _aswfa_
00000000000d4950 T _aswfa_
000000000011e4b0 d _oblate_aswfa_data
000000000011e510 d _oblate_aswfa_nocv_data
(snip)
$ nm -arch i386 /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so | grep _aswfa_
U _aswfa_
0002e96c d _oblate_aswfa_data
0002e99c d _oblate_aswfa_nocv_data
(snip)
however, I can't yet explain its absence.
Have you tried using scipy compiled using macports?
sudo port install scipy +universal
(of course you must have the rest of the chain, python, py26-numpycompiled with the same option)
I get:
$ arch -x86_64 /opt/local/bin/python -c "import scipy.interpolate; print 'success'"
success
$ arch -i386 /opt/local/bin/python -c "import scipy.interpolate; print 'success'"
success
you may then use the setting and knowledge that the macports maintainers used to make your own compilation.
How did you install scipy, for which python version, and with which fortran compiler ?
You may also want to check that the missing symbol is indeed in both archs (I don't remember off-hand where the function is, but you should be able to find ti by yourself pretty easily using a combination of nm/otool).
Related
MacOS 11.1
I have no idea what this error is trying to tell me. Are there additional arguments I need to pass to pyinstaller to get it to build me a MacOS executable?
$ which python3
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3
$ python3 -m PyInstaller chambers.py
...
$ dist/chambers/chambers
[12313] Error loading Python lib '/Users/fy/chambers3/dist/chambers/Python': dlopen: dlopen(/Users/fy/chambers3/dist/chambers/Python, 10): no suitable image found. Did find:
/Users/fy/chambers3/dist/chambers/Python: code signature invalid for '/Users/fy/chambers3/dist/chambers/Python'
$ file dist/chambers/Python
dist/chambers/Python: Mach-O 64-bit dynamically linked shared library x86_64
As of Dec 21, 2020, there seems to be an issue with MacOS 11.1 or BigSur update as indicated by another post: https://stackoverflow.com/a/65383402/14386048
Until it gets patched, you can look into two alternatives for completing the same objective of converting a python file into an executable:
py2app : https://github.com/ronaldoussoren/py2app
cx_freeze : https://github.com/marcelotduarte/cx_Freeze
I'm trying to compile the msgpack-python python module with gcc (v4.7) on solaris 10. The python installed is 2.6.8. Distutils is automatically picking up a incorrect compiler option (-xcode=pic32) that I want to remove from command.
The full command that distutils is putting together is:
/opt/csw/bin/gcc-4.7 -DNDEBUG -O -O2 -pipe -mcpu=v9 -I/opt/csw/include -xcode=pic32 -I/opt/csw/include/python2.6 -c msgpack/_msgpack.c -o build/temp.solaris-2.10-sun4v-2.6/msgpack/_msgpack.o
but produces this error:
gcc-4.7: error: language code=pic32 not recognized
then fails. If I remove that -xcode=pic32 option and manually execute the above command the module compiles successfully.
I need to be able to do this in an automated fashion though (using a buildfarm to produce the packages). The question is, Without modifying or changing the current python or distutils, is there a way to "remove" this option that distutils is picking up, so I can have the python setup.py process build the module appropriately (i.e. without the pic32 option)?
Thanks
Do not compile with that gcc. -xcode=pic32 is Sun Studio complier command line parameter. It will lead to linking problems too, even if you compile OK. Compile with SUN CoolTools gcc which can understand such parameter, or use Oracle Solaris Studio for SPARC.
Some hints:
GCC produce very slow code for SPARC, that's for why SUN created Cool Tools.
You haven't to remove -xcode=pic32, but change for -m32 -fpic, when you insist on gcc-4.7
To get mature setup of OSS tool I'm using pkgsrc compiling with Studio Express to particular CPU (-xtarget=native)
You may also find luck by setting the follow env vars:
export CC=$gcc_dir_path # Example: /usr/bin/gcc
export CXX=$gxx_dir_path # Example: /usr/bin/g++
export CFLAGS=''
export CPPFLAGS=''
export CXXFLAGS=''
export LDFLAGS=''
Note: There is a difference between unset env var, and set-as-empty env var. I had build bugs with Python packages when my *FLAGS env vars were unset. (Calling gcc with option -xO2 was the cause.) Setting as empty did the trick.
I have a rather long setup, then three questions at the end.
On OS X, the System Python framework contains three executables (let me give them short names):
> F=/System/Library/Frameworks/Python.framework/Versions/2.6
> A=$F/bin/python2.6
> B=$F/Resources/Python.app/Contents/MacOS/Python
> C=$F/Python
$A and $B are clearly too small to be Python itself.
> ls -s $A; ls -s $B; ls -s $C
16 /System/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
16 /System/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python
3152 /System/Library/Frameworks/Python.framework/Versions/2.6/Python
> $A
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
> $B
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
> $C
-bash: /System/Library/Frameworks/Python.framework/Versions/2.6/Python: cannot execute binary file
Despite equal size and apparently, effect, the first two are different, e.g.:
> cmp -lc $A $B
Also, in /usr/bin, python2.6 is a symlink to $C, but there is also:
> D=/usr/bin/python
> ls -s $D
48 /usr/bin/python
I want to sort out how these are connected; the command which doesn't help.
> export DYLD_PRINT_LIBRARIES=1
> $A
..
dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python
dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.6/Python
Summary: $A loads $B followed by $C; $B loads $C; $D loads $B followed by $C
So my questions are:
Is this documented anywhere?
What roles do these play?
Most important, what tools would be useful in tracing connections like this?
The Apple-supplied Pythons in OS X 10.6 are built and installed using the standard Python framework build option, with a few customization tweaks. It is not in Apple's documentation because the specific layout is not an Apple invention; it has evolved over the years by the Python project using other OS X framework layouts as a starting point. If you install a version of Python on OS X using one of the python.org installers, say from here, you will see the same pattern, with the framework rooted at /Library/Frameworks/ rather than /System/Library/Frameworks. So, if you are really curious, you can download the source and look at the configure script and Makefile templates. It can be heavy reading, though. Apple also makes available here the source used to build open source components, including Python, in each OS X release along with the customization patches so, in theory, you can see exactly how Apple built what it released.
That said, to address your questions, in Python 2.6:
$A is the pythonw wrapper that ensures Python is recognized as a GUI application by OS X (see the source of pythonw.c here). Note, the Apple version of pythonw has been customized to add the preferred execution modes (see Apple's man 1 python). A somewhat different approach to this is provided in the upstream source of newer versions of Python (2.7 and 3.2).
$B is the actual executable of the Python interpreter. It is what is execed by the pythonw executable, $A. You should be able to easily verify that by actually running Python and looking at the value of sys.executable but there is a bug with the Apple-supplied Python 2.6 (probably due to the added feature mentioned above) that causes the wrong value to be assigned to it. The python.org Python 2.6.6 shows the correct value:
$ cd /Library/Frameworks/Python.framework/Versions/2.6
$ ./bin/python2.6 -c 'import sys;print(sys.executable)'
/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python`
$C is the shared library containing all of the interpreter's loadable modules. You can see that by using otool on $B:
$ cd /System/Library/Frameworks/Python.framework/Versions/2.6
$ cd Resources/Python.app/Contents/MacOS/
$ otool -L ./Python
Python:
/System/Library/Frameworks/Python.framework/Versions/2.6/Python (compatibility version 2.6.0, current version 2.6.1)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)
The tools to use are ls and file.
ls -l will give what the symbolic link goes to. The size of a symbolic link is the number of chafracters in the path it points to.
file x will give the type of the file
e.g.
file /System/Library/Frameworks/Python.framework/Versions/2.6/Python
/System/Library/Frameworks/Python.framework/Versions/2.6/Python: Mach-O universal binary with 3 architectures
/System/Library/Frameworks/Python.framework/Versions/2.6/Python (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64
/System/Library/Frameworks/Python.framework/Versions/2.6/Python (for architecture i386): Mach-O dynamically linked shared library i386
/System/Library/Frameworks/Python.framework/Versions/2.6/Python (for architecture ppc7400): Mach-O dynamically linked shared library ppc
OSX Frameworks are described in Apple developer docs
/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python and
/System/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6 are the actual python interpreter, I think they are hard links to the same binary.
/usr/bin/python is the python on the path - I think it is hard linked to /usr/bin/pythonw. These are wrappers that call exec to the real python interpreter in /System/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6 see python bug tracker
/System/Library/Frameworks/Python.framework/Versions/Current is a symlink to System/Library/Frameworks/Python.framework/Versions/2.6 using the standard OSX Framework versioning
/System/Library/Frameworks/Python.framework/Versions/2.6/Python is the shared library that does all the work - set up as a library so that you can write programs in other languages that can embed a python interpreter.
For other details look at Python docs but I suspect you would have to search the apple python mailing list
I am answering my own question, but because I stayed up all night figuring it out, I will hopefully save others some pain. If you are getting either of the following messages after properly installing pytidylib or utidylib, this answer may help.
Learning Python on Snow Leopard, I had installed the 32-bit version of Python 2.7 so that I could use the IDLE interpreter/editor. Stackoverflow has a great explanation why I had to do this.
When I installed utidylib, I got the following error from 'import tidy':
'OSError: Couldn't find libtidy, please make sure it is installed.'
With pytidylib, I got this error when I tried 'from tidylib import tidy_document':
'OSError: Could not load libtidy using any of these names: libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy.'
If you are getting these errors, please read this answer. I hope it helps you.
I had a similar issue (see stacktrace below) on Linux.
With pytidylib==0.2.1 installed with pip in a virtualenv on both Ubuntu 11.10 (Python 2.7) and Gentoo (Python 2.6).
Installing apt-get install libtidy-dev on Ubuntu and emerge app-text/htmltidy on Gentoo, solved it.
----------------------------------------------------------------------
File "/home/andre/Progz/some_site/djcode/apps/default/tests.py", line ?, in default.tests.__test__.doctest
Failed example:
from tidylib import tidy_document
Exception raised:
Traceback (most recent call last):
File "/home/andre/Progz/some_site/lib/python2.6/site-packages/django/test/_doctest.py", line 1267, in __run
compileflags, 1) in test.globs
File "", line 1, in
from tidylib import tidy_document
File "/home/andre/Progz/some_site/lib/python2.6/site-packages/tidylib/__init__.py", line 71, in
raise OSError("Could not load libtidy using any of these names: %s" % (",".join(LIB_NAMES)))
OSError: Could not load libtidy using any of these names: libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy
----------------------------------------------------------------------
Although it looked like the Python packages could not find the files, what was really happening was that I compiled libtidy for the wrong architecture: x86_64 instead of i386. Using the 32-bit Python installer made them incompatible.
For neophytes like me, lipo will tell you the architecture of your files:
$ lipo -info /Library/Frameworks/Python.framework/Versions/2.7/bin/python
Architectures in the fat file: /Library/Frameworks/Python.framework/Versions/2.7/bin/python are: ppc i386
$ lipo -info /usr/lib/libtidy.dylib
Non-fat file: /usr/lib/libtidy.dylib is architecture: x86_64
You may have to locate your libtidy.dylib and python files to compare. To fix the problem, you have to re-compile libtidy for i386 (or link to an i386-compiled libtidy already on your system). To re-compile for i386, use these instructions with the following modification:
cvs -z3 -d:pserver:anonymous#tidy.cvs.sourceforge.net:/cvsroot/tidy co -P tidy
cd tidy
sh build/gnuauto/setup.sh
./configure --prefix=/usr CFLAGS="-arch i386"
make && sudo make install
This worked for me. I hope it also works for you.
I had a same issue.I installed tidylib,but still come out error as followed.
raise OSError("Could not load libtidy using any of these names: %s" % (",".join(LIB_NAMES)))
OSError: Could not load libtidy using any of these names: libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy
then I copy /usr/bin/tidy to my project folder ,and it got solved.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.6/site-packages/psycopg2/__init__.py", line 60, in <module>
from _psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: dlopen(/Library/Python/2.6/site-packages/psycopg2/_psycopg.so, 2): Symbol not found: _PQbackendPID
Referenced from: /Library/Python/2.6/site-packages/psycopg2/_psycopg.so
Expected in: flat namespace
in /Library/Python/2.6/site-packages/psycopg2/_psycopg.so
Psycopg2 was working fine before, but now I get this error.
Any ideas on this issue much appreciated.
EDIT: so after dealing with so many psycopg2 errors everytime I set up my mac, I've decided to use VMWareFusion running Ubuntu instead.
You get this error because your 64-bit version of python can't find a 64-bit psycopg2.
You can either downgrade your python to run in 32-bit mode or try to get a 64-bit psycopg2. There is more discussion on this topic over on Ben Kreeger's blog.
Could it be that the postgres installation was removed/updated? The symbol is supposed to come from libpq.
This is broken for me too, and in my case it doesn't appear to be a 32 vs 64 bit issue:
decibel#workbook.1[6:55]~/src:85%file /opt/local/lib/postgresql83/libpq.dylib
/opt/local/lib/postgresql83/libpq.dylib: Mach-O 64-bit dynamically linked shared library x86_64
decibel#workbook.1[6:56]~/src:86%file ~/.python-eggs/psycopg2-2.0.14-py2.6-macosx-10.6-universal.egg-tmp/psycopg2/_psycopg.so
/Users/decibel/.python-eggs/psycopg2-2.0.14-py2.6-macosx-10.6-universal.egg-tmp/psycopg2/_psycopg.so: Mach-O universal binary with 3 architectures
/Users/decibel/.python-eggs/psycopg2-2.0.14-py2.6-macosx-10.6-universal.egg-tmp/psycopg2/_psycopg.so (for architecture i386): Mach-O bundle i386
/Users/decibel/.python-eggs/psycopg2-2.0.14-py2.6-macosx-10.6-universal.egg-tmp/psycopg2/_psycopg.so (for architecture ppc7400): Mach-O bundle ppc
/Users/decibel/.python-eggs/psycopg2-2.0.14-py2.6-macosx-10.6-universal.egg-tmp/psycopg2/_psycopg.so (for architecture x86_64): Mach-O 64-bit bundle x86_64
decibel#workbook.1[6:56]~/src:87%
resolved similar issue by forcing Apache executables 32 bit execution