Develop Reference

PyInstaller failing

MacOS 11.1
I have no idea what this error is trying to tell me. Are there additional arguments I need to pass to pyinstaller to get it to build me a MacOS executable?
$ which python3
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3
$ python3 -m PyInstaller chambers.py
...
$ dist/chambers/chambers
[12313] Error loading Python lib '/Users/fy/chambers3/dist/chambers/Python': dlopen: dlopen(/Users/fy/chambers3/dist/chambers/Python, 10): no suitable image found. Did find:
/Users/fy/chambers3/dist/chambers/Python: code signature invalid for '/Users/fy/chambers3/dist/chambers/Python'
$ file dist/chambers/Python
dist/chambers/Python: Mach-O 64-bit dynamically linked shared library x86_64

As of Dec 21, 2020, there seems to be an issue with MacOS 11.1 or BigSur update as indicated by another post: https://stackoverflow.com/a/65383402/14386048
Until it gets patched, you can look into two alternatives for completing the same objective of converting a python file into an executable:
py2app : https://github.com/ronaldoussoren/py2app
cx_freeze : https://github.com/marcelotduarte/cx_Freeze

waf check_python_headers fails with python installed with pyenv

I'm trying to build a project using the waf build tool.
The build fails in the configure step when checking the python headers.
I narrowed the problem down to this minimal example of my wscript:
def options(ctx):
ctx.load('compiler_c')
def configure(ctx):
ctx.load('compiler_c python')
ctx.check_python_version((2,7,0))
if int(ctx.env.PYTHON_VERSION[0]) == 2:
print ('→ Configuring for python2')
else:
print ('→ Configuring for python3')
ctx.check_python_headers(features='pyext')
I use python versions (3.6.12 & 3.7.9) that I installed with pyenv on Ubuntu 20.04.
I run that like this and it fails with an error that I cannot make sense of:
$ python waf configure
Setting top to : /home/myuser/waf-test
Setting out to : /home/myuser/waf-test/build
Checking for 'gcc' (C compiler) : /usr/bin/gcc
Checking for program 'python' : /home/myuser/.pyenv/versions/3.7.9/bin/python
Checking for python version >= 2.7.0 : 3.7.9
→ Configuring for python3
python-config : /home/myuser/.pyenv/versions/3.7.9/bin/python-config
Asking python-config for pyext '--cflags --libs --ldflags' flags : yes
Testing pyext configuration : Could not build python extensions
The configuration failed
(complete log in /home/myuser/waf-test/build/config.log)
$ tail -n 20 build/config.log
[1/2] Compiling build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/test.c
['/usr/bin/gcc', '-fPIC', '-g', '-fwrapv', '-O3', '-I../../../../.pyenv/versions/3.7.9/include/python3.7m', '-DPYTHONDIR="/usr/local/lib/python3.7/site-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python3.7/site-packages"', '-DNDEBUG', '../test.c', '-c', '-o/home/myuser/waf-test/build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/test.c.1.o']
[2/2] Linking build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/testprog.cpython-37m-x86_64-linux-gnu.so
['/usr/bin/gcc', '-shared', 'test.c.1.o', '-o/home/myuser/waf-test/build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/testprog.cpython-37m-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/home/myuser/.pyenv/versions/3.7.9/lib/python3.7/config-3.7m-x86_64-linux-gnu', '-L/home/myuser/.pyenv/versions/3.7.9/lib', '-lpython3.7m', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lpython3.7m', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm']
err: /usr/bin/ld: /home/myuser/.pyenv/versions/3.7.9/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(pylifecycle.o): relocation R_X86_64_PC32 against symbol `Py_VerboseFlag' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
from /home/myuser/waf-test: Test does not build: Traceback (most recent call last):
File "/home/myuser/waf-test/.waf3-2.0.20-36f5354d605298f6a89c09e0c7ef6c1d/waflib/Configure.py", line 335, in run_build
bld.compile()
File "/home/myuser/waf-test/.waf3-2.0.20-36f5354d605298f6a89c09e0c7ef6c1d/waflib/Build.py", line 176, in compile
raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
-> task in 'testprog' failed with exit status 1 (run with -v to display more information)
Could not build python extensions
from /home/myuser/waf-test: The configuration failed
Note that this problem does not appear with python 3.8.6 that was also installed with pyenv. I can configure without errors.
My questions: How can it be that the python versions in pyenv differ in a way that waf checks fail in one version and not in another?
Since I simply install the versions with pyenv install 3.6.12, what do I need to change in the install so my build succeeds?

Well, the error is that, when creating the test program to check if pyext is correctly set, the compiler complains about the fact that pylifecycle.o included in libpython3.7m.a is non relocatable ie non compiled with the -fPIC flag.
As the code fragment generated by waf is compiled with -fPIC, it seems that the problem is that pylifecycle.o in libpython3.7m.a is not compiled with -fPIC, which makes sense as it is in a static library. Maybe the problem is that it uses the .a archive lib instead of the shared object one ie the .so.
Check if you have the shared object version of libpython ie libpython3.7m.so.

How to remove a compiler option that python disutils includes (language code=pic32 not recognized)

I'm trying to compile the msgpack-python python module with gcc (v4.7) on solaris 10. The python installed is 2.6.8. Distutils is automatically picking up a incorrect compiler option (-xcode=pic32) that I want to remove from command.
The full command that distutils is putting together is:
/opt/csw/bin/gcc-4.7 -DNDEBUG -O -O2 -pipe -mcpu=v9 -I/opt/csw/include -xcode=pic32 -I/opt/csw/include/python2.6 -c msgpack/_msgpack.c -o build/temp.solaris-2.10-sun4v-2.6/msgpack/_msgpack.o
but produces this error:
gcc-4.7: error: language code=pic32 not recognized
then fails. If I remove that -xcode=pic32 option and manually execute the above command the module compiles successfully.
I need to be able to do this in an automated fashion though (using a buildfarm to produce the packages). The question is, Without modifying or changing the current python or distutils, is there a way to "remove" this option that distutils is picking up, so I can have the python setup.py process build the module appropriately (i.e. without the pic32 option)?
Thanks

Do not compile with that gcc. -xcode=pic32 is Sun Studio complier command line parameter. It will lead to linking problems too, even if you compile OK. Compile with SUN CoolTools gcc which can understand such parameter, or use Oracle Solaris Studio for SPARC.
Some hints:
GCC produce very slow code for SPARC, that's for why SUN created Cool Tools.
You haven't to remove -xcode=pic32, but change for -m32 -fpic, when you insist on gcc-4.7
To get mature setup of OSS tool I'm using pkgsrc compiling with Studio Express to particular CPU (-xtarget=native)

You may also find luck by setting the follow env vars:
export CC=$gcc_dir_path # Example: /usr/bin/gcc
export CXX=$gxx_dir_path # Example: /usr/bin/g++
export CFLAGS=''
export CPPFLAGS=''
export CXXFLAGS=''
export LDFLAGS=''
Note: There is a difference between unset env var, and set-as-empty env var. I had build bugs with Python packages when my *FLAGS env vars were unset. (Calling gcc with option -xO2 was the cause.) Setting as empty did the trick.

Mac 10.6 Universal Binary scipy: cephes/specfun "_aswfa_" symbol not found

I can't get scipy to function in 32 bit mode when compiled as a i386/x86_64 universal binary, and executed on my 64 bit 10.6.2 MacPro1,1.
My python setup
With the help of this answer, I built a 32/64 bit intel universal binary of python 2.6.4 with the intention of using the arch command to select between the architectures. (I managed to make some universal binaries of a few libraries I wanted using lipo.) That all works. I then installed scipy according to the instructions on hyperjeff's article, only with more up-to-date numpy (1.4.0) and skipping the bit about moving numpy aside briefly during the installation of scipy.
Now, everything except scipy seems to be working as far as I can tell, and I can indeed select between 32 and 64 bit mode using arch -i386 python and arch -x86_64 python.
The error
Scipy complains in 32 bit mode:
$ arch -x86_64 python -c "import scipy.interpolate; print 'success'"
success
$ arch -i386 python -c "import scipy.interpolate; print 'success'"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/interpolate/__init__.py", line 7, in <module>
from interpolate import *
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/interpolate/interpolate.py", line 13, in <module>
import scipy.special as spec
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/__init__.py", line 8, in <module>
from basic import *
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/basic.py", line 8, in <module>
from _cephes import *
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so, 2): Symbol not found: _aswfa_
Referenced from: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so
Expected in: flat namespace
in /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so
Attempt at tracking down the problem
It looks like scipy.interpolate imports something called _cephes, which looks for a symbol called _aswfa_ but can't find it in 32 bit mode. Browsing through scipy's source, I find an ASWFA subroutine in specfun.f. The only scipy product file with a similar name is specfun.so, but both that and _cephes.so appear to be universal binaries:
$ cd /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/
$ file _cephes.so specfun.so
_cephes.so: Mach-O universal binary with 2 architectures
_cephes.so (for architecture i386): Mach-O bundle i386
_cephes.so (for architecture x86_64): Mach-O 64-bit bundle x86_64
specfun.so: Mach-O universal binary with 2 architectures
specfun.so (for architecture i386): Mach-O bundle i386
specfun.so (for architecture x86_64): Mach-O 64-bit bundle x86_64
Ho hum. I'm stuck. Things I may try but haven't figured out how yet include compiling specfun.so myself manually, somehow.
I would imagine that scipy isn't broken for all 32 bit machines, so I guess something is wrong with the way I've installed it, but I can't figure out what.
I don't really expect a full answer given my fairly unique (?) setup, but if anyone has any clues that might point me in the right direction, they'd be greatly appreciated.
(edit) More details to address questions:
I'm using gfortran (GNU Fortran from GCC 4.2.1 Apple Inc. build 5646).
Python 2.6.4 was installed more-or-less like so:
cd /tmp
curl -O http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2
tar xf Python-2.6.4.tar.bz2
cd Python-2.6.4
# Now replace buggy pythonw.c file with one that supports the "arch" command:
curl http://bugs.python.org/file14949/pythonw.c | sed s/2.7/2.6/ > Mac/Tools/pythonw.c
./configure --enable-framework=/Library/Frameworks --enable-universalsdk=/ --with-universal-archs=intel
make -j4
sudo make frameworkinstall
Scipy 0.7.1 was installed pretty much as described as here, but it boils down to a simple sudo python setup.py install.
It would indeed appear that the symbol is undefined in the i386 architecture if you look at the _cephes library with nm, as suggested by David Cournapeau:
$ nm -arch x86_64 /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so | grep _aswfa_
00000000000d4950 T _aswfa_
000000000011e4b0 d _oblate_aswfa_data
000000000011e510 d _oblate_aswfa_nocv_data
(snip)
$ nm -arch i386 /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so | grep _aswfa_
U _aswfa_
0002e96c d _oblate_aswfa_data
0002e99c d _oblate_aswfa_nocv_data
(snip)
however, I can't yet explain its absence.

Have you tried using scipy compiled using macports?
sudo port install scipy +universal
(of course you must have the rest of the chain, python, py26-numpycompiled with the same option)
I get:
$ arch -x86_64 /opt/local/bin/python -c "import scipy.interpolate; print 'success'"
success
$ arch -i386 /opt/local/bin/python -c "import scipy.interpolate; print 'success'"
success
you may then use the setting and knowledge that the macports maintainers used to make your own compilation.

How did you install scipy, for which python version, and with which fortran compiler ?
You may also want to check that the missing symbol is indeed in both archs (I don't remember off-hand where the function is, but you should be able to find ti by yourself pretty easily using a combination of nm/otool).

How to link C lib against python for embedding under Windows?

I am working on an application written in C. One part of the application should embed python and there is my current problem. I try to link my source to the Python library but it does not work.
As I use MinGW I have created the python26.a file from python26.lib with dlltool and put the *.a file in C:/Program Files (x86)/python/2.6/libs.
Therefore, I compile the file with this command:
gcc -shared -o mod_python.dll mod_python.o "-LC:\Program Files (x86)\python\2.6\libs" -lpython26 -Wl,--out-implib,libmod_python.a -Wl,--output-def,mod_python.def
and I get those errors:
Creating library file: libmod_python.a
mod_python.o: In function `module_init':
mod_python.c:34: undefined reference to `__imp__Py_Initialize'
mod_python.c:35: undefined reference to `__imp__PyEval_InitThreads'
... and so on ...
My Python "root" folder is C:\Program Files (x86)\python\2.6
The Devsystem is a Windows Server 2008
GCC Information: Reading specs from C:/Program Files (x86)/MinGW/bin/../lib/gcc/mingw32/3.4.5/specs
Configured with: ../gcc-3.4.5-20060117-3/configure --with-gcc --with-gnu-ld --with-gnu-as --host=mingw32 --target=mingw32 --prefix=/mingw --enable-threads --disable-nls --enable-languages=c,c++,f77,ada,objc,java --disable-win32-registry --disable-shared --enable-sjlj-exceptions --enable-libgcj --disable-java-awt --without-x --enable-java-gc=boehm --disable-libgcj-debug --enable-interpreter --enable-hash-synchronization --enable-libstdcxx-debug
Thread model: win32
gcc version 3.4.5 (mingw-vista special r3)
What I do wrong? How I get it compiled and linked :-)?
Cheers, gregor
Edit:
I forgot to write information about my Python installation: It's the official python.org installation 2.6.1
... and how I created the python.a file:
dlltool -z python.def --export-all-symbols -v c:\windows\system32\python26.dll
dlltool --dllname c:\Windows\system32\python26.dll --def python.def -v --output-lib python26.a

Well on Windows the python distribution comes already with a libpython26.a in the libs subdir so there is no need to generate .a files using dll tools.
I did try a little example with a single C file toto.c:
gcc -shared -o ./toto.dll ./toto.c -I/Python26/include/ -L/Python26/libs -lpython26
And it works like a charm. Hope it will help :-)

Python (at least my distribution) comes with a "python-config" program that automatically creates the correct compiler and linker options for various situations. However, I have never used it on Windows. Perhaps this tool can help you though?

IIRC, dlltool does not always work. Having python 2.6 + Wow makes things even more less likely to work. For numpy, here is how I did it. Basically, I use obdump.exe to build the table from the dll, which I parse to generate the .def. You should check whether your missing symbols are in the .def, or otherwise it won't work.