Develop Reference

Cython compilation using wrong Mac OS libraries

I'm working on a project built with Cython and have been having some interesting installation issues on some particular systems. The problem boils down to either a mismatch in Mac OS versions, or a mismatch in my understanding of Mac OS versions.
When installing a Cython library on my system, the compilation log shows that I am using Mac OS version 10.12, even though my system tells me I am on 10.13. An excerpt of the compilation log is below:
creating build/temp.macosx-10.12-x86_64-3.6
This is not a problem for me, but another user is having trouble where their system release on 10.13, but their compilation is showing 10.7 (which requires some other work arounds).
Is it possible to update this build system? Or is there something about Mac OS that I'm misunderstanding?
Is there a way to tell from a Python script which build libraries will be used?
Thank you!
Edit:
Output of platform information:
>>> python -c "import platform; print(platform.mac_ver())"
('10.13.6', ('', '', ''), 'x86_64')
>>> python -c "import distutils.util; print(distutils.util.get_platform())"
macosx-10.12-x86_64

My context
A similar issue came up with Big Sur (macOS 10.16), python3.6, and cython.
Checking the build directory of cython (.pyxbld) I find temp.macOSX-10.7 builds and files, which apparently is incompatible with 10.16.
Some Explanations why this is going on
Under the hood cython uses distutils to determine the platform version (as opposed to other things). https://github.com/python/cpython/blob/master/Lib/distutils/util.py So then what's up with distutils?
After a lot of searching (including this post), I came across a really interesting and good post (although from 2014) that explains this phenomenon.
https://lepture.com/en/2014/python-on-a-hard-wheel
Basically, it has to do with how distutils was build and that it has hard-coded platform tags... (this also applies to the wheel files).
Solution (kinda)
It seems that the solution is to :
reinstall python (and then all packages)
make sure that python -c "import distutils.util; print(distutils.util.get_platform())" macosx-10.12-x86_64 outputs a correct version.
Edit
Apparently On Big Sur with Anaconda this solution does not work yet... (work in progress will update if solution on Anaconda will work) [Fri 20 Nov 2020]

Installing graph-tool on Mac OS X (10.7) - already have Boost installed, but keep getting this error

I've been stuck on this issue for a while now. I'm trying to install graph-tool - http://graph-tool.skewed.de/download#macos - and I have the prereqs from following these steps, which the graph-tool site links to: https://gist.github.com/openp2pdesign/8864593
Instead of brew install, which didn't seem to give me all the files, I went to Boost's official site and downloaded from there properly, following these steps: http://www.boost.org/doc/libs/1_41_0/more/getting_started/unix-variants.html It's mainly getting a tar file and untarring it.
I then put my boost install here:
/usr/local/boost_1_55_0
I did a small C++ example and confirmed Boost works (using "Build a Simple Program Using Boost" from http://www.boost.org/doc/libs/1_41_0/more/getting_started/unix-variants.html.
Now the meat of the problem: trying to install graph-tool. In the very last step, I do
./configure PYTHON_EXTRA_LDFLAGS="-L/usr/local/bin"
(The PYTHON_EXTRA_LDFLAGS="-L/usr/local/bin" just makes the configure script find Python alright.)
But I get this error. (It finds Python fine, but not boost!)
...
================
Detecting python
================
checking for a Python interpreter with version >= 2.6... python
checking for python... /Users/daze/Library/Enthought/Canopy_64bit/User/bin/python
checking for python version... 2.7
checking for python platform... darwin
checking for python script directory... ${prefix}/lib/python2.7/site-packages
checking for python extension module directory... ${exec_prefix}/lib/python2.7/site-packages
checking for python2.7... (cached) /Users/daze/Library/Enthought/Canopy_64bit/User/bin/python
checking for a version of Python >= '2.1.0'... yes
checking for a version of Python == '2.7.3'... yes
checking for the distutils Python package... yes
checking for Python include path... -I/Applications/Canopy.app/appdata/canopy-1.1.0.1371.macosx-x86_64/Canopy.app/Contents/include/python2.7
checking for Python library path... -L/Applications/Canopy.app/appdata/canopy-1.1.0.1371.macosx-x86_64/Canopy.app/Contents/lib/python2.7/config -lpython2.7
checking for Python site-packages path... /Users/daze/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages
checking python extra libraries... -ldl -framework CoreFoundation
checking python extra linking flags... -L/usr/local/bin
checking consistency of all components of python development environment... yes
graph-tool will be installed at: /Users/daze/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages
===========================
Using python version: 2.7.3
===========================
checking for boostlib >= 1.38.0... configure: error: We could not detect the boost
libraries (version 1.38 or higher). If you have a staged boost library (still not installed)
please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option.
If you are sure you have boost installed, then check your version number looking in
<boost/version.hpp>. See http://randspringer.de/boost for more documentation.
Attempt 2: I then tried setting BOOST_ROOT properly:
In my ~/.bash_profile:
export BOOST_ROOT="/usr/local/boost_1_55_0"
But it still did no good, so I unset that.
Attempt 3: I then tried explicitly specifying where boost is installed:
./configure --with-boost="/usr/local/boost_1_55_0" PYTHON_EXTRA_LDFLAGS="-L/usr/local/bin"
But it still can't find boost, and yields that same error in the end of "We could not detect the boost libraries (version 1.38 or higher)."
It's been bugging me all day. I've read carefully, and went to the randspringer.de/boost site and saw this in the FAQ - http://www.randspringer.de/boost/faq.html#id2514912:
Q: I do not understand the configure error message
At configure time I get:
checking for boostlib >= 1.33... configure: error: We could not detect
the boost libraries (version 1.33 or higher). If you have a staged
boost library (still not installed) please specify $BOOST_ROOT in your
environment and do not give a PATH to --with-boost option. If you are
sure you have boost installed, then check your version number looking
in . See http://randspringer.de/boost for more
documentation.
I don't know if I use a staged version of boost. What is it and what
can I do ?
A: If you did not compile Boost by yourself you don't have a staged
version and you don't have to set BOOST_ROOT. Look here for an
explanation of different kind of installations.
If you are sure you have Boost installed then specify the directory
with
./configure --with-boost=your-boost-directory.
If it still does not work, please check the version number in
boost/version.hpp and compare it with the version requested in
configure.ac.
And I don't know what to see when comparing version numbers. There's nothing I found interesting there.
Hoping someone has at least an idea on what other approaches to take.

Hooray, my first chance to give back to Stack Overflow! I've been dealing with this issue myself the past 2 days.
Solution
Upgrade clang via Xcode
Make a symlink to boost that includes the version number
/usr/local/include/boost-1_55.0 -> ../Cellar/boost/1.55.0/include/boost
(included because I installed Boost using Brew and had this issue)
Edit the generation of CXXFLAGS in configure so that it looks like this:
old_cxxflags="$CXXFLAGS"
CXXFLAGS="${CXXFLAGS} -std=gnu++11 -stdlib=libc++"
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C++ compiler supports -std=gnu++11" >&5
$as_echo_n "checking whether C++ compiler supports -std=gnu++11... " >&6; }
Run
./configure --disable-sparsehash CXX="/usr/bin/clang++" PYTHON_EXTRA_LDFLAGS="-L/usr/local/bin"
Versions
OS: Mac OS X 10.8.5
Clang: Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin12.5.0
Thread model: posix
Graph-tool: 2.2.29.1
Boost: 1.55.0
Explanation
If you go through the configure code and try and compile the confdefs.h files made in configure, you'll see clang error out upon encountering the -Wno-unused-local-typedefs flag. This is the actual cause of the "We could not detect the boost libraries (version 1.33 or higher)" error, not the fact that it can't find the boost files. This issue is fixed with newer versions of clang.
The configure test for version number is goofy. It expects the boost include directory to contain the version number.
While running make, you may run into the following errors:
./../graph_adjacency.hh:26:10: fatal error: 'tuple' file not found
This is caused by referencing the wrong standard library [1]
./../graph_adaptor.hh:655:39: error: expected ';' in 'for' statement specifier
for(typeof(removed_edges.begin()) iter = removed_edges.begin();
./../graph_adaptor.hh:655:39: error: use of undeclared identifier 'tier'
This is caused by referencing the wrong C++ standard (c++11 instead of gnu++11)
References
[1] No member named 'forward' in namespace 'std'
[2] I'm having some trouble with C++11 in Xcode

I think that you're currently pointing --with-boost to the boost parent directory, not the boost libraries.
Try
./configure --with-boost="/usr/local/boost_1_55_0/libs/" PYTHON_EXTRA_LDFLAGS="-L/usr/local/bin"

Building LLVM fails with empty error message

I'm trying to build LLVM 3.1 and Clang 3.1. I followed the Getting Started guide from Clang's website- check out the repositories in the requisite places, get Python, etc.
If I have Python 3.3 installed, it gives a Python semantic error- from main import main, no module called main. If I have Python 2.7 installed, it gives
CMake Error at CMakeLists.txt:307 (message):
Unexpected failure executing llvm-build:
Configuring incomplete, errors occurred!
This is most unhelpful. Any suggestions as to what I can do to build LLVM and Clang on Windows, or at least attempt to determine what the problem is?
Just as a note, I am attempting to build with Visual Studio 2012, which is officially supported by CMake but was not released when LLVM 3.1 and Clang 3.1 were created.
Edit: Here are the requisite lines from CMakeLists.txt
message(STATUS "Constructing LLVMBuild project information")
execute_process(
COMMAND ${PYTHON_EXECUTABLE} ${LLVMBUILDTOOL}
--native-target "${LLVM_NATIVE_ARCH}"
--enable-targets "${LLVM_TARGETS_TO_BUILD}"
--enable-optional-components "${LLVMOPTIONALCOMPONENTS}"
--write-library-table ${LLVMCONFIGLIBRARYDEPENDENCIESINC}
--write-cmake-fragment ${LLVMBUILDCMAKEFRAG}
ERROR_VARIABLE LLVMBUILDOUTPUT
ERROR_VARIABLE LLVMBUILDERRORS
OUTPUT_STRIP_TRAILING_WHITESPACE
ERROR_STRIP_TRAILING_WHITESPACE
RESULT_VARIABLE LLVMBUILDRESULT)
# On Win32, CMake doesn't properly handle piping the default output/error
# streams into the GUI console. So, we explicitly catch and report them.
if( NOT "${LLVMBUILDOUTPUT}" STREQUAL "")
message(STATUS "llvm-build output: ${LLVMBUILDOUTPUT}")
endif()
if( NOT "${LLVMBUILDRESULT}" STREQUAL "0" )
message(FATAL_ERROR
"Unexpected failure executing llvm-build: ${LLVMBUILDERRORS}")
endif()
Here, you can see that ${LLVMBUILDRESULT} is not good, but ${LLVMBUILDERRORS} which is obviously supposed to contain the build errors does not contain anything.

The problem is two-fold:
First, LLVM requires Python 2.x. If you try to force it to use 3.3, it will fail. The docs probably should say something about this (and I couldn't find any reference to it when I looked), but at this stage it's still pretty common for projects to just say, e.g., "Python" or "Python 2.6+" when they mean "2.6-2.7 but not 3.x".
Second, like most configuration/build tools, CMake only detects dependency changes in the code, not in your system configuration. So if you try to build, then change your system, then try to build again, it won't notice the change and adjust its configuration accordingly. It was already configured to use Python 3.3, and it didn't notice you'd replaced it with Python 2.7, hence the problem. You have to force it to reconfigure—which you can do by clearing the cache, but the simplest and cleanest answer is to just untar/git/whatever yourself a clean directory and start over. (Or, if you can build out-of-tree, as you can with many projects, just wipe out the build directory and start over.)
Finally, according to the docs, you really don't need Python to build llvm and clang, unless you want to run the tests. So, unless the docs are wrong (which is of course possible), if you had just done a clean build with no Python at all, it would have worked. The reason it didn't work when you tried it is the same reason it didn't work after you installed 2.7: You were in a partially-configured state, it thought it had Python, and therefore it insisted on using it.

I used something like
"c:\Program Files (x86)\CMake\bin\cmake.exe" -G "Visual Studio 12 2013" -DPYTHON_EXECUTABLE="C:\Python27\python27.exe" -DLLVM_TARGETS_TO_BUILD="X86" ..\llvm-3.5.0.src
and it worked for me. In my case cmake was not finding python so I had to use
-DPYTHON_EXECUTABLE="C:\Python27\python27.exe"
Notice that I didn't override my system folders then somebody with the same problem should use something different for each folder of the command.

Apparently, this is a CMake fail primarily, it would have succeeded if I had cleared the cache after installing Python 2.7

Python distutils not using correct version of gcc

I am trying to compile a package on Mac OSX 10.6.5. The package's install script relies on distutils. The problem is that the computer's default gcc is version 4.2 (I determined this by just running gcc --version in a terminal window) but when I run 'python setup.py build', I see from the output that the distutils is choosing gcc-4.0 instead of 4.2 This is a big problem because the code I am using require gcc >= 4.2. I do not have admin rights on this machine, so as a workaroud, I created some symlinks that send gcc-4.0 to gcc-4.2. The result is the code compiles, but the generated .so files do not work (when I try to import them in python, I get errors complaining about a missing init function in the shared object).
I have tried compiling this code on a different mac (10.6.6) and it works like a charm: distutils chooses 4.2 without being forced to do so and I can import the generated shared object without a problem. So, what I would like to do is to compile the code on my computer without having to do this symlink trickery...I just want distutils to choose 4.2 automatically as it should. I have tried taking the .so files that compile properly and transferring them to my computer, but that fails for a number of reasons (they are linked against libraries that are not present on my machine/are a different version that those installed).
Does anyone have any advice here?
Thanks,
Josh

To force distutils to use a separate compiler, you can redefine a few variables via the environment. First, find out what distutils is using as defaults:
>>> from distutils import sysconfig
>>> sysconfig.get_config_var('LDSHARED')
'gcc-4.0 -Wl,-F. -bundle -undefined dynamic_lookup'
>>> sysconfig.get_config_var('CC')
'gcc-4.0'
Next you need to redefine those, substituting in the version of gcc you'd like to use:
% LDSHARED="gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup" CC=gcc-4.2 \
/usr/bin/python setup.py build_ext
Keep in mind that the sysconfig defaults are pulled from the Makefile which was originally used to compile python, so fudging with them may produce unintended results:
>>> path = sysconfig.get_python_lib(plat_specific=1, standard_lib=1)
>>> os.path.join(path, 'config', 'Makefile')
'/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config/Makefile'

Have you tried setting a custom CC environment variable?
CC=gcc-4.2 python setup.py build

You don't say which versions of Python you are using but chances are that the default Python on the machine you are using is not the Apple-supplied Python 2.6.1, more likely one installed from python.org. Try running the build script with /usr/bin/python2.6 which should be the Apple-supplied Python 2.6; that does use gcc-4.2.
When you use Python's Distutils (typically by running a setup.py script or using easy_install), Distutils tries to make sure that any C extension modules in the package you are installing will be compiled with the same compiler version and with compatible compilation options (CPU archs, ABI, etc) as was used to build Python itself. Traditionally, most python.org installers for OS X provide a universal Python that works on multiple versions of OS X and multiple CPU archs. That's why they are built with gcc-4.0. If you need to use gcc-4.2 for some other library, then the safest thing is to use a Python that was built with gcc-4.2. The Apple-supplied system Pythons on OS X 10.6 are so built. Also, for the most recent releases of Python (2.7.1 and 3.2), python.org provides a second OS X installer variant for OS X 10.6 that is also built with gcc-4.2. But if you do not have admin access to your machine, that's not an option anyway.
You can see which python is being used by default by:
which python

You can either adjust your distutils.cfg (see here) or you can pass command line arguments as --compiler=gcc42 (not sure about the last one).

How to install 64-bit Python on Solaris?

I am trying to install Python 2.6 on Solaris by building the source on Solaris machine. I installed one this way and it appears that it is 32-bit. I downloaded some source tar ball as Linux or Unix for this purpose. Everything works well but I need 64-bit Python.
I looked up the Python download site and there is no separate installation for a 64-bit Python.
That makes me think that there must be some option while running configure and/or install commands to install Python. I tried reading README.txt of the installation but could not find any info. I am very new to installations on "Unix" like systems.
How can I install 64-bit Python on Solaris?

I would strongly suggest seeing if you can get away with the 32 bit version of Python. If your new to compiling stuff on Solaris, this will save you many headaches. However, it is possible, and I do have a working 64 bit version of Python. I'm using cc: Sun C 5.8 2005/10/13 to compile. Additionally, I've already compiled 64-bit version of readline and ncurses.
My configure line looks like this:
../Python-2.6.1/configure CCSHARED="-KPIC" LDSHARED="cc -xarch=generic64 -G -KPIC" LDFLAGS="-xarch=generic64 -L/opt/tools/lib -R/opt/tools/lib -L/opt/tools/ssl/lib -ltermcap -lz -R $ORIGIN/../lib" CC="cc" CPP="cc -xarch=generic64 -E -I/opt/tools/include -I/opt/tools/include/ncurses -I/opt/tools/include/readline" BASECFLAGS="-xarch=generic64 -I/opt/tools/include -I/opt/tools/include/ncurses" OPT="-xO5" CFLAGS="-xarch=generic64 -I/opt/tools/include -I/opt/tools/include/ncurses -I/opt/tools/include/readline" CXX="CC -xarch=generic64 -I/opt/tools/include -I/opt/tools/include/ncurses" --prefix=/opt/tools/python-2.6.1 --enable-64-bit --without-gcc --disable-ipv6 --with-ssl=openssl --with-ncurses --with-readline
Additionally, I modified these two lines in Modules/Setup.local to include the required locations:
readline readline.c -I/opt/tools/include/readline -L/opt/tools/lib -lreadline -ltermcap
_ssl _ssl.c -I/opt/tools/ssl/include -L/opt/tools/ssl/lib -lssl -lcrypto
Now, just pray you don't need to compile in some Sybase bindings or some other 64-bit libraries.

It's currently an acknowledged bug that Solaris 64-bit support is suboptimal, but that bug report looks to contain some flags that you might want to use. See also this mailing list posting.