I'm working on a project built with Cython and have been having some interesting installation issues on some particular systems. The problem boils down to either a mismatch in Mac OS versions, or a mismatch in my understanding of Mac OS versions.
When installing a Cython library on my system, the compilation log shows that I am using Mac OS version 10.12, even though my system tells me I am on 10.13. An excerpt of the compilation log is below:
creating build/temp.macosx-10.12-x86_64-3.6
This is not a problem for me, but another user is having trouble where their system release on 10.13, but their compilation is showing 10.7 (which requires some other work arounds).
Is it possible to update this build system? Or is there something about Mac OS that I'm misunderstanding?
Is there a way to tell from a Python script which build libraries will be used?
Thank you!
Edit:
Output of platform information:
>>> python -c "import platform; print(platform.mac_ver())"
('10.13.6', ('', '', ''), 'x86_64')
>>> python -c "import distutils.util; print(distutils.util.get_platform())"
macosx-10.12-x86_64
My context
A similar issue came up with Big Sur (macOS 10.16), python3.6, and cython.
Checking the build directory of cython (.pyxbld) I find temp.macOSX-10.7 builds and files, which apparently is incompatible with 10.16.
Some Explanations why this is going on
Under the hood cython uses distutils to determine the platform version (as opposed to other things). https://github.com/python/cpython/blob/master/Lib/distutils/util.py So then what's up with distutils?
After a lot of searching (including this post), I came across a really interesting and good post (although from 2014) that explains this phenomenon.
https://lepture.com/en/2014/python-on-a-hard-wheel
Basically, it has to do with how distutils was build and that it has hard-coded platform tags... (this also applies to the wheel files).
Solution (kinda)
It seems that the solution is to :
reinstall python (and then all packages)
make sure that python -c "import distutils.util; print(distutils.util.get_platform())" macosx-10.12-x86_64 outputs a correct version.
Edit
Apparently On Big Sur with Anaconda this solution does not work yet... (work in progress will update if solution on Anaconda will work) [Fri 20 Nov 2020]
Related
I created an executable out of a simple Python script, using pyinstaller on Ubuntu 18.04, and tested it in a different computer (also with Ubutnu 18) and worked perfectly.
However when trying the same with a more complex script (more library imports) the executable fails in the other computer with the error
ImportError: /lob/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.25' not found
This can't be a Python incompatibility (see https://github.com/pyinstaller/pyinstaller/issues/4758), as the other script did work fine. So it most probably is based on some of the libraries the second script imports.
How can I include the libraries imported in the executable made by Pyinstaller (if that is even the origin of this error)?
Solution A
I have not confirmed this solution, but it sometimes helps. Delete directories ./build and ./dist, then try creating the executable again with pyinstaller.
Solution B
The solution, for me at least, is to build your executable on an older version of your OS.
I was seeing the same error.
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/_MEIjdcWu4/./libX11.so.6)
[32614] Failed to execute script 'test_executable' due to unhandled exception!
I built my exectuable with Pyinstaller on Ubuntu 22.04. Then I copied and ran the executable on the older Ubuntu 20.04 and the error was encountered.
Per the comment below, this might be a compatibility issue where the executable built on a newer OS is not compatible with older OSs.
"For what is worth, the issue could be that the libraries bundled with
the built program conflict with the system libraries, preventing the
DRI driver from properly loading.
The culprit could be either standard c/c++ libraries (libgcc_s.so.1,
libstdc++.so.6) or maybe the X11 libraries (libX11.so.6, libXau.so.6,
libXdmcp.so.6, libXext.so.6, `libXrender.so.1˙). Perhaps more likely
former than the latter.
For example, if libstdc++.so.6 on the build system is older than the
one used by the target system, then the non-bundled libraries will
fail to load due to missing symbols (which are present in the newer,
system version of the library, but not in the bundled one). This is
actually quite a common issue with binary-only software on linux,
especially on more bleeding edge distributions. In those cases,
removing the bundled version of the offending library may help.
(You have a similar issue with system libgvfsdbus.so, which is missing
a symbol that is not available in the bundled libglib-2.0.so.0, which
is probably older than the glib library available on the system)."
Source:
https://github.com/cryptoadvance/specter-desktop/issues/373#issuecomment-694476451
I'm running clang on mac to compile a c file created by running a very simple program through cython, but the compiler always give me a "Python.h not found" fatal error. I've tried every solution I could find, reinstalling python 3.9, using the -I/path/to/headerfile method, and rewriting the include statement in the code to contain the full filepath, but nothing has worked. When I do include the full filepath, I get fatal error: 'cpython/initconfig.h' file not found. What could the issue possibly be, and how would I fix it? The program itself works fine in the standard python interpreter, pyinstaller, and nuitka.
Today I needed to embed Python into a C application developed on XCode with Clang compiler, and I've met the same problems and solved them. Let me share the tips:
1. Python not found error
You should install a Python framework to your MacOS, and attach it into Frameworks and Libraries project settings tab. Also, specify it's path in Framework Search Paths inside Build Settings tab if XCode hasn't done it automatically.
2. Undefined symbols / Implicit declaration errors
Verify you're using a modern version of Python framework. Versions prior to 3.6 had different ABI which leads to the mentioned errors.
3. cpython/initconfig.h file not found
Versions prior to ~3.9 had some header inclusion problem for the cpython/initconfig.h file (issues 40642, 39026).
Check the builds of Python that your OS currently has. I found Pythons installed in the following paths:
# Default 2.7 MacOSX installation is here
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/
/Library/Frameworks/
Or install a recent Python version. I recommend using "universal binary" build, see the docs.
I'm running into a cython-unicode problem which I hope you can help me with -
I'm trying to compile with cython a python module which works alot with unicode objects.
I'm working on Ubuntu 12.04 32bit, (on a VM on a Win7 64bit host if it has any importance), I'm using Py2.5.4 (can't change that) which I've installed side-by-side with the provided Py2.7.3 (and soft-linked /usr/bin/python to /usr/bin/python2.5), and Cython 19.02 (current one). I'm compiling with no special flags.
Now, as compiling goes seemingly OK, when I open a Py (same 2.5.4) and try to import the resulting. SO file, I get a PyUnicodeUCS2 type error. That is supposed to mean, as I understand, that the SO was compiled on a python configured with UCS4, and imported to python configured with UCS2. But It's the same python I compiled on and imported to.
for compilation I use $ python setup.py bild_ext --inplace where setup.py uses distutlis.extension.Extension and Cython.Distutile.build_ext .
Can someone enlighten me with what's going on and what I can do with it?
I'll be happy to provide more details if I wasn't clear enough.
Thanks a lot..
I'm using CXFreeze with PySide (QT). I get an error:
cx_Freeze: Python error in main script.
myscript.py line 33, in
File ExtensionLoader_Pyside_QtGUI.py, line 11, in
Import Error: DLL load failed: The specified module could not be found
When running a fresh install of Windows server 2008.
I'm running the frozen EXE package (with the folder). It seems to work on my own system and other systems. What might be the issue?
After reading, online, I tried to replace the Qt4Gui file, but this didn't solve the issue.
Python version is 2.7
Based on your Import Error: DLL load failed it is most likely an installation issue causing the missing DLL. To figure our exactly which DLL you are missing, use http://www.dependencywalker.com/ Run the .exe and open the .pyd file for File ExtensionLoader_Pyside_QtGUI.py and it will show you exactly which DLL's are missing and more importantly the locations where they should be. You can probably then track down the missing DLL online.
there are known issues with pyside 1.2.0 and cxFreeze. All should be fixed in development version (available on git repo). Please build the PySide from latest sources yourself or wait for PySide version 1.2.1. Build instructions are here [1].
[1] https://github.com/PySide/pyside-setup#building-pyside-on-a-windows-system
I used Py2exe instead of CXFreeze and it worked perfectly.
Also, apparently Python requires the MS Visual C++ Dependency Files:
http://www.microsoft.com/en-us/download/details.aspx?id=29
So any bundling needs that as well, if it's a fresh install. (Although I think they are now bundled with newer Windows versions.)
Other Notes:
In my experience, sometimes you should try CXFreeze, Py2EXE and PyInstaller quickly and see if one works best. As ideal as CXFreeze is re: cross platform, it just isn't going to happen perfectly.
Also, while I don't know if this was a factor, I set up a Windows 2000 Pro virtual machine and ran Py2exe on that. That was to ensure compatibility for all older Windows versions, and seemed to work well. (NOTE: Many things won't even run on Win2000 anymore so be careful that your other tools and libraries will run on it.)
Finally, be extra careful to match the bit level (32 vs 64) of all your libraries, and your Python install itself. If you have 32-bit python, ensure that your PySide, CXFreeze and any other libraries you use are 32-bit. (Or 64-bit if you're using 64-bit python.)
I am trying to compile a package on Mac OSX 10.6.5. The package's install script relies on distutils. The problem is that the computer's default gcc is version 4.2 (I determined this by just running gcc --version in a terminal window) but when I run 'python setup.py build', I see from the output that the distutils is choosing gcc-4.0 instead of 4.2 This is a big problem because the code I am using require gcc >= 4.2. I do not have admin rights on this machine, so as a workaroud, I created some symlinks that send gcc-4.0 to gcc-4.2. The result is the code compiles, but the generated .so files do not work (when I try to import them in python, I get errors complaining about a missing init function in the shared object).
I have tried compiling this code on a different mac (10.6.6) and it works like a charm: distutils chooses 4.2 without being forced to do so and I can import the generated shared object without a problem. So, what I would like to do is to compile the code on my computer without having to do this symlink trickery...I just want distutils to choose 4.2 automatically as it should. I have tried taking the .so files that compile properly and transferring them to my computer, but that fails for a number of reasons (they are linked against libraries that are not present on my machine/are a different version that those installed).
Does anyone have any advice here?
Thanks,
Josh
To force distutils to use a separate compiler, you can redefine a few variables via the environment. First, find out what distutils is using as defaults:
>>> from distutils import sysconfig
>>> sysconfig.get_config_var('LDSHARED')
'gcc-4.0 -Wl,-F. -bundle -undefined dynamic_lookup'
>>> sysconfig.get_config_var('CC')
'gcc-4.0'
Next you need to redefine those, substituting in the version of gcc you'd like to use:
% LDSHARED="gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup" CC=gcc-4.2 \
/usr/bin/python setup.py build_ext
Keep in mind that the sysconfig defaults are pulled from the Makefile which was originally used to compile python, so fudging with them may produce unintended results:
>>> path = sysconfig.get_python_lib(plat_specific=1, standard_lib=1)
>>> os.path.join(path, 'config', 'Makefile')
'/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config/Makefile'
Have you tried setting a custom CC environment variable?
CC=gcc-4.2 python setup.py build
You don't say which versions of Python you are using but chances are that the default Python on the machine you are using is not the Apple-supplied Python 2.6.1, more likely one installed from python.org. Try running the build script with /usr/bin/python2.6 which should be the Apple-supplied Python 2.6; that does use gcc-4.2.
When you use Python's Distutils (typically by running a setup.py script or using easy_install), Distutils tries to make sure that any C extension modules in the package you are installing will be compiled with the same compiler version and with compatible compilation options (CPU archs, ABI, etc) as was used to build Python itself. Traditionally, most python.org installers for OS X provide a universal Python that works on multiple versions of OS X and multiple CPU archs. That's why they are built with gcc-4.0. If you need to use gcc-4.2 for some other library, then the safest thing is to use a Python that was built with gcc-4.2. The Apple-supplied system Pythons on OS X 10.6 are so built. Also, for the most recent releases of Python (2.7.1 and 3.2), python.org provides a second OS X installer variant for OS X 10.6 that is also built with gcc-4.2. But if you do not have admin access to your machine, that's not an option anyway.
You can see which python is being used by default by:
which python
You can either adjust your distutils.cfg (see here) or you can pass command line arguments as --compiler=gcc42 (not sure about the last one).