Develop Reference

Nuitka - compiling module including precompiled .so libraries

I need to use both nuitka (code obfuscation) and numba (code optimization), but nuitka crashes when executing #njit (understandable).
My idea is to precompile some code with numba using it's AOT compiling feature, so nuitka can use it later when doing it's thing.
Issues I encountered :
when leaving only the precompiled .so file, nuitka ignores it => ImportError
when leaving source .py file with .so file, nuitka uses the .py file despite seeing both files
it displays a warning about finding both files
I tried adding --no-prefer-source-code (which should be default) but it still uses the .py file instead of the precompiled .so
Did anyone at some point managed to import external .so modules in a nuitka process ?
NB: I know, I can just copy the precompiled numba .so file and get it to be imported 'normally' by the compiled nuitka .so, but it's not scalable if I add more precompiled numba code in the futur.
The idea would be to include the precompiled numba .so inside nuitka's .so (is it even possible ?)

Mac OSX CPython ctypes library functionality and correct folder location

I have a question regarding the CPython open-source code. I noticed some CPython libraries are compiled to a directory listed below on a Mac OSX operating system.
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload
Regarding the CPython ctypes library. Is this the library that is compiled to the file named _ctypes.cpython-38-darwin.so? If so is this the ctypes source code for the library in CPython located at tree/master/Lib/ctypes on Github. I am asking this because I am trying to modify the correct source code and learn some things regarding CPython. I am trying to determine if the cyptyes library simply is just loading native frameworks and libraries through this, or implementing ctypes through something else. Or is it implemented through something like the types.py library?
Reference tree/master/Lib/ctypes/util.py
# Line 330
if sys.platform == "darwin":
print(cdll.LoadLibrary("libm.dylib"))
print(cdll.LoadLibrary("libcrypto.dylib"))
print(cdll.LoadLibrary("libSystem.dylib"))
print(cdll.LoadLibrary("System.framework/System"))

Like many Python libraries, ctypes has both C and Python written parts.
The C module is internal, called _ctypes. Your _ctypes.cpython-38-darwin.so (note the underscore at start) is native code, compiled from the C file at https://github.com/python/cpython/blob/master/Modules/_ctypes/_ctypes.c
The public interface to it is written in Python, and it has platform dependend code to load native libs from the underlying OS, that code you pasted.
You can see where the module files are using Python itself like this:
>>> import ctypes
>>> ctypes.__file__
'C:\\Users\\antont\\AppData\\Local\\Programs\\Python\\Python38\\lib\\ctypes\\__init__.py'
>>> import _ctypes
>>> _ctypes.__file__
'C:\\Users\\antont\\AppData\\Local\\Programs\\Python\\Python38\\DLLs\\_ctypes.pyd'
That's on windows where the native code lib is a DLL, called .pyd for Python DLLs -- same as .dylib on Mac and .so on Linux and others.

Does Cython compile imported modules as part of the binary?

I'm just now reading into cython and I'm wondering if cython compiles imported modules as part of the executable of if you still need to have the modules installed on the target machine to run the cython binary.

The "interface" of a Cython module remains at the Python level. When you import a module in Cython, the module becomes available only at the Python level of the code and uses the regular Python import mechanism.
So:
Cython does not "compile in" the dependencies.
You need to install the dependencies on the target machine.
For "Cython level" code, including the question of "cimporting" module, Cython uses the equivalent of C headers (the .pxd declaration files) and dynamically loaded libraries to access external code. The .so files (for Linux, DLL for windows and dylib for mac) need to be present on the target machine.

Find boost_python version from setup.py

I have developed a Python module using Boost Python. This extension has a setup.py file to install it. This file requests the extension to be linked against libboost_python by doing something like the following:
my_module = Extension('_mymodule', files=...
libraries=['boost_python'],
...)
Recently, the boost developers seem to have changed the naming convention of Boost Python from naming the library libboost_python to naming it libboost_python27 (to reflect the fact that they were calling libboost_python3 the corresponding library for Python 3.x).
What is the best way, from the setup.py script, to detect how the Boost Python library should be named (given that it is installed in a non-standard location)?

How to auto compile python/c extension?

I wrote a python/c extension file lda_model.c
and I added setup.py:
from setuptools import setup, Extension
modules = [Extension('c_lda_model', sources=["lda_model.c"])]
setup(ext_modules=modules)
Now I have to compile the C code by
python setup.py build
before running python code where call the C code.
Is there any way to automatically compile the invoked C extension,
while running the python code?

Not with the standard way of writing extensions, which is what you just do.
However, there are a couple other approaches of writing native code extensions to Python which do compilation in execution time.
One such example is Weave that comes with Scipy - there are others ways if you look for.