Develop Reference

How to point python on dll which should be link against imported extension python module at runtime?

I have an extension python module built with cmake. Module requires third party libutil.dll which linkes against it at runtime. Both module and libutil.dll package into wheel.
Are there some ways to help python find libutil.dll for module through setup.py config?
Module and .dll located in the same folder in package.
I investigated some ways but all of them were rejected:
Add a path to dll in PATH variable or some else environment variables;
Launch python from directory where dll stored
setup.py has next content:
from setuptools import setup
setup(
name ='my_ext_module',
version = '0.0.1',
classifiers = ["Programming Language :: Python :: 3"],
packages = ['my_ext_module'],
package_dir = {'my_ext_module' : 'src/module_dir'},
package_data = {'' : ['*']},
python_requires = ">=3.5"
)
The project has next structure:
setup.py
src
|---module_dir
|---__init__.py
|---ext_module.pyd
|---libutil.dll

Python extension modules on Windows - controlling the DLL path for dependencies

I am porting a swig extension module that was designed for Python 2 to Python 3 under Windows 10 with a stock Python 3.9.1. The module builds properly and depends on several Windows dynamic load libraries (*.dll) that I have packaged into the data directory.
Importing the module fails, and using gflags to track down the problem shows that the dependent dll's are not being loaded as the search path does not include the module name: lib/site-packages/dbxml. When I manually move the dll's into the lib/site-packages, they can be located and successfully loaded.
I am confused on how to modify setup to generate a DLL path that includes the module name. I saw this post that seemed to imply that I could simply use the data_files option, but that does not appear to work in my case.
Here is what I am doing with what I believe to be the relevant variables replaced with their values:
setup(name = "dbxml",
version = "6.1.4",
description = "Berkeley DB XML Python API",
long_description = """removed...""",
author = "Oracle",
author_email = "berkeleydb-info_us#oracle.com",
url = "http://www.oracle.com",
py_modules = ["dbxml"],
ext_modules = [Extension("_dbxml", ["dbxml_python3_wrap.cpp"],
include_dirs = INCLUDES,
library_dirs = ['../../../lib', '../../build_windows/Release',
'../../../db-6.2.23/build_windows/Release',
'../../../xqilla/lib',
'../../../xerces-c-src/Build/Win32/VC10'],
define_macros = DEFINES,
libraries = ['libdbxml61', 'libdb62', 'xqilla23', 'xerces-c_3'],
extra_compile_args = ['/GR', '/EHsc']
)],
# The DLLs below are copied into lib/site-packages/dbxml
# but the DLL search path is:
# C:\Users\[...omitted...]\python3.9\lib\site-packages;
# C:\Users\[...omitted...]\python3.9;
# C:\WINDOWS\SYSTEM32
# and they are not found.
data_files = [('lib/site-packages/dbxml',
['../../../bin/libdbxml61.dll',
'../../../bin/libdb62.dll',
'../../../bin/xqilla23.dll',
'../../../bin/xerces-c_3_1.dll',
'../../build_windows/zlib1.dll',
'../../build_windows/zlibwapi.dll'
])
]
)
My understanding is that Extension's runtime_library_dirs keyword argument only controls the runtime library path on UNIX-like operating systems and is ignored on Windows 10. Obviously, I am doing something wrong. Any suggestions on how to fix this would be appreciated.
Thank you - Marie

It appears that the DLL library loading behavior changed in Python 3.8. The path for DLLs in Windows must be set explicitly using os module function add_dll_directory. I was able to resolve this by adding:
# add peer directory to dll search path
import os
dll_dir = os.path.join(os.path.dirname(__file__), "dbxml")
os.add_dll_directory(dll_dir)
There may be a better way to do this, but this will let the extension module load dyanmicaly linked libraries from a peer directory in site-packages.

Cython unable to find shared object file

I am trying to link to my own C library from Cython, following the directions I've found on the web, including this answer:
Using Cython To Link Python To A Shared Library
I am running IPython through Spyder.
My setup.py looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy as np
setup(
ext_modules = cythonize(
[Extension("*",["*.pyx"],
libraries =["MyLib"],
extra_compile_args = ["-fopenmp","-O3"],
extra_link_args=["-L/path/to/lib"])
]),
include_dirs = [np.get_include()],
)
The file libMyLib.so is in /path/to/lib and it compiles fine.
I have a Python script in my IPython profile startup folder that does this
try:
os.environ["LD_LIBRARY_PATH"] += ":/path/to/lib"
except KeyError:
os.environ["LD_LIBRARY_PATH"] = "/path/to/lib"
I can confirm that this is running, because if I type os.environ["LD_LIBRARY_PATH"] into the IPython interpreter, it returns /path/to/lib
But when I try to load the Cython module (i.e. import mycythonmodule) I get:
ImportError: libMyLib.so: cannot open shared object file: No such file or directory
I've also tried putting libMyLib.so in other places to see if cython would find it:
In the directory where Python is running
On the Python path
In the same folder as the cython module
But it still doesn't find the shared library. The only way I can get it to find the library is by dropping it in /usr/lib, but I don't want it there, I want to be able to set the library path.
Am I missing something?

I'm self-answering, in case anyone else runs into the same problem. Looks like the answers are here:
Set LD_LIBRARY_PATH before importing in python
Changing LD_LIBRARY_PATH at runtime for ctypes
According to these answers (and my experience), the linker reads LD_LIBRARY_PATH when python is launched, so changing it from within python doesn't have any useful effect, at least not the effect I was hoping for. The only solution is to either wrap python in a shell script that sets LD_LIBRARY_PATH, or else drop the shared object somewhere on the linker search path.
Kind of a pain, but it is what it is.

I have fixed it by change setup.py.
I have a C++ dynamic shared library called "libtmsmdreader.so". and a header file named "TmsMdReader.hpp"
I wrapper C++ shared library to cython library called "tmsmdreader-pythonxxxxxx.so"
from setuptools import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
name="tmsmdreader",
ext_modules=cythonize([
Extension(
name="tmsmdreader",
language="c++",
sources=["TmsMdReaderApi.pyx"],
libraries=["tmsmdreader"],
library_dirs=["."],
include_dirs=["."],
extra_compile_args=["-std=c++14"],
compiler_directives={'language_level': 3},
runtime_library_dirs=["."])
]))
library_dirs=["."] and runtime_library_dirs=["."] can fixed LD_LIBRARY_PATH if libtmsmdreader.so in python scripy directory

Using Cython To Link Python To A Shared Library

I am trying to integrate a third party library written in C with my python application using Cython. I have all of the python code written for a test. I am having trouble finding an example for setting this up.
I have a pyd/pyx file I created manually. The third party has given me a header file (*.h) and a shared library (*.so). As far as I can tell, there are no other dependencies. Can someone provide an example of how to set this up using Cython and disutils?
Thanks

Sure !
(In the following, I assume that you already know how to deal with cimport and the interactions between .pxd and .pyx. If this is not completely the case, just ask and I will develop that part as well)
The sample (grabbed from a C++ project of mine, but a C project would work pretty much the same) :
1. The Distutils setup file :
Assuming that the extension to be created will be called myext and the 3rd party shared library is libexternlib.so (note the lib* prefix, here)...
# setup.py file
import sys
import os
import shutil
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("myext") and not(name.endswith(".pyx") or name.endswith(".pxd"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
# build "myext.so" python extension to be added to "PYTHONPATH" afterwards...
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("myext",
sources=["myext.pyx",
"SomeAdditionalCppClass1.cpp",
"SomeAdditionalCppClass2.cpp"
],
libraries=["externlib"], # refers to "libexternlib.so"
language="c++", # remove this if C and not C++
extra_compile_args=["-fopenmp", "-O3"],
extra_link_args=["-DSOME_DEFINE_OPT",
"-L./some/extra/dependency/dir/"]
)
]
)
Note : Your external .so file is linked via the libraries option :
libraries=["externlib"] # Without the 'lib' prefix and the '.so' extension...
Note : the sources option can be used to get some additional source files compiled.
Important : myext.pxd (do not confound with .pyd - Windows stuff) and myext.pyx should be in the same directory. At compile time the definition file, if it exists, is processed first (more).
2. Then run it as follows :
After having changed directory to the one containing your myext.pxd, your myext.pyx, as well as the above setup.py script :
# setup.sh
# Make the "myext" Python Module ("myext.so")
CC="gcc" \
CXX="g++" \
CFLAGS="-I./some/path/to/includes/ -I../../../DEPENDENCIES/python2.7/inc -I../../../DEPENDENCIES/gsl-1.15" \
LDFLAGS="-L./some/path/to/externlib/" \
python setup.py build_ext --inplace
Where :
libexternlib.so is assumed to be located at ./some/path/to/externlib/
yourheader.h is assumed to be located at ./some/path/to/includes/
Note : CFLAGS could also have been setup using the extra_compile_args option :
extra_compile_args=["-I./some/path/to/includes/", "-fopenmp", "-O3"]
Note : LDFLAGS could also have been setup using the extra_link_args option :
extra_link_args=["-L./some/path/to/externlib/", "-DSOME_DEFINE_OPT", "-L./some/extra/dependency/dir/"]
Once distutils is done with the build, you get some new files, specially the myext.cpp, myext.h and most importantly, the myext.so.
3. After that, you're good to go :
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./some/path/to/externlib/
export PYTHONPATH=$PYTHONPATH:./some/path/to/myext/
# Run some script requiring "myext.so"
python somescript.py
Where your freshly created Python extension can be imported by its name :
# somescript.py
import myext
from myext import PySomeFeature
...
Note about Optimization : By default -O2 is used for compiling the extension, but this can be overloaded (see above setup where -O3 is specified).
Note about Cython paths : If Cython was installed in a custom directory, you might want to add it to your environment, before all :
PYTHONPATH=$PYTHONPATH:../../../DEPENDENCIES/Cython-0.18 export PYTHONPATH;
PATH=$PATH:../../../DEPENDENCIES/Cython-0.18/bin; export PATH;
Well, hope I covered the main points...

setup.py for packages that depend on both cython and f2py

I would like to create a setup.py script for a python package with several submodules that depend on both cython and f2py. I have attempted to use setuptools and numpy.distutils, but have so far failed:
Using setuptools
I am able to compile my cython extensions (and create an installation for the rest of the package) using setuptools. I have, however, been unable to figure out how to use setuptools to generate the f2py extension. After extensive searching, I only found rather old messages like this one that state that f2py modules must be compiled using numpy.distutils.
Using numpy.distutils
I am able to compile my f2py extensions (and create an installation for the rest of the package) using numpy.distutils. I have, however, been unable to figure out how to get numpy.distutils to compile my cython extensions as it always attempts to use pyrex to compile it (and I am using extensions specific to cython) recent. I have done a search to figure out how to get numpy.distutils for cython files and - at least as of a year ago - they recommend applying a monkey patch to numpy.distutils. It seems applying such a monkey patch also restricts the options that can be passed to Cython.
My question is: what is the recommended way to write a setup.py script for packages that depend on both f2py and cython? Is applying a patch to numpy.distutils really the way to go still?

You can just call each separately in your setup.py as in
http://answerpot.com/showthread.php?601643-cython%20and%20f2py
# Cython extension
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
ext_modules = [Extension( 'cext', ['cext.pyx'] )],
cmdclass = {'build_ext': build_ext},
script_args = ['build_ext', '--inplace'],
)
# Fortran extension
from numpy.distutils.core import setup, Extension
setup(
ext_modules = [Extension( 'fext', ['fext.f90'] )],
)
Your calling context (I think they call this namespace, not sure)
has to change as to what the current object Extension and function
setup() is.
The first setup() call, it's the distutils.extension.Extension
and distutils.core.setup()
The second setup() call, it's the numpy.distutils.core.Extension
and numpy.distutils.core.setup()

Turns out this is no longer true. With both setuptools and distutils (at least the numpy version) it is possible to have extensions with C, Cython and f2py. The only caveat is that to compile f2py modules one must always use numpy.distutils for both the setup and Extension functions. But setuptools can still be used for the installation (for example, allowing the installation of a developer version with python setup.py develop).
To use distutils exclusively you use the following:
from numpy.distutils.core import setup
from numpy.distutils.extension import Extension
To use setuptools, you need to import it before the distutils imports:
import setuptools
And then the rest of the code is identical:
from numpy import get_include
from Cython.Build import cythonize
NAME = 'my_package'
NUMPY_INC = get_include()
extensions = [
Extension(name=NAME + ".my_cython_ext",
include_dirs=[NUMPY_INC, "my_c_dir"]
sources=["my_cython_ext.pyx", "my_c_dir/my_ext_c_file.c"]),
Extension(name=NAME + ".my_f2py_ext",
sources=["my_f2py_ext.f"]),
]
extensions = cythonize(extensions)
setup(..., ext_modules=extensions)
Obviously you need to put all your other stuff in the setup() call. In the above I assume that you'll use numpy with Cython, along with an external C file (my_ext_c_file.c) that will be at my_c_dir/, and that the f2py module is only one Fortran file. Adjust as needed.