Develop Reference

DLL not found after switching from Python 3.7.6 to Python 3.10.3 [duplicate]

This question already has answers here:
Can't import dll module in Python
(6 answers)
Closed 11 months ago.
I am trying to upgrade a library to Python 3.10. Up to now, I have been using 3.7.6.
In my library, I am using DLLs. In the package __init__.py, I add the DLL path to the PATH variable. Since I want to have the code compatible with 32 and 64 bit systems, I check the bitness at runtime and add the appropriate DLL folder to the PATH:
import os
import sys
# Add libs to PATH environment variable such that DLLs can be loaded.
# Load either x64 or x86 depending on whether we are running a 32/64
# bit system.
package_directory = os.path.dirname(os.path.abspath(__file__))
if sys.maxsize > 2**32:
path_dir = os.path.join(package_directory, 'libs', 'x64')
else:
path_dir = os.path.join(package_directory, 'libs', 'x86')
os.environ['PATH'] += os.pathsep + path_dir
The corrresponding folder structure is
package_root
|- libs
|- x64
|- libbristolpolled.dll
...
|- x86
|- libbristolpolled.dll
...
Then in a sub-package, I am using:
from ctypes import CDLL
dll = "libbristolpolled.dll"
_lib_bristlp = CDLL(dll)
This has worked fine in 3.7.6 but fails in 3.10.3 with the following error message:
File "C:\...\lib\site-packages\optoMD\sensor\optical\bristol\dll_api.py", line 37, in <module>
_lib_bristlp = CDLL(dll) # The correct DLL
File "C:\Program Files\Python310\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'libbristolpolled.dll' (or one of its dependencies). Try using the full path with constructor syntax.
If I instead use the absolute path, as the error message suggests, it works:
from ctypes import CDLL
dll = r"C:\...\libs\x64\libbristolpolled.dll"
_lib_bristlp = CDLL(dll)
What is the reason the old method doesn't work anymore, or how can I make it work again? I'd prefer not to have absolute paths in my code and I liked the fact that I handle the DLL path once in the init file and afterwards I just need to import the DLL using the file name.

This particular features (by adding the path to find a .dll to PATH environment variable) has been removed in Python 3.8. As per the notice, the new function os.add_dll_directory is provided; quoting the documentation for the reason for the change:
New in version 3.8: Previous versions of CPython would resolve DLLs using the default behavior for the current process. This led to inconsistencies, such as only sometimes searching PATH or the current working directory, and OS functions such as AddDllDirectory having no effect.
The more portable solution going forward is to attempt to use that function, alternatively just generate the absolute path as you have current done using os.path.dirname and os.path.join on the dll variable - i.e. to maintain compatibility with prior Python versions; also this assumes those .dll files do not have external dependencies, if there are, this thread has a significantly more detail regarding the issues at hand (even with absolute paths - though given that it did appear to work for your program and library, this should not an issue for your use case).

How to point python on dll which should be link against imported extension python module at runtime?

I have an extension python module built with cmake. Module requires third party libutil.dll which linkes against it at runtime. Both module and libutil.dll package into wheel.
Are there some ways to help python find libutil.dll for module through setup.py config?
Module and .dll located in the same folder in package.
I investigated some ways but all of them were rejected:
Add a path to dll in PATH variable or some else environment variables;
Launch python from directory where dll stored
setup.py has next content:
from setuptools import setup
setup(
name ='my_ext_module',
version = '0.0.1',
classifiers = ["Programming Language :: Python :: 3"],
packages = ['my_ext_module'],
package_dir = {'my_ext_module' : 'src/module_dir'},
package_data = {'' : ['*']},
python_requires = ">=3.5"
)
The project has next structure:
setup.py
src
|---module_dir
|---__init__.py
|---ext_module.pyd
|---libutil.dll

How to properly link to a C library in cython C++ build

I have a compiled C library htslib in my module top-level directory, this is first built using ./configure; make outside of python before running python setup.py install. I am trying to use headers listed in htslib. I have cython scripts e.g. script.pyx which wrap supporting header files cpp_header.h, but these are compiled using C++ via the setup script. Setup of the python module runs without problems, but at runtime htslib symbols are missing, I get ImportError: foo.cpython-37m-x86_64-linux-gnu.so: undefined symbol: hts_close. My directory structure is:
foo/
htslib/ # c library
objectfiles.o
htslib/
headers.h
cram/
more_headers.h
more_objects.o
foo/
script.pyx
cpp_header.h # c++
setup.py
In my setup script I list some libraries needed for htslib, some library_dirs, include_dirs, and extra_compile_args for building the Extension:
root = os.path.abspath(os.path.dirname(__file__))
htslib = os.path.join(root, "htslib")
libraries = [htslib]
library_dirs = [htslib, numpy.get_include(), root, "htslib", "foo"]
include_dirs = [numpy.get_include(), root, htslib]
for item in ["htslib", "cram"]:
include_dirs.append(os.path.join(htslib, item))
ext_modules.append(Extension(f"foo.script",
[f"foo/script.pyx"],
libraries=libraries,
library_dirs=library_dirs,
include_dirs=include_dirs,
extra_compile_args=extras,
define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
language="c++"))
Additionally, I also add htslib to the package_data option in setup:
setup(...
packages=["foo"],
package_data={"foo": ["htslib/*"]},
ext_modules=cythonize(ext_modules),
...
)
Inside the cpp_header.h file the htslib headers are included as:
#include "../htslib/htslib/sam.h"
Im trying this on unix. Would any one know how I can get this working?

Cython unable to find shared object file

I am trying to link to my own C library from Cython, following the directions I've found on the web, including this answer:
Using Cython To Link Python To A Shared Library
I am running IPython through Spyder.
My setup.py looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy as np
setup(
ext_modules = cythonize(
[Extension("*",["*.pyx"],
libraries =["MyLib"],
extra_compile_args = ["-fopenmp","-O3"],
extra_link_args=["-L/path/to/lib"])
]),
include_dirs = [np.get_include()],
)
The file libMyLib.so is in /path/to/lib and it compiles fine.
I have a Python script in my IPython profile startup folder that does this
try:
os.environ["LD_LIBRARY_PATH"] += ":/path/to/lib"
except KeyError:
os.environ["LD_LIBRARY_PATH"] = "/path/to/lib"
I can confirm that this is running, because if I type os.environ["LD_LIBRARY_PATH"] into the IPython interpreter, it returns /path/to/lib
But when I try to load the Cython module (i.e. import mycythonmodule) I get:
ImportError: libMyLib.so: cannot open shared object file: No such file or directory
I've also tried putting libMyLib.so in other places to see if cython would find it:
In the directory where Python is running
On the Python path
In the same folder as the cython module
But it still doesn't find the shared library. The only way I can get it to find the library is by dropping it in /usr/lib, but I don't want it there, I want to be able to set the library path.
Am I missing something?

I'm self-answering, in case anyone else runs into the same problem. Looks like the answers are here:
Set LD_LIBRARY_PATH before importing in python
Changing LD_LIBRARY_PATH at runtime for ctypes
According to these answers (and my experience), the linker reads LD_LIBRARY_PATH when python is launched, so changing it from within python doesn't have any useful effect, at least not the effect I was hoping for. The only solution is to either wrap python in a shell script that sets LD_LIBRARY_PATH, or else drop the shared object somewhere on the linker search path.
Kind of a pain, but it is what it is.

I have fixed it by change setup.py.
I have a C++ dynamic shared library called "libtmsmdreader.so". and a header file named "TmsMdReader.hpp"
I wrapper C++ shared library to cython library called "tmsmdreader-pythonxxxxxx.so"
from setuptools import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
name="tmsmdreader",
ext_modules=cythonize([
Extension(
name="tmsmdreader",
language="c++",
sources=["TmsMdReaderApi.pyx"],
libraries=["tmsmdreader"],
library_dirs=["."],
include_dirs=["."],
extra_compile_args=["-std=c++14"],
compiler_directives={'language_level': 3},
runtime_library_dirs=["."])
]))
library_dirs=["."] and runtime_library_dirs=["."] can fixed LD_LIBRARY_PATH if libtmsmdreader.so in python scripy directory

How to find path to shared library when using python ctypes with distutils

I am working on a python package that is basically a python wrapper to a c-backend. The c-backend is called form python via ctypes.
It works fine if I give ctypes the absolute path to the library to load it. But I don't know how to find the absolute path to the library if the library is compiled via distutils' setup.py? (The library is built successfully by the install script.)
Here is the important snippet of setup.py:
libreboundmodule = Extension('librebound',
sources = [ 'librebound/librebound.c' ],
)
setup(name='rebound',
...
ext_modules = [libreboundmodule])
Here is the part of the rebound module __init__.py. How can I let ctypes know the path to my library?
import ctypes
PATH = ???
librebound = ctypes.CDLL(PATH+"librebound.so", RTLD_GLOBAL)
EDIT:
Is it ok to do the following?
import ctypes
PATH = os.path.dirname(__file__)
librebound = ctypes.CDLL(PATH+"/../librebound.so", RTLD_GLOBAL)