I have a python module which contains a C++ extension as well a shared library on which the C++ extension depends. The shared library is picked up by setuptools as an extra_object on the extension. After running python setup.py bdist_wheel, the module wheel object gets properly generated has a directory structure as follows:
+-pymodule.whl
| +-pymodule
| +-pymodule-version.dist-info
| +-extension.so
| +-shared_lib.so
To install this wheel, in my python environment I call pip install pymodule.whl which copies the python sources as well as the .so files into the environment's site-packages directory.
After installing the module, one can then attempt to import the module by calling import pymodule in a terminal for the environment. This triggers an exception to be thrown:
ImportError: shared_lib.so: cannot open shared object file: No such file or directory
This exception can be resolved by appending the appropriate site-packages directory to the LD_LIBRARY_PATH variable; however, it seems that this should work out of the box, especially considering that python is clearly able to locate extension.so.
Is there a way to force python to locate this shared library without having to explicitly point LD_LIBRARY_PATH at the installation location(i.e. site-packages)?
This question works around a similar problem by using package data and explicitly specifying an install location for the shared library. The issue I have with this approach is that the shared object is decoupled from the extension. In my case, the shared library and extension are both targets built by the same cmake build. I had previously attempted to use skbuild to build cmake based extentions; however, as per this issue, there is a similar issue in skbuild with including other libraries generated as part of the extension build.
Related
Is there a way to import numpy without installing it?
I have a general application built into an .exe with PyInstaller. The application has a plugin system which allows it to be extended through Python scripts. The plugin import system works fine for basic modules (lone .py files, classes, functions, and simple packages). Internally, it globs a plugin directory and then imports accordingly using __import__ or importlib.import_module.
The application is built with minimal dependencies in order to reduce the overall size of the executable. Also, it's not possible to know what dependencies a future plugin will require nor practical to include everything. However, some plugins will inevitably require dependencies. numpy is a good test case for solving this class of problems.
Here's what I've tried.
A wheel file is really just a directory. It can be added to sys.path and the contents imported.
import sys
sys.path.append(r"C:\path\to\numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl")
import numpy as np
The wheel file is read, but the import generates an error.
*** ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
(removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
1. Check that you are using the Python you expect (you're using c:\projects\zip_test\venv\Scripts\python.exe),
and that you have no directories in your PATH or PYTHONPATH that can
interfere with the Python and numpy versions you're trying to use.
2. If (1) looks fine, you can open a new issue at
https://github.com/numpy/numpy/issues. Please include details on:
- how you installed Python
- how you installed numpy
- your operating system
- whether or not you have multiple versions of Python installed
- if you built from source, your compiler versions and ideally a build log
Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.
Original error was: No module named 'numpy.core._multiarray_umath'
The puzzling part is that the wheel file contains a .pyd for _multiarray_umath.
C:\projects\zip_test\plugins\New folder\numpy\core>dir
Volume in drive C is OS
Volume Serial Number is FE3D-6596
Directory of C:\projects\zip_test\plugins\New folder\numpy\core
07/25/2019 02:37 PM <DIR> .
07/25/2019 02:37 PM <DIR> ..
....
04/22/2019 02:55 AM 101,376 _multiarray_tests.cp36-win_amd64.pyd
04/22/2019 02:55 AM 2,494,976 _multiarray_umath.cp36-win_amd64.pyd
....
74 File(s) 583,173,551 bytes
5 Dir(s) 309,925,851,136 bytes free
This feels like a pathing issue. Yet adding the direct path for core/ to sys.path generates the same error.
sys.path.append(r"C:\projects\zip_test\plugins\numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl\numpy\core")
What's the deal? Why can't Python find numpy.core._multiarray_umath?
In response to repsones:
The wheel file was obtained from Christoph Gohlke.
My operating system is Windows 10 Pro 64-bit.
I am running Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32 (installed via Python Foundation build 3.6.8150.0) in a venv. The "system" Python is not on PATH.
I can install the numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl file via pip and import numpy as np works fine. I then uninstall numpy via pip uninstall -y numpy.
Running dumpbin /dependents _multiarray_umath.cp36-win_amd64.pyd produces:
Microsoft (R) COFF/PE Dumper Version 14.22.27905.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file _multiarray_umath.cp36-win_amd64.pyd
File Type: DLL
Image has the following dependencies:
mkl_rt.dll
python36.dll
KERNEL32.dll
VCRUNTIME140.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-utility-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
Summary
77000 .data
1000 .gfids
1C000 .pdata
30000 .rdata
3000 .reloc
1000 .rsrc
1BD000 .text
Of the dependencies, I can find mkl_rt.dll, python36.dll, and VCRUNTIME140.dll in the either the .whl or the venv. There are apparently two versions of KERNEL32.dll, one for x86 and one for x64. One of these resides in C:\Windows\System32.
The only information I could find about the api-ms-win-crt-*.dll are described on MSDN within the "Update for Universal C Runtime in Windows",
The Windows 10 Universal CRT is a Windows operating system component
that enables CRT functionality on the Windows operating system. This
update allows Windows desktop applications that depend on the Windows
10 Universal CRT release to run on earlier Windows operating systems.
These are for non-Windows 10 systems. There seems to be no "official" way to obtain them for a Windows 10 system. I was able to copy the dlls from a Windows 7 system. Naively putting them in the PYTHONPATH (not surprisingly) doesn't work. If they were to work at all, they would likely need to be registered. But 1) registering Windows 7 dlls on Windows 10, I am told, can make the system unstable and 2) definitely doesn't seem like a universal, portable solution.
First.
Build application with and without numpy. Check difference between builds.
Original error was: No module named 'numpy.core._multiarray_umath' Can means two things:
cannot found file
some dependence of this this pyd file is not satisfied. You may try to check it with http://dependencywalker.com/.
Yes, depending on how you define "install".
Numpy requires the use of .pyd files. A pyd file is essentially a dll. These are compiled code files which the Python interpreter references during run-time. Unfortunately, the Windows API calls used to load dll files requires them to exist at the filesystem level. When you try to import directly from a wheel, the pyd files (and the functions/classes they contain) aren't accessible. Hence, the error.
The (Relatively) Easy solution
One solution is to make the pyd files accessible on the filesystem. To do that, it's probably simplest to unpack the whole wheel file to disk1.
In that case, consider "how you define 'install'":
If you could run numpy straight from the wheel, wouldn't that mean numpy was installed?
What difference does it really make whether you're writing a wheel file or the contents of that wheel file?
Since the plugin system is downloading a wheel file onto the filesystem, the application must have write access. Download the wheel file and extract it into the plugins directory. As long as the plugin directory is on the PYTHONPATH (i.e. sys.path.append), the import will work. Smarts and so forth can be added from there. Of course, what level of smarts you need is a whole other issue. But that's the basic idea.
The Wizard's Path
Another solution is to create your own functionality to import a dll from memory. This is precisely the goal of Joachim Bauch's MemoryModule project. It seems the Py2Exe project used the MemoryModule at one point, through a module called zipextimporter.py. Unfortunately, the code is hard to find and seems outdated (Python 2.4). There is also a similar module called pymemimporter.py which is slightly more recent.
1 Numpy expects the pyd files to be in specific locations. If you want to extract only the pyd files, they'll need to be nested in the appropriate folders (e.g. numpy/core/_multiarray_umath.cp36-win_amd64.pyd). This is sufficient to import numpy, but unless the other modules are available, rather pointless.
I have created a small Python module using C++ and Boost, built with CMake. Now I need to install it into standard python modules directory, however it needs to be a path inside CMAKE_INSTALL_PREFIX, so it can be packaged with standard Debian packaging system.
Now I have this, where the last line is obviously wrong, as it is an absolute path to a system directory.
cmake_minimum_required(VERSION 3.12)
project(foo)
# Not Shown: Extract ${PYTHON_VERSION} from interpreter
find_package(Python ${PYTHON_VERSION} REQUIRED COMPONENTS Interpreter Development)
find_package(Boost 1.55.0 REQUIRED COMPONENTS system python${Python_VERSION_MAJOR}${Python_VERSION_MINOR})
Python_add_library(foo MODULE
src/foo.cc src/python_interface.cc
)
set_target_properties(foo PROPERTIES PREFIX "")
target_link_libraries(foo
PUBLIC
Boost::python${Python_VERSION_MAJOR}${Python_VERSION_MINOR}
)
# This is wrong
install(TARGETS foo DESTINATION ${Python_SITEARCH}/foo)
I also had a look at Python's distutils and setuptools, however they don't seem suited to standalone use.
When trying to build numpy on a linux platform, I can't make the configure script look in the right place.
I use
python setup.py config --library-dirs=/software/intel/mkl/10.2.2.025/lib/em64t/
but then I get
mkl_info:
libraries mkl,vml,guide not found in /software/intel/mkl/10.2.2.025
libraries mkl,vml,guide not found in /software/intel/mkl/10.2.2.025/include
libraries mkl,vml,guide not found in /software/intel/mkl/10.2.2.025/lib
So it looks like it never actually looks into the subdirectory emt64/. The path I'm giving is also present in my LD_LIBRARY_PATH.
How can I give the script the right path?
Thanks in advance!
Had a similar problem with rpy2. Did not have root permissions and could not alter the existing R installation or add to its core library directory. R was not built as a shared object library, so I could not link the rpy2 build to a libR.so.
I had to cross compile libR.so on a separate machine (same R version, same Linux family) and copy it to a different directory. I wanted that directory to be seen by setup.py.
Couldn't get -L to work on the command line. It appeared that this argument was deactivated.
(FAIL) python setup.py -L${LD_LIBRARY_PATH} build install
What I ended up doing was editing setup.py and changing a line that accepts library directory entries.
(old) r_libs = []
(new) [os.path.join('/root','path','to_my','install','R','lib'),]
Reran it as: python setup.py build install
Success!
Perhaps
export PYTHONLIB="/software/intel/mkl/10.2.2.025/lib/em64t/"
python setup.py config
Recently I found about this tool easy_install that help me to easy install additional python modules. The problem is that for each module it creates additional *.egg folder (sometime there is only an egg file?) (no source?) and I don't know how to setup eclipse paths.
By default I have included C:\Python26\Lib\site-packages and this is enough when I install python modules from source... but not when I'm using easy_intall
For example django instaled with easy_install is located in C:\Python26\Lib\site-packages\django-1.2.5-py2.6.egg\django and installed from source it's located in C:\Python26\Lib\site-packages\django
In fact when I'm using easy_install all installed modules are working without a problem, the only problem is that eclipse can't locate where is the source and gives me a false unresolved import errors
Where I'm wrong?
I'm assuming that eclipse does not search the egg files for source. Eggs, like jar files in Java, are just zipfiles of python code with some included metadata.
You'll also note that in site-packages you've got easy-install.pth and setuptools.pth files. Those files are parsed by python and used to add other directories and egg files to your PYTHONPATH (import sys; sys.path) so that Python can find the code in those locations. Eclipse isn't seeing those imports as valid because it is most likely not setup to take pth files into account.
To get Eclipse to recognize that Django is really installed you may want to try removing your easy_installed django package and reinstalling it with:
easy_install --always-unzip django
That way rather than installing a compressed egg file you'll have a normal package directory that eclipse should have a fairly easy time opening.
Alternatively, in your screenshot above it looks like you may just need to explicitly add each egg file you want eclipse to use.
I'm trying to import pycurl:
$ python -c "import pycurl"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: libcurl.so.4: cannot open shared object file: No such file or directory
Now, libcurl.so.4 is in /usr/local/lib. As you can see, this is in sys.path:
$ python -c "import sys; print(sys.path)"
['', '/usr/local/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
'/usr/local/lib/python25.zip', '/usr/local/lib/python2.5',
'/usr/local/lib/python2.5/plat-linux2', '/usr/local/lib/python2.5/lib-tk',
'/usr/local/lib/python2.5/lib-dynload',
'/usr/local/lib/python2.5/sitepackages', '/usr/local/lib',
'/usr/local/lib/python2.5/site-packages']
Any help will be greatly appreciated.
sys.path is only searched for Python modules. For dynamic linked libraries, the paths searched must be in LD_LIBRARY_PATH. Check if your LD_LIBRARY_PATH includes /usr/local/lib, and if it doesn't, add it and try again.
Some more information (source):
In Linux, the environment variable
LD_LIBRARY_PATH is a colon-separated
set of directories where libraries
should be searched for first, before
the standard set of directories; this
is useful when debugging a new library
or using a nonstandard library for
special purposes. The environment
variable LD_PRELOAD lists shared
libraries with functions that override
the standard set, just as
/etc/ld.so.preload does. These are
implemented by the loader
/lib/ld-linux.so. I should note that,
while LD_LIBRARY_PATH works on many
Unix-like systems, it doesn't work on
all; for example, this functionality
is available on HP-UX but as the
environment variable SHLIB_PATH, and
on AIX this functionality is through
the variable LIBPATH (with the same
syntax, a colon-separated list).
Update: to set LD_LIBRARY_PATH, use one of the following, ideally in your ~/.bashrc
or equivalent file:
export LD_LIBRARY_PATH=/usr/local/lib
or
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
Use the first form if it's empty (equivalent to the empty string, or not present at all), and the second form if it isn't. Note the use of export.
Ensure your libcurl.so module is in the system library path, which is distinct and separate from the python library path.
A "quick fix" is to add this path to a LD_LIBRARY_PATH variable. However, setting that system wide (or even account wide) is a BAD IDEA, as it is possible to set it in such a way that some programs will find a library it shouldn't, or even worse, open up security holes.
If your "locally installed libraries" are installed in, for example, /usr/local/lib, add this directory to /etc/ld.so.conf (it's a text file) and run ldconfig
The command will run a caching utility, but will also create all the necessary "symbolic links" required for the loader system to function. It is surprising that the make install for libcurl did not do this already, but it's possible it could not if /usr/local/lib is not in /etc/ld.so.conf already.
PS: it's possible that your /etc/ld.so.conf contains nothing but include ld.so.conf.d/*.conf. You can still add a directory path after it, or just create a new file inside the directory it's being included from. Dont forget to run ldconfig after it.
Be careful. Getting this wrong can screw up your system.
Additionally: make sure your python module is compiled against THAT version of libcurl. If you just copied some files over from another system, this wont always work. If in doubt, compile your modules on the system you intend to run them on.
You can also set LD_RUN_PATH to /usr/local/lib in your user environment when you compile pycurl in the first place. This will embed /usr/local/lib in the RPATH attribute of the C extension module .so so that it automatically knows where to find the library at run time without having to have LD_LIBRARY_PATH set at run time.
Had the exact same issue. I installed curl 7.19 to /opt/curl/ to make sure that I would not affect current curl on our production servers.
Once I linked libcurl.so.4 to /usr/lib:
sudo ln -s /opt/curl/lib/libcurl.so /usr/lib/libcurl.so.4
I still got the same error! Durf.
But running ldconfig make the linkage for me and that worked. No need to set the LD_RUN_PATH or LD_LIBRARY_PATH at all. Just needed to run ldconfig.
As a supplement to above answers - I'm just bumping into a similar problem, and working completely of the default installed python.
When I call the example of the shared object library I'm looking for with LD_LIBRARY_PATH, I get something like this:
$ LD_LIBRARY_PATH=/path/to/mysodir:$LD_LIBRARY_PATH python example-so-user.py
python: can't open file 'example-so-user.py': [Errno 2] No such file or directory
Notably, it doesn't even complain about the import - it complains about the source file!
But if I force loading of the object using LD_PRELOAD:
$ LD_PRELOAD=/path/to/mysodir/mypyobj.so python example-so-user.py
python: error while loading shared libraries: libtiff.so.5: cannot open shared object file: No such file or directory
... I immediately get a more meaningful error message - about a missing dependency!
Just thought I'd jot this down here - cheers!
I use python setup.py build_ext -R/usr/local/lib -I/usr/local/include/libcalg-1.0 and the compiled .so file is under the build folder.
you can type python setup.py --help build_ext to see the explanations of -R and -I
For me what works here is to using a version manager such as pyenv, which I strongly recommend to get your project environments and package versions well managed and separate from that of the operative system.
I had this same error after an OS update, but was easily fixed with pyenv install 3.7-dev (the version I use).