Is there a way to import numpy without installing it?
I have a general application built into an .exe with PyInstaller. The application has a plugin system which allows it to be extended through Python scripts. The plugin import system works fine for basic modules (lone .py files, classes, functions, and simple packages). Internally, it globs a plugin directory and then imports accordingly using __import__ or importlib.import_module.
The application is built with minimal dependencies in order to reduce the overall size of the executable. Also, it's not possible to know what dependencies a future plugin will require nor practical to include everything. However, some plugins will inevitably require dependencies. numpy is a good test case for solving this class of problems.
Here's what I've tried.
A wheel file is really just a directory. It can be added to sys.path and the contents imported.
import sys
sys.path.append(r"C:\path\to\numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl")
import numpy as np
The wheel file is read, but the import generates an error.
*** ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
(removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
1. Check that you are using the Python you expect (you're using c:\projects\zip_test\venv\Scripts\python.exe),
and that you have no directories in your PATH or PYTHONPATH that can
interfere with the Python and numpy versions you're trying to use.
2. If (1) looks fine, you can open a new issue at
https://github.com/numpy/numpy/issues. Please include details on:
- how you installed Python
- how you installed numpy
- your operating system
- whether or not you have multiple versions of Python installed
- if you built from source, your compiler versions and ideally a build log
Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.
Original error was: No module named 'numpy.core._multiarray_umath'
The puzzling part is that the wheel file contains a .pyd for _multiarray_umath.
C:\projects\zip_test\plugins\New folder\numpy\core>dir
Volume in drive C is OS
Volume Serial Number is FE3D-6596
Directory of C:\projects\zip_test\plugins\New folder\numpy\core
07/25/2019 02:37 PM <DIR> .
07/25/2019 02:37 PM <DIR> ..
....
04/22/2019 02:55 AM 101,376 _multiarray_tests.cp36-win_amd64.pyd
04/22/2019 02:55 AM 2,494,976 _multiarray_umath.cp36-win_amd64.pyd
....
74 File(s) 583,173,551 bytes
5 Dir(s) 309,925,851,136 bytes free
This feels like a pathing issue. Yet adding the direct path for core/ to sys.path generates the same error.
sys.path.append(r"C:\projects\zip_test\plugins\numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl\numpy\core")
What's the deal? Why can't Python find numpy.core._multiarray_umath?
In response to repsones:
The wheel file was obtained from Christoph Gohlke.
My operating system is Windows 10 Pro 64-bit.
I am running Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32 (installed via Python Foundation build 3.6.8150.0) in a venv. The "system" Python is not on PATH.
I can install the numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl file via pip and import numpy as np works fine. I then uninstall numpy via pip uninstall -y numpy.
Running dumpbin /dependents _multiarray_umath.cp36-win_amd64.pyd produces:
Microsoft (R) COFF/PE Dumper Version 14.22.27905.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file _multiarray_umath.cp36-win_amd64.pyd
File Type: DLL
Image has the following dependencies:
mkl_rt.dll
python36.dll
KERNEL32.dll
VCRUNTIME140.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-utility-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
Summary
77000 .data
1000 .gfids
1C000 .pdata
30000 .rdata
3000 .reloc
1000 .rsrc
1BD000 .text
Of the dependencies, I can find mkl_rt.dll, python36.dll, and VCRUNTIME140.dll in the either the .whl or the venv. There are apparently two versions of KERNEL32.dll, one for x86 and one for x64. One of these resides in C:\Windows\System32.
The only information I could find about the api-ms-win-crt-*.dll are described on MSDN within the "Update for Universal C Runtime in Windows",
The Windows 10 Universal CRT is a Windows operating system component
that enables CRT functionality on the Windows operating system. This
update allows Windows desktop applications that depend on the Windows
10 Universal CRT release to run on earlier Windows operating systems.
These are for non-Windows 10 systems. There seems to be no "official" way to obtain them for a Windows 10 system. I was able to copy the dlls from a Windows 7 system. Naively putting them in the PYTHONPATH (not surprisingly) doesn't work. If they were to work at all, they would likely need to be registered. But 1) registering Windows 7 dlls on Windows 10, I am told, can make the system unstable and 2) definitely doesn't seem like a universal, portable solution.
First.
Build application with and without numpy. Check difference between builds.
Original error was: No module named 'numpy.core._multiarray_umath' Can means two things:
cannot found file
some dependence of this this pyd file is not satisfied. You may try to check it with http://dependencywalker.com/.
Yes, depending on how you define "install".
Numpy requires the use of .pyd files. A pyd file is essentially a dll. These are compiled code files which the Python interpreter references during run-time. Unfortunately, the Windows API calls used to load dll files requires them to exist at the filesystem level. When you try to import directly from a wheel, the pyd files (and the functions/classes they contain) aren't accessible. Hence, the error.
The (Relatively) Easy solution
One solution is to make the pyd files accessible on the filesystem. To do that, it's probably simplest to unpack the whole wheel file to disk1.
In that case, consider "how you define 'install'":
If you could run numpy straight from the wheel, wouldn't that mean numpy was installed?
What difference does it really make whether you're writing a wheel file or the contents of that wheel file?
Since the plugin system is downloading a wheel file onto the filesystem, the application must have write access. Download the wheel file and extract it into the plugins directory. As long as the plugin directory is on the PYTHONPATH (i.e. sys.path.append), the import will work. Smarts and so forth can be added from there. Of course, what level of smarts you need is a whole other issue. But that's the basic idea.
The Wizard's Path
Another solution is to create your own functionality to import a dll from memory. This is precisely the goal of Joachim Bauch's MemoryModule project. It seems the Py2Exe project used the MemoryModule at one point, through a module called zipextimporter.py. Unfortunately, the code is hard to find and seems outdated (Python 2.4). There is also a similar module called pymemimporter.py which is slightly more recent.
1 Numpy expects the pyd files to be in specific locations. If you want to extract only the pyd files, they'll need to be nested in the appropriate folders (e.g. numpy/core/_multiarray_umath.cp36-win_amd64.pyd). This is sufficient to import numpy, but unless the other modules are available, rather pointless.
Related
I have a python module which contains a C++ extension as well a shared library on which the C++ extension depends. The shared library is picked up by setuptools as an extra_object on the extension. After running python setup.py bdist_wheel, the module wheel object gets properly generated has a directory structure as follows:
+-pymodule.whl
| +-pymodule
| +-pymodule-version.dist-info
| +-extension.so
| +-shared_lib.so
To install this wheel, in my python environment I call pip install pymodule.whl which copies the python sources as well as the .so files into the environment's site-packages directory.
After installing the module, one can then attempt to import the module by calling import pymodule in a terminal for the environment. This triggers an exception to be thrown:
ImportError: shared_lib.so: cannot open shared object file: No such file or directory
This exception can be resolved by appending the appropriate site-packages directory to the LD_LIBRARY_PATH variable; however, it seems that this should work out of the box, especially considering that python is clearly able to locate extension.so.
Is there a way to force python to locate this shared library without having to explicitly point LD_LIBRARY_PATH at the installation location(i.e. site-packages)?
This question works around a similar problem by using package data and explicitly specifying an install location for the shared library. The issue I have with this approach is that the shared object is decoupled from the extension. In my case, the shared library and extension are both targets built by the same cmake build. I had previously attempted to use skbuild to build cmake based extentions; however, as per this issue, there is a similar issue in skbuild with including other libraries generated as part of the extension build.
I'm trying to get libarchive module working in python 3.4 on windows.
I've installed libarchive-c with pip and all went ok but whenever I try to import it into my code or even to run it alone I'me getting error:
OSError: [WinError 126] The specified module could not be found
This is coming from ffi.py from the code below:
libarchive_path = os.environ.get('LIBARCHIVE') or find_library('archive')
libarchive = ctypes.cdll.LoadLibrary(libarchive_path)
I've never used ctypes before but if I understand correctly it is looking for external DLL. So found and installed http://gnuwin32.sourceforge.net/packages/libarchive.htm also I've added C:\Program Files (x86)\GnuWin32\bin to my %PATH% in environmental variables but it still cannot load the module. As it does not give me the name I'm not sure what module it is looking for.
What am I missing?
(disclaimer) I contribute to https://github.com/Changaco/python-libarchive-c and I maintain https://github.com/nexB/scancode-toolkit
Both contain a ctypes binding for libarchive, though ScanCode is for extraction only.
My answer here is for python-libarchive-c, but ScanCode contains some of the DLL you are looking for so I am lacing in a bit of both.
To get python-libarchive-c going on Windows you need a libarchive DLL and its deps that can then be loaded.
There are no pre-built DLLs bundled in python-libarchive-c but I have prebuilt Windows binaries for another project here:
https://github.com/nexB/scancode-toolkit/tree/develop/src/extractcode/bin/win-32/bin
The corresponding source code is there: https://github.com/nexB/scancode-thirdparty-src
And you have MinGW32 build instructions there if you want to rebuild from source yourself: https://github.com/nexB/scancode-thirdparty-src/blob/master/libarchive/build.sh#L47
In general to load a DLL from a path -- assuming that the var libarchive contains the full path to that DLL -- use this:
lib = ctypes.CDLL(libarchive)
Now this is for Scancode. For python-libarchive-c, you could try to set the LIBARCHIVE variable to point the path of your DLL with:
set LIBARCHIVE="C:\.....\libarchive.dll"
Then start Python, import the library and use it.
NB: I did not test this (yet) , but this should work. IF not please file a bug.
I did not run any test on Python 3.4 either. I use primarily Python 2.7.
But the DLL and the code is not Python 2.7-specific at all.
FWIW, the way scancode loads the library is a tad more engaged since it can from the same code load DLLs Win/Linux/Mac for specific 32 or 64 bits archs using conventional locations. You can see the code in action there:
https://github.com/nexB/scancode-toolkit/blob/develop/src/extractcode/libarchive2.py#L64
ScanCode is NOT using python-libarchive-c ATM yet but a different/custom ctypes binding focused on a more specific use case of extraction only. At least it gives you access to a Win DLL and its deps (or instruction a build them) and an example on how to load it correctly.
/HTH
I am trying to import the following api wrapper / device driver as in this python package:
import com.oceanoptics.omnidriver.api.wrapper.Wrapper
Python just returns that there is no module named like this:
ImportError: No module named com.oceanoptics.omnidriver.api.wrapper.Wrapper
I installed Omnidriver from the device manufacturer's website. Specifically, I used the installer OmniDriver-2.37-win32-installer.exe and installed the "Development version". It installs a bunch of dlls in C:\Program Files (x86)\Ocean Optics\OmniDriver\OOI_HOME.
The wrapper is working properly in Matlab after adding C:\Program Files (x86)\Ocean Optics\OmniDriver\OOI_HOME to C:\Program Files (x86)\MATLAB\R2012b\toolbox\local\librarypath.txt and C:\Program Files (x86)\Ocean Optics\OmniDriver\OOI_HOME\OmniDriver.jar to C:\Program Files (x86)\MATLAB\R2012b\toolbox\local\classpath.txt. Thereafter, I can load the wrapper in Matlab with wrapper = com.oceanoptics.omnidriver.api.wrapper.Wrapper().
I guess my python installation (Enthought Canopy 1.4.1 win 32bit) is not looking for the dlls in the correct path because I would have to tell first.
So, my question is, how do I instruct python to successfully execute the import statement above?
Another approach for interfacing to the spectrometer using Python would be to use the python-seabreeze package. The package does not have thorough documentation, but if you're willing to be patient and try things out for yourself, then you should be able to get it to work. The author has put considerable work into making the package compatible with the vast majority of Ocean Optics' spectrometers. I just finished installing it on my Windows laptop and got it to work in under an hour.
I checked the website, and can't find any reference to python support. I believe the instructions you referenced are instructions how to install the java classes. I could find no information that mentioned or discussed python modules. You should contact Oceanview for clarification.
I'm using CXFreeze with PySide (QT). I get an error:
cx_Freeze: Python error in main script.
myscript.py line 33, in
File ExtensionLoader_Pyside_QtGUI.py, line 11, in
Import Error: DLL load failed: The specified module could not be found
When running a fresh install of Windows server 2008.
I'm running the frozen EXE package (with the folder). It seems to work on my own system and other systems. What might be the issue?
After reading, online, I tried to replace the Qt4Gui file, but this didn't solve the issue.
Python version is 2.7
Based on your Import Error: DLL load failed it is most likely an installation issue causing the missing DLL. To figure our exactly which DLL you are missing, use http://www.dependencywalker.com/ Run the .exe and open the .pyd file for File ExtensionLoader_Pyside_QtGUI.py and it will show you exactly which DLL's are missing and more importantly the locations where they should be. You can probably then track down the missing DLL online.
there are known issues with pyside 1.2.0 and cxFreeze. All should be fixed in development version (available on git repo). Please build the PySide from latest sources yourself or wait for PySide version 1.2.1. Build instructions are here [1].
[1] https://github.com/PySide/pyside-setup#building-pyside-on-a-windows-system
I used Py2exe instead of CXFreeze and it worked perfectly.
Also, apparently Python requires the MS Visual C++ Dependency Files:
http://www.microsoft.com/en-us/download/details.aspx?id=29
So any bundling needs that as well, if it's a fresh install. (Although I think they are now bundled with newer Windows versions.)
Other Notes:
In my experience, sometimes you should try CXFreeze, Py2EXE and PyInstaller quickly and see if one works best. As ideal as CXFreeze is re: cross platform, it just isn't going to happen perfectly.
Also, while I don't know if this was a factor, I set up a Windows 2000 Pro virtual machine and ran Py2exe on that. That was to ensure compatibility for all older Windows versions, and seemed to work well. (NOTE: Many things won't even run on Win2000 anymore so be careful that your other tools and libraries will run on it.)
Finally, be extra careful to match the bit level (32 vs 64) of all your libraries, and your Python install itself. If you have 32-bit python, ensure that your PySide, CXFreeze and any other libraries you use are 32-bit. (Or 64-bit if you're using 64-bit python.)
I need to keep a large number of Windows XP machines running the same version of python, with an assortment of modules, one of which is python-win32. I thought about installing python on a network drive that is mounted by all the client machines, and just adjust the path on the clients. Python starts up fine from the network, but when importing win32com I get a pop-up error saying:
The procedure entry point ?PyWinObject_AsHANDLE##YAHPAU_object##PAPAXH#Z could not be located in the dynamic link library pywintypes24.dll
after dismissing the message dialog I get in the console:
ImportError: DLL load failed: The specified procedure could not be found.
I searched the python directory for the pywintypes24.dll and it is present in "Lib\site-packages\pywin32_system32" .
What am I missing and is there another way in which I can install Python + Python-Win32 + additional module once and have them running on many machines? I don't have access to the Microsoft systems management tools, so I need to be a bit more low-tech than that.
On every machine you have to basically run following pywin32_postinstall.py -install once. Assuming your python installation on the network is N:\Python26, run following command on every client:
N:\Python26\python.exe N:\Python26\Scripts\pywin32_postinstall.py -install
Another important thing is Good Luck!. The reason is that you might need to do this as admin. In my case such setup worked for all but one computer. I still did not figure out why.
Python (or precisely, the OS) searches the DLLs using os.environ["PATH"] and not by searching sys.path.
So you could start Python using a simple .cmd file instead which adds \server\share\python26 to the path (given the installer (or you) copied the DLLs from \server\share\python26\lib\site-packages\pywin32-system32 to \server\share\python26).
Or, you can add the following code to your scripts before they try to import win32api etc:
# Add Python installation directory to the path,
# because on Windows 7 the pywin32 installer fails to copy
# the required DLLs to the %WINDIR%\System32 directory and
# copies them to the Python installation directory instead.
# Fortunately, in Python it is possible to modify the PATH
# before loading the DLLs.
os.environ["PATH"] = sys.prefix + ";" + os.environ.get("PATH")
import win32gui
import win32con
You could use batch files running at boot to
Mount the network share (net use \\server\share)
Copy the Python and packages installers from the network share to a local folder
Check version of the msi installer against the installed version
If different, uninstall Python and all version dependent packages
Reinstall all packages
This would be pretty much a roll your own central management system for that software.