I'm trying to get libarchive module working in python 3.4 on windows.
I've installed libarchive-c with pip and all went ok but whenever I try to import it into my code or even to run it alone I'me getting error:
OSError: [WinError 126] The specified module could not be found
This is coming from ffi.py from the code below:
libarchive_path = os.environ.get('LIBARCHIVE') or find_library('archive')
libarchive = ctypes.cdll.LoadLibrary(libarchive_path)
I've never used ctypes before but if I understand correctly it is looking for external DLL. So found and installed http://gnuwin32.sourceforge.net/packages/libarchive.htm also I've added C:\Program Files (x86)\GnuWin32\bin to my %PATH% in environmental variables but it still cannot load the module. As it does not give me the name I'm not sure what module it is looking for.
What am I missing?
(disclaimer) I contribute to https://github.com/Changaco/python-libarchive-c and I maintain https://github.com/nexB/scancode-toolkit
Both contain a ctypes binding for libarchive, though ScanCode is for extraction only.
My answer here is for python-libarchive-c, but ScanCode contains some of the DLL you are looking for so I am lacing in a bit of both.
To get python-libarchive-c going on Windows you need a libarchive DLL and its deps that can then be loaded.
There are no pre-built DLLs bundled in python-libarchive-c but I have prebuilt Windows binaries for another project here:
https://github.com/nexB/scancode-toolkit/tree/develop/src/extractcode/bin/win-32/bin
The corresponding source code is there: https://github.com/nexB/scancode-thirdparty-src
And you have MinGW32 build instructions there if you want to rebuild from source yourself: https://github.com/nexB/scancode-thirdparty-src/blob/master/libarchive/build.sh#L47
In general to load a DLL from a path -- assuming that the var libarchive contains the full path to that DLL -- use this:
lib = ctypes.CDLL(libarchive)
Now this is for Scancode. For python-libarchive-c, you could try to set the LIBARCHIVE variable to point the path of your DLL with:
set LIBARCHIVE="C:\.....\libarchive.dll"
Then start Python, import the library and use it.
NB: I did not test this (yet) , but this should work. IF not please file a bug.
I did not run any test on Python 3.4 either. I use primarily Python 2.7.
But the DLL and the code is not Python 2.7-specific at all.
FWIW, the way scancode loads the library is a tad more engaged since it can from the same code load DLLs Win/Linux/Mac for specific 32 or 64 bits archs using conventional locations. You can see the code in action there:
https://github.com/nexB/scancode-toolkit/blob/develop/src/extractcode/libarchive2.py#L64
ScanCode is NOT using python-libarchive-c ATM yet but a different/custom ctypes binding focused on a more specific use case of extraction only. At least it gives you access to a Win DLL and its deps (or instruction a build them) and an example on how to load it correctly.
/HTH
Related
Is there a way to import numpy without installing it?
I have a general application built into an .exe with PyInstaller. The application has a plugin system which allows it to be extended through Python scripts. The plugin import system works fine for basic modules (lone .py files, classes, functions, and simple packages). Internally, it globs a plugin directory and then imports accordingly using __import__ or importlib.import_module.
The application is built with minimal dependencies in order to reduce the overall size of the executable. Also, it's not possible to know what dependencies a future plugin will require nor practical to include everything. However, some plugins will inevitably require dependencies. numpy is a good test case for solving this class of problems.
Here's what I've tried.
A wheel file is really just a directory. It can be added to sys.path and the contents imported.
import sys
sys.path.append(r"C:\path\to\numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl")
import numpy as np
The wheel file is read, but the import generates an error.
*** ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
(removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
1. Check that you are using the Python you expect (you're using c:\projects\zip_test\venv\Scripts\python.exe),
and that you have no directories in your PATH or PYTHONPATH that can
interfere with the Python and numpy versions you're trying to use.
2. If (1) looks fine, you can open a new issue at
https://github.com/numpy/numpy/issues. Please include details on:
- how you installed Python
- how you installed numpy
- your operating system
- whether or not you have multiple versions of Python installed
- if you built from source, your compiler versions and ideally a build log
Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.
Original error was: No module named 'numpy.core._multiarray_umath'
The puzzling part is that the wheel file contains a .pyd for _multiarray_umath.
C:\projects\zip_test\plugins\New folder\numpy\core>dir
Volume in drive C is OS
Volume Serial Number is FE3D-6596
Directory of C:\projects\zip_test\plugins\New folder\numpy\core
07/25/2019 02:37 PM <DIR> .
07/25/2019 02:37 PM <DIR> ..
....
04/22/2019 02:55 AM 101,376 _multiarray_tests.cp36-win_amd64.pyd
04/22/2019 02:55 AM 2,494,976 _multiarray_umath.cp36-win_amd64.pyd
....
74 File(s) 583,173,551 bytes
5 Dir(s) 309,925,851,136 bytes free
This feels like a pathing issue. Yet adding the direct path for core/ to sys.path generates the same error.
sys.path.append(r"C:\projects\zip_test\plugins\numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl\numpy\core")
What's the deal? Why can't Python find numpy.core._multiarray_umath?
In response to repsones:
The wheel file was obtained from Christoph Gohlke.
My operating system is Windows 10 Pro 64-bit.
I am running Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32 (installed via Python Foundation build 3.6.8150.0) in a venv. The "system" Python is not on PATH.
I can install the numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl file via pip and import numpy as np works fine. I then uninstall numpy via pip uninstall -y numpy.
Running dumpbin /dependents _multiarray_umath.cp36-win_amd64.pyd produces:
Microsoft (R) COFF/PE Dumper Version 14.22.27905.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file _multiarray_umath.cp36-win_amd64.pyd
File Type: DLL
Image has the following dependencies:
mkl_rt.dll
python36.dll
KERNEL32.dll
VCRUNTIME140.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-utility-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
Summary
77000 .data
1000 .gfids
1C000 .pdata
30000 .rdata
3000 .reloc
1000 .rsrc
1BD000 .text
Of the dependencies, I can find mkl_rt.dll, python36.dll, and VCRUNTIME140.dll in the either the .whl or the venv. There are apparently two versions of KERNEL32.dll, one for x86 and one for x64. One of these resides in C:\Windows\System32.
The only information I could find about the api-ms-win-crt-*.dll are described on MSDN within the "Update for Universal C Runtime in Windows",
The Windows 10 Universal CRT is a Windows operating system component
that enables CRT functionality on the Windows operating system. This
update allows Windows desktop applications that depend on the Windows
10 Universal CRT release to run on earlier Windows operating systems.
These are for non-Windows 10 systems. There seems to be no "official" way to obtain them for a Windows 10 system. I was able to copy the dlls from a Windows 7 system. Naively putting them in the PYTHONPATH (not surprisingly) doesn't work. If they were to work at all, they would likely need to be registered. But 1) registering Windows 7 dlls on Windows 10, I am told, can make the system unstable and 2) definitely doesn't seem like a universal, portable solution.
First.
Build application with and without numpy. Check difference between builds.
Original error was: No module named 'numpy.core._multiarray_umath' Can means two things:
cannot found file
some dependence of this this pyd file is not satisfied. You may try to check it with http://dependencywalker.com/.
Yes, depending on how you define "install".
Numpy requires the use of .pyd files. A pyd file is essentially a dll. These are compiled code files which the Python interpreter references during run-time. Unfortunately, the Windows API calls used to load dll files requires them to exist at the filesystem level. When you try to import directly from a wheel, the pyd files (and the functions/classes they contain) aren't accessible. Hence, the error.
The (Relatively) Easy solution
One solution is to make the pyd files accessible on the filesystem. To do that, it's probably simplest to unpack the whole wheel file to disk1.
In that case, consider "how you define 'install'":
If you could run numpy straight from the wheel, wouldn't that mean numpy was installed?
What difference does it really make whether you're writing a wheel file or the contents of that wheel file?
Since the plugin system is downloading a wheel file onto the filesystem, the application must have write access. Download the wheel file and extract it into the plugins directory. As long as the plugin directory is on the PYTHONPATH (i.e. sys.path.append), the import will work. Smarts and so forth can be added from there. Of course, what level of smarts you need is a whole other issue. But that's the basic idea.
The Wizard's Path
Another solution is to create your own functionality to import a dll from memory. This is precisely the goal of Joachim Bauch's MemoryModule project. It seems the Py2Exe project used the MemoryModule at one point, through a module called zipextimporter.py. Unfortunately, the code is hard to find and seems outdated (Python 2.4). There is also a similar module called pymemimporter.py which is slightly more recent.
1 Numpy expects the pyd files to be in specific locations. If you want to extract only the pyd files, they'll need to be nested in the appropriate folders (e.g. numpy/core/_multiarray_umath.cp36-win_amd64.pyd). This is sufficient to import numpy, but unless the other modules are available, rather pointless.
Question:
I want to compile a third-party library like Pillow or Numbpy but I want to change the name of the python27.dll to corepython27.dll. How do I do this during the compile process? Is it something I need to change in the setup.py? Or the distutils library?
I should explain that I have no experience in compiling at all. I just know that I will need to make this change as I learn more about the basics of compiling.
Explanation:
Corel's PaintShop Pro uses an embedded python interpreter to run scripts inside the program. And I would like to be able to use third-party libraries like pillow and numpy but they always fail to load. The version of python that is included with PaintShop Pro is 2.7.5. I've made sure to download the appropriate versions of these libraries but it always fails with a "DLL module doesn't exist" type error.
Using a PE viewer I was able to see that other libraries like TKinter were using imports corepython27.dll instead of python27.dll like pillow was.
Also pillow for 2.7 was using msvcr90.dll but the custom version of the tkinter library included with PSP was compiled with msvcr110.dll. Do you think this will be an issue? Do I need to compile pillow with the appropriate version of msvcr DLL? Or is matching versions (2.7) and making sure it uses the correct python.dll (corepython27.dll) the only important thing?
You can create a symbolic link named corepython27.dll showing to the installed python27.dll. You can do this in your console via the command
MKLINK <path_to_corepython.dll> <path_to_python27.dll>
I'm trying to follow the instructions here to use the Boost.Python. The source code is in that webpage. I can compile, link this simple sample code but I cannot import the resulting module in python command line. It always error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named hello_ext
I have no idea what the matter is because that page just says: "That's it. We're done. We can now build this as a shared library. The resulting DLL is now visible to Python." This is my building environment:
Windows 7 64 bit, I am the Administrator and run cmd as Administrator
boost version is 1.64.0 (precompiled binary boost_1_64_0-msvc-14.0-64.exe downloaded from here)
python version is 2.7.13, 64 bit
Visual studio 2015, Update 3
The target is a DLL
The project name is ConsoleApplication1, so the output is ConsoleApplication1.dll. I have changed the filename to hello_ext.dll but same error.
I built with x64 configuration and I have verified with dumpbin that the output ConsoleApplication1.dll is really 64 bit
I have added the path ......\ConsoleApplication1\x64\Release which contains ConsoleApplication1.dll into sys.path inside python command line.
So, could you please tell me how to import the module in python? Thanks a lot.
I solved the problem myself. Thank jagerman for his useful suggestions.
(1) Just change the output filename from ConsoleApplication1.dll to hello_ext.pyd. You can automate this rename by setting Pages->General->Target Extension to ".pyd". Make sure the file hello_ext.pyd is in python's search path. You can just throw it to C:\Python27\DLLs which is one of python's built-in search paths.
(2) Now you will got a different import error: DLL load failed: The specified module could not be found. If you look closely at the file size of hello_ext.pyd, you'll likely notice something wired -- it's only 19KB. That means it doesn't contain everything needed to import into python, so python has to find the missing part to properly import it. Yes, you may guess that -- the only possible missing stuff is Boost.Python libraries, so add path to it into PATH environment variable -- for me, it is C:\local\boost_1_64_0\lib64-msvc-14.0.
Then the problem is solved. Note: some answers in other related questions may suggest build as a static library, That way, you will got another import error: DLL load failed: %1 is not a valid Win32 application. So just build as DLL. PS: you don't need to specify boost_python-vc140-mt-1_64.lib or boost_python-vc140-mt-gd-1_64.lib in Property Pages->Linker->Input->Additional Dependencies as some comments suggested.
I am trying to import the following api wrapper / device driver as in this python package:
import com.oceanoptics.omnidriver.api.wrapper.Wrapper
Python just returns that there is no module named like this:
ImportError: No module named com.oceanoptics.omnidriver.api.wrapper.Wrapper
I installed Omnidriver from the device manufacturer's website. Specifically, I used the installer OmniDriver-2.37-win32-installer.exe and installed the "Development version". It installs a bunch of dlls in C:\Program Files (x86)\Ocean Optics\OmniDriver\OOI_HOME.
The wrapper is working properly in Matlab after adding C:\Program Files (x86)\Ocean Optics\OmniDriver\OOI_HOME to C:\Program Files (x86)\MATLAB\R2012b\toolbox\local\librarypath.txt and C:\Program Files (x86)\Ocean Optics\OmniDriver\OOI_HOME\OmniDriver.jar to C:\Program Files (x86)\MATLAB\R2012b\toolbox\local\classpath.txt. Thereafter, I can load the wrapper in Matlab with wrapper = com.oceanoptics.omnidriver.api.wrapper.Wrapper().
I guess my python installation (Enthought Canopy 1.4.1 win 32bit) is not looking for the dlls in the correct path because I would have to tell first.
So, my question is, how do I instruct python to successfully execute the import statement above?
Another approach for interfacing to the spectrometer using Python would be to use the python-seabreeze package. The package does not have thorough documentation, but if you're willing to be patient and try things out for yourself, then you should be able to get it to work. The author has put considerable work into making the package compatible with the vast majority of Ocean Optics' spectrometers. I just finished installing it on my Windows laptop and got it to work in under an hour.
I checked the website, and can't find any reference to python support. I believe the instructions you referenced are instructions how to install the java classes. I could find no information that mentioned or discussed python modules. You should contact Oceanview for clarification.
I have just installed OpenCV on my Windows 7 machine. As a result, I get a new directory:
C:\OpenCV2.2\Python2.7\Lib\site-packages
In this directory, I have two files: cv.lib and cv.pyd.
Then I try to use the opencv from Python. I do the following:
import sys
sys.path.append('C:\OpenCV2.2\Python2.7\Lib\site-packages')
import cv
As a result I get the following error message:
File "<stdin>", line 1, in <module>
ImportError: DLL load failed: The specified module could not be found.
What am I doing wrong?
ADDED
As it was recommended here, I have copied content of C:\OpenCV2.0\Python2.6\Lib\site-packages to the C:\Python26\Lib\site-packages. It did not help.
ADDED 2
My environment variables have the following values:
Path=C:\Program Files\MiKTex\miktex\bin;C:\OpenCV2.2\bin;C:\Python26;
PYTHONPATH=C:\OpenCV2.2\Python2.7\Lib\site-packages
Do I need to change something? Do I need to add something?
ADDED 3
I think my question is general: How to use a library? Probably I need to find a *.ddl file somewhere? Then I need to use the name of the directory containing this file as a value to some environment variables? Or maybe I need to use sys.addpath? I also need to know how the way to call the library is related to the name of the file that contains the library.
ADDED 4
It is interesting that when I type import cv, I get:
ImportError: DLL load failed: The specified module could not be found.
But when I type import opencv I get:
ImportError: No module named opencv
ADDED 5
It has been suggested that I usthe e inconsistent version of python. In more details, OpenCV tries to use Python2.7 and I had Python2.6. So, I have installed Python 2.7. It makes difference. Now I do not have the old error message, but I have a new one:
ImportError: numpy.core.multiarray failed to import
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: numpy.core.multiarray failed to import
ADDED 6
I have managed to resolve the problem by installing numpy. It took some time because I did not realized that there are different numpy installer corresponding to different versions of Python. Some details can be found in my answer to my own question (see bellow).
The problem was resolved. The following steps has been done:
A new version of python (version 2.7) has been installed.
After that I still was unable to run OpenCV because I had some problems with the numpy library.
I tired to install numpy but the installer did not see my new version of the Python.
I deleted the old version of Python as well as links to the old version in the Path system vatriable.
After that numpy installer was not able to finish the installation.
I have realized that I need to run another numpy installer that is associated with the Python 2.7. It can be found here.
Finally everything worked. I was able to "import cv".
I suspect you have the same problem I've run into. If you have a 64-bit version of Python, it cannot load 32-bit DLLs. OpenCV currently only ships 32-bit binaries. If you want 64-bit .pyd and .dll files, you have to compile them yourself. There are some instructions on the OpenCV Wiki, but it's not for the faint of heart. Expect to have a substantial time investment.
The easiest solution is to:
Uninstall 64-bit Python
Install a 32-bit distribution.
The PythonXY distribution includes pyopencv -- a good set of OpenCV hooks. The only limitation is that it's 32-bit, so don't make plans to process gigapixel astronomy data with it! ;)
If you must have the 64-bit version, follow these instructions to get it OpenCV to compile with Visual Studio 2010. There's a discussion on stackoverflow that describes building 64-bit apps with VC Express.
EDIT: OpenCV now ships with 64-bit Python binaries. The .dll files need to go somewhere in your path (I put them in the scripts folder), and the .pyd files go in your site-packages directory.
I had trouble interfacing OpenCV with Python, and I was looking all over the place for help. Here's what worked for me. I basically followed this post: http://opencvpython.blogspot.com/2012/05/install-opencv-in-windows-for-python.html. After downloading and extracting OpenCV 2.4.6, you basically get a folder called "opencv" with a bunch of stuff in it. Navigate to build->python->2.7. Inside, there is only one file called "cv2.pyd". I copied this file and pasted it in "python-2.7.5\Lib\site-packages". I'm actually using the Spyder IDE, and it works fine. In the python interpreter, typing in "import cv" worked for me.
Maybe you should edit your environment variable
right click on the "My Computer" or something like this, click on properties.
In the properties window click on the Advanced tab.
Then, the environment variables button.
Change the path.