Develop Reference

python doesn't recognize a folder

I'm working on linux ubuntu 20.04. I opened new python project using Pycharm IDE, and I've installed a package called aihwkit, the documentation and the source.
When running the examples given with the source code, which use modules imported from a directory called inference such as example 06_lenet5_hardware_aware.py I'm getting an error:
ModuleNotFoundError: No module named 'aihwkit.inference'
although other folders, in the same directory as inference are imported as well and they work well. I'm trying to import this manually, but not sure how to do it.
This is the hierarchy of the folders: I'm calling inference like this:
from aihwkit.inference import PCMLikeNoiseModel
from the file examples/06_lenet5_hardware_aware.py when the module PCMLikeNoiseModel source code lies in the following path: src/aihwkit/inference/noise/pcm.py
please note that the problem is that the name aihwkit.inference is not found, while other names such as aihwkit.nn do not raise any error, and they reside in similar path to inference.
I'm adding a picture of the hierarchy in case it helps:
how can I import this folder manually?
Thank you

Since you're on Linux, you can try to install your package before importing it with:
sudo apt-get install python-#name_of_your_package
written in your terminal, or you can try to install it with pip

How to specify the dynamic library location when writing a setup file for python c-extension?

I'm trying to write a c-extension for python and I'm using a library (FFTW) which has some dynamic library files (.dll). So I wrote some c code and also a setup.py file to install it and when I run the setup file everything goes well and my c-extensions will be installed successfully. But when I try to use my new module I will encounter the following error:
ImportError: DLL load failed: The specified module could not be found.
So I assume it is referring to the dll files of FFTW library. If I copy the dll files to my C:/windows/system32 My code will run without any error. But I wish not to do that and instead have the dll files alongside my code.
According to python documentation for writing the setup script:
You can also specify the libraries to link against when building your
extension, and the directories to search for those libraries. The
libraries option is a list of libraries to link against, library_dirs
is a list of directories to search for libraries at link-time, and
runtime_library_dirs is a list of directories to search for shared
(dynamically loaded) libraries at run-time.
But when I use the runtime_library_dirs option and try to install my extension by running the setup.py I will encounter the following error.
warning: I don't know what to do with 'runtime_library_dirs':
['./c-extension/bin']
error: don't know how to set runtime library
search path for MSVC
I'm using python 3.6.
Here is a sample of my setup.py code:
import numpy
from distutils.core import setup, Extension
def main():
setup(name="simple_ext",
version="1.0.0",
description="Python interface for the simple C extension library function",
author="My Name",
author_email="my_email#email.com",
data_files=[('./c-extension/bin', ['libfftw3-3.dll','libfftw3l-3.dll','libfftw3f-3.dll'])],
ext_modules=[Extension("simple_ext", ["simple_ext.cpp"],
include_dirs=[numpy.get_include(), './c-extension/include']),
runtime_library_dirs=['./c-extension/bin'],
library_dirs=['./c-extension/lib'],
libraries = ['libfftw3-3','libfftw3l-3','libfftw3f-3']])
if __name__ == "__main__":
main()
So does anyone any idea what wrong and what should I do to fix it?
Edit:
So based on Patrik Polakovic's comment I managed to somehow solve the problem. The thing is when I used data_files like this:
data_files=[('./c-extension/bin', ['libfftw3-3.dll','libfftw3l-3.dll','libfftw3f-3.dll'])],
There was an hidden error message that I had missed and apparently was ignored by the compiler too.
error: can't copy 'libfftw3-3.dll': doesn't exist or not a regular
file
But when I changed it to the form below:
data_files=[('', ['./c-extension/bin/libfftw3-3.dll','./c-extension/bin/libfftw3l-3.dll','./c-extension/bin/libfftw3f-3.dll'])],
the setup was able to locate my shared library files and the following message was also displayed:
copying c-extension\bin\libfftw3-3.dll ->
C:\Users\user\.conda\envs\newEnv\
copying c-extension\bin\libfftw3l-3.dll ->
C:\Users\user\.conda\envs\newEnv\
copying c-extension\bin\libfftw3f-3.dll ->
C:\Users\user\.conda\envs\newEnv\
So I can use this as a temporary solution. But I still don't know what's wrong with the first form of data_files option that I used (Which looks legit to me) and more importantly I don't understand what's wrong with using the runtime_library_dirs option.

Yes, on Windows runtime_library_dirs does not work and will cause MSVC to error.
In your original question, you used data_files incorrectly, as from the relevant part of the setuptools docs on writing the setup script, note the sentence
Each (directory, files) pair in the sequence specifies the installation directory and the files to install there.
Your DLLs are located in ./c-extension/bin, so you need to prepend that relative path to the names of each DLL. The following from the same page explains directory concisely:
The directory should be a relative path. It is interpreted relative to the installation prefix (Python’s sys.prefix for system installations; site.USER_BASE for user installations).
That is, the files specified in files will be copied to C:\Users\user\.conda\envs\newEnv\directory. So passing ./c-extension/bin to directory is also a mistake.

Using modulefinder in python 2.7 to discover dependencies in a pyd file

I am trying to load a module from a .pyd file I have downloaded from a GitHub repo. I am running python in the directory with the .pyd file and references to other files in the directory work, so I think the problem isn't a path issue. Other files in the GitHub show how the module is loaded by that author, so it isn't a naming problem, either.
When I attempt to load module XYZ from XYZ.pyd I get the following error:
ImportError: DLL load failed: The specified module could not be found.
Searching this error tells me that the problem is likely that XYZ.pyd depends on another DLL that I don't have, so I want to trace dependency. I'm using modulefinder, which is part of the python standard library.
When I try to run modulefinder on the module in XYZ.pyd, I get the following error:
TypeError: compile() expected string without null bytes
Search results for this error are mostly about JSON, which I'm not sure have anything to do with this problem. I can get modulefinder to work on ordinary .py files that I have written myself.
My questions are: is it generally possible to use modulefinder on a .pyd file, and if so, is the syntax different from the the usage given in the manual. More specifically, how do I use modulefinder to see the dependencies on a .pyd file?

The dll created by Boost.Python cannot be imported (following Boost Python's QuickStart)

I'm trying to follow the instructions here to use the Boost.Python. The source code is in that webpage. I can compile, link this simple sample code but I cannot import the resulting module in python command line. It always error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named hello_ext
I have no idea what the matter is because that page just says: "That's it. We're done. We can now build this as a shared library. The resulting DLL is now visible to Python." This is my building environment:
Windows 7 64 bit, I am the Administrator and run cmd as Administrator
boost version is 1.64.0 (precompiled binary boost_1_64_0-msvc-14.0-64.exe downloaded from here)
python version is 2.7.13, 64 bit
Visual studio 2015, Update 3
The target is a DLL
The project name is ConsoleApplication1, so the output is ConsoleApplication1.dll. I have changed the filename to hello_ext.dll but same error.
I built with x64 configuration and I have verified with dumpbin that the output ConsoleApplication1.dll is really 64 bit
I have added the path ......\ConsoleApplication1\x64\Release which contains ConsoleApplication1.dll into sys.path inside python command line.
So, could you please tell me how to import the module in python? Thanks a lot.

I solved the problem myself. Thank jagerman for his useful suggestions.
(1) Just change the output filename from ConsoleApplication1.dll to hello_ext.pyd. You can automate this rename by setting Pages->General->Target Extension to ".pyd". Make sure the file hello_ext.pyd is in python's search path. You can just throw it to C:\Python27\DLLs which is one of python's built-in search paths.
(2) Now you will got a different import error: DLL load failed: The specified module could not be found. If you look closely at the file size of hello_ext.pyd, you'll likely notice something wired -- it's only 19KB. That means it doesn't contain everything needed to import into python, so python has to find the missing part to properly import it. Yes, you may guess that -- the only possible missing stuff is Boost.Python libraries, so add path to it into PATH environment variable -- for me, it is C:\local\boost_1_64_0\lib64-msvc-14.0.
Then the problem is solved. Note: some answers in other related questions may suggest build as a static library, That way, you will got another import error: DLL load failed: %1 is not a valid Win32 application. So just build as DLL. PS: you don't need to specify boost_python-vc140-mt-1_64.lib or boost_python-vc140-mt-gd-1_64.lib in Property Pages->Linker->Input->Additional Dependencies as some comments suggested.

python, unable to make executable from script when I use scipy

I have a python application and I want to extract the executable. I have used py2exe with no problem in other scripts. But in this one I get errors that I think have to do with the fact that I import and use scipy.
My code runs fine in eclipse. Trying to make the procedure with scipy at first I got 3 dlls missing ("libmmd.dll","MSVCP90.dll","libifcoremd.dll"). At first I exluded these from setup.py and the .exe file was made but it did not run. After that I downloaded the dll files (also deleted the exclusion of setup.py) and added them in "C:\Python27\DLLs" and also "C:\Windows\System32" and "C:\Windows\SysWOW64" (I have 64 bit windows7). Again the exe is created in the folder dist, but trying to run it I get this:
http://imageshack.us/photo/my-images/145/py2exe.png/
Then I tried to do it with cx_Freeze. Again the .exe file was created but I get this error:
http://img521.imageshack.us/img521/3843/cxfreeze.png
So I suppose it has nothing to do with py2exe or cx_Freeze, but with the fact that scipy needs some dependencies that for some reason work in eclipse, but not when I make the executable.
Has anybody faced this problem? Any ideas on what I should do?
Thank you in advance!
EDIT: I did it with pyinstaller-2.0. It did not run at first. Then I added the DLLs to the folder, after that I added numpy and scipy to the folder and finally after an error message I added kdtree.py to the folder. I was thrilled to see that it runned! But then dissapointed again cause it runs ONLY in my pc (some useful .exe there)! I added the folder it to a dropbox folder, from where it runs in me with no problem but not for others. The error in other machines I tried is:
importError: DLL load failed: %1 is not a valid Win32 application

You could try putting the DLL files directly next to your executable. Also my py2exe gives some information about possible binary files that you might need to distribute with your application:
*** binary dependencies ***
Your executable(s) also depend on these dlls which are not included,
you may or may not need to distribute them.
Make sure you have the license if you distribute any of them, and
make sure you don't distribute files belonging to the operating system.
//Snip list of paths to dlls