I'm trying to produce a simple fibonacci algorithm with Cython.
I have fib.pyx:
def fib(int n):
cdef int i
cdef double a=0.0, b=1.0
for i in range(n):
a, b = a + b, a
return a
and setup.py in the same folder:
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize('fib.pyx'))
Then I open cmd and cd my way to this folder and build the code with (I have [http://www.microsoft.com/en-us/download/details.aspx?id=44266](this compiler) :
python setup.py build
Which produces this result:
C:\Users\MyUserName\Documents\Python Scripts\Cython>python setup.py build
Compiling fib.pyx because it changed.
Cythonizing fib.pyx
running build
running build_ext
building 'fib' extension
creating build
creating build\temp.win-amd64-2.7
creating build\temp.win-amd64-2.7\Release
C:\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Anaconda\include -IC:
\Anaconda\PC -c fib.c -o build\temp.win-amd64-2.7\Release\fib.o
writing build\temp.win-amd64-2.7\Release\fib.def
creating build\lib.win-amd64-2.7
C:\Anaconda\Scripts\gcc.bat -DMS_WIN64 -shared -s build\temp.win-amd64-2.7\Relea
se\fib.o build\temp.win-amd64-2.7\Release\fib.def -LC:\Anaconda\libs -LC:\Anacon
da\PCbuild\amd64 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\fib.pyd
So it seems the compiling worked and I should be able to import this module with
import fib
ImportError: No module named fib
Where did I go wrong?
Edit:
os.getcwd()
Out[6]: 'C:\\Users\\MyUserName\\Documents\\Python Scripts\\Cython\\build\\temp.win-amd64-2.7\\Release'
In [7]: import fib
Traceback (most recent call last):
File "<ipython-input-7-6c0ab2f0a4e0>", line 1, in <module>
import fib
ImportError: No module named fib
The compiling worked however the library was placed under: build\lib.win-amd64-2.7 so you have to either copy/move the file in the current directory or change your current directory to that one.
You can see this in the last part of the command run:
-o build\lib.win-amd64-2.7\fib.pyd
The -o option stands for output and specifies where it should put the final compiled file.
Use
python setup.py build_ext --inplace
--inplace flag will put your pyd in the working directory
Related
I'm having some trouble with cython. I recently found out about it and decided to use it in a project, but for some reason I'm unable to import the module/function that I've just compiled.
Here is what my sum.pyx file looks like:
def add(x, y):
return x+y
And this is my setup.py file:
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize('image_converter.pyx'))
When I run python setup.py build_ext --inplace, the output shows this error:
Compiling sum.pyx because it changed.
[1/1] Cythonizing sum.pyx
C:\Users\User-name\AppData\Local\Programs\Python\Python39\lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\User-name\Documents\Cython\sum.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
running build_ext
building 'sum' extension
error: Unable to find vcvarsall.bat
Only one file was created called sum.c, but I'm aware that there should be some .so file created in a new directory but I couldn't find any, so I just continued and created the test.py file to see if the cython file worked:
import sum
print(sum.add(5,6))
But if I run this I get the error:
ModuleNotFoundError: No module named 'sum'
I am using Python 3.9.7 and Visual Studio Code, and I'm using the g++ compiler (MSYS2).
Thanks in advance.
Checking dependencies with ldd on the *.so file produced by Cython, the dependencies contains myLib.o instead of libforcython.o
I do not grasp why it is trying to reach a myLib.o instead of libforcython as indicated in my setup.py .
During python execution of the module that produces an error similar to Cython unable to find shared object file . However contrary to the included links and answer, my problem does not seem to happen during the python initialization, but rather during the cythonization itself.
using these files:
example.pyx :
cdef extern from "myLib.h":
void testFunction ()
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
pythoncallsc_extension = Extension (
name = "pythoncallsc",
sources=["./example.pyx"],
libraries=["forcython"])
setup ( name = "pythoncallsc",
ext_modules = cythonize ([pythoncallsc_extension]))
When I look at the log generated by python3 setup.py build_ext --inplace I can then clearly see in the commandline launching gcc, and it contains:
... -lforcython -lpython3.7m ...
So clearly gcc is linking against my library libforcython.
The lib contains:
header myLib.h
generated libforcython.so file for the function void testFunction(void).
This lib is built separately and elsewhere in my system. I have checked the include and lib and they are clearly in the $PATH of my cython project.
The cythonization produces the library pythoncallsc.cpython-37m-x86_64-linux-gnu.so
But against all my expectations when I do:
ldd pythoncallsc.cpython-37m-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffee40fb000)
myLib.o => not found <=== HERE ???
libpython3.7m.so.1.0 => /path/to/lib...
libpthread.so.0 => ...
libc.so.6 => ...
...
Why is cython producing an output that depends on a myLib.o file and not on libforcython.so ?
This myLib.o file does not exists, and consequently that produces an error when I launch my module:
`ImportError: myLib.o: cannot open shared object file: No such file or directory`
Any clue ?
In case we have this error, we should check how the c library is built.
As suggested in the comments, the libforcython library was not built properly.
It was wrongly built using soname with gcc in the Makefile.
Correct:
gcc myLib.o -shared -fPIC -Wl,-soname,libforcython.so -o libforcython.so -lc
Incorrect:
gcc -shared -fPIC -Wl,-soname,myLib.o -o libforcython.so myLib.o -lc
In this case soname is not really useful because I don't use a version number for the library: a simpler use case.
I have a setup.py file that is very similar to the one shown here: https://stackoverflow.com/a/49866324/4080129. It looks like this:
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy
sources = ["hs/subfolder/detect.pyx",
"hs/subfolder/Donline.cpp",
"hs/subfolder/Handler.cpp",
"hs/subfolder/Process.cpp",
"hs/subfolder/Filter.cpp",
"hs/subfolder/Localize.cpp"]
exts = [Extension(name='hs.detect',
sources=sources,
extra_compile_args=['-std=c++11', '-O3'],
include_dirs=[numpy.get_include()])]
setup(
ext_modules=cythonize(exts),
include_dirs=[numpy.get_include()]
)
There's a package with some pure-Python, and a submodule that contains Cython files. The setup.py is in the parent folder, not in the Cython one:
setup.py
hs/
some_python.py
subfolder/
detect.pyx
Donline.cpp
...etc
Now, setup.py correctly compiles all the files module/submodule/file1.cpp etc. and saves the build to build/temp.linux-x86_64-3.6/module/submodule/file1.o .
However, just after that, it tries to compile a file called file1.cpp, which doesn't exist (the correct one is module/submodule/file1.cpp, and has already been compiled).
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ihs/subfolder -I/[...]/python3.6/site-packages/numpy/core/include -I/[...]/python3.6/site-packages/numpy/core/include -I/[...]/include -I/disk/scratch/miniconda/envs/my_default/include/python3.6m -c Donline.cpp -o build/temp.linux-x86_64-3.6/Donline.o -std=c++11 -O3
gcc: error: Donline.cpp: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4
I'm very confused, this completely prevents my code from compiling...
It turns out the .pyx file contains a line
# distutils: sources = Donline.cpp Handler.cpp Process.cpp Filter.cpp Localize.cpp
which tells distutils what to compile. I wasn't aware of it, and since it looks an awful lot like a commented-out line, I didn't realise it was there.
Cython tries to compile also these, other than the ones contained in the setup.pyfile, i.e. neither of the two sources list overrides the other. Apparently, these sources, despite being listed in the pyx file, which is in a subfolder, are expected to be in paths relative to the file where the setup.py file is, or perhaps relative to the folder I'm calling python from.
Anyway, removing the line solved the issue.
(UPDATE 3 contains the questions I'd like to get answers to. UPDATE 2 refers to corrections I did trying to understand and fix this issue)
I'm trying to get a Python extension module to wrap a C library (in this case, it is only an example described in the book Python Cookbook (3er edition.) The problem is that I'm encountering the classic error
ImportError: dynamic module does not define init function (initsample)
when I'm trying to import the module.
The book itself has the following comment:
For all of the recipes that follow, assume that the preceding code is found in a file named
sample.c, that definitions are found in a file named sample.h and that it has been com‐
piled into a library libsample that can be linked to other C code. The exact details of
compilation and linking vary from system to system, but that is not the primary focus.
It is assumed that if you’re working with C code, you’ve already figured that out.
And I think I'm getting that wrong. This is what I'm doing: First of all, I have the following files:
➜ Sample ls
csample.pxd sample.c sample.h sample.pyx setup.py
The content of the files is according to this github repository containing the files described in the book. Assuming everything is copied correctly. I then compile sample.c
➜ Sample gcc -c -fPIC -I/path/to/python2.7 sample.c
This creates sample.o and I proceed to create a shared library:
➜ Sample gcc -shared sample.o -o libsample.so
It creates a libsample.so and finally run the setup.py file:
➜ Sample python setup.py build_ext --inplace
running build_ext
skipping 'sample.c' Cython extension (up-to-date)
building 'sample' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/include/python2.7 -c sample.c -o build/temp.linux-x86_64-2.7/sample.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/sample.o -L. -L/path/to/lib -lsample -lpython2.7 -o /path/to/sample.so
However, I obtain the same error. What am I missing?
This is my setup.py (eventually, I copied a part to clean previous build):
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import sys
import os
import shutil
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("sample") and not(name.endswith(".pyx") or name.endswith(".pxd") or name.endswith(".h"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
ext_modules = [
Extension('sample',
['sample.pyx'],
libraries=['sample'],
library_dirs=['.'])]
setup(
name = 'Sample extension module',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
UPDATE 2
I was wondering if it is normal that after running
python setup.py build_ext --inplace
the file sample.c gets replaced with a cythonized version.
I assume that running setup.py is using cython sample.c, but I'm asking this because yesterday I was mistakenly compiling the cythonized version sample.c (after removing libsample.so, sample.o, sample.so that were created in a previous run) and after updating sample.pyx, I got this error:
>>> import sample
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./sample.so: undefined symbol: divide
After this, I restored sample.c to its original content, removed all remaining files from previous builds (including temporal files) and crucially, I had to change sample.pyx in some way (e.g. adding a blank line.)
I noticed I have to do this to avoid the
ImportError: dynamic module does not define init function (initsample)
error. Then I proceeded as usual:
➜ Sample gcc -c -fPIC -I/path/to/python2.7 sample.c
➜ Sample gcc -shared sample.o -o libsample.so
➜ Sample python setup.py build_ext --inplace
running build_ext
cythoning sample.pyx to sample.c
warning: sample.pyx:27:42: Use boundscheck(False) for faster access
building 'sample' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/include/python2.7 -c sample.c -o build/temp.linux-x86_64-2.7/sample.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/sample.o -L. -L/path/to/lib -lsample -lpython2.7 -o /path/to/Sample/sample.so
➜ Sample python
>>> import sample
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libsample.so: cannot open shared object file: No such file or directory
This time I get cythoning sample.pyx to sample.c in the output above and a new error. However, libsample.so is in the Sample directory. What could be the issue here?
UPDATE 3:
I finally was able to import this module successfully but the last error
ImportError: libsample.so: cannot open shared object file: No such file or directory
implied that I needed to set the current directory in the environment variable LD_LIBRARY_PATH. However, I did it in a crude way:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/Sample
What is the correct way to do this? Maybe using some parameter in setup.py? Regarding the several issues I found, I'd like to get an answer as to why all of this happened:
sample.c gets replaced when running python setup.py build_ext --inplace Is this okay?
Do I need to restore sample.c every time I'm doing this procedure?
Do I need to change sample.pyx in order to get a correct output (instead of skipping 'sample.c' Cython extension (up-to-date) with a misleading output)
What is the correct way to solve ImportError: libsample.so: cannot open shared object file: No such file or directory?
A possible solution to 4. is to add in setup.py:
runtime_library_dirs=['./'],
However, this requires libsample.so to be located in the same directory as setup.py.
I'm trying to test a small cython module on win32, and I'm having trouble building it.
The file is called linalg_cython.pyx and has these contents:
from __future__ import absolute_import, division, print_function
import numpy as np
cimport numpy as np
import cython
##cython.boundscheck(False)
#np.ndarray[np.float32]
##cython.wraparound(False)
def L2_sqrd_float32(np.ndarray hist1, np.ndarray hist2):
""" returns the squared L2 distance
seealso L2
Test:
hist1 = np.random.rand(4, 2)
hist2 = np.random.rand(4, 2)
out = np.empty(hist1.shape, dtype=hist1.dtype)
"""
return (np.abs(hist1 - hist2) ** 2).sum(-1) # this is faster
L2_sqrd = L2_sqrd_float32
I was able to get this compiling by using a setup.py, but I don't want to have to rely on setup.py to build the extensions. This is because I haven't fully understood the cython compilation process yet. I want to compile it on my own first before I start trusting setup.py. That being said I was able get a good start by looking at the output of "setup.py build_ext":
C:\MinGW\bin\gcc.exe -mdll -O -Wall ^
-IC:\Python27\Lib\site-packages\numpy\core\include ^
-IC:\Python27\include -IC:\Python27\PC ^
-c vtool\linalg_cython.c ^
-o build\temp.win32-2.7\Release\vtool\linalg_cython.o
writing build\temp.win32-2.7\Release\vtool\linalg_cython.def
C:\MinGW\bin\gcc.exe -shared \
-s \
build\temp.win32-2.7\Release\vtool\linalg_cython.o \
build\temp.win32-2.7\Release\vtool\linalg_cython.def \
-LC:\Python27\libs \
-LC:\Python27\PCbuild \
-lpython27 \
-lmsvcr90 \
-o build\lib.win32-2.7\vtool\linalg_cython.pyd
The pyd file that this created seemed to work, but my goal is understanding, not just making it work.
Copying this format (and trying some things myself) I'm currently using these commands to build everything manually.
C:\Python27\Scripts\cython.exe vtool\linalg_cython.pyx
C:\MinGW\bin\gcc.exe -mdll -O -DNPY_NO_DEPRECATED_API -Wall -Wno-unknown-pragmas
-Wno-format -Wno-unused-function -m32 -shared
-IC:\Python27\Lib\site-packages\numpy\core\include -IC:\Python27\include
-IC:\Python27\PC -IC:\Python27\Lib\site-packages\numpy\core\include
-LC:\Python27\libs -LC:\Python27\PCbuild -lpython27 -lmsvcr90 -o
vtool\linalg_cython.pyd -c vtool\linalg_cython.c
The main difference between my command and the setup.py command is that I'm trying to call gcc in one line instead of splitting it up into two lines. I would have called it in two commands, but the def file seems to be autogenerated by setup.py and I'm not sure what its all about.
Its contents seem simple:
LIBRARY linalg_cython.pyd
EXPORTS
initlinalg_cython
but I'd like to know more about what it is before I split my command into two steps and autogenerate this def file myself. Either way shouldn't it be possible to create the .pyd in one call to gcc?
With the command that I'm using I'm able to get a .pyd file in the right place, but when I try to import it I get
<type 'exceptions.ImportError'>: DLL load failed: %1 is not a valid Win32 application.
which is supposed to be a x86/x64 mismatch, hence why I tried adding the flag -m32.
In summary: When trying to comiple a simple cython module my gcc command is giving me 32/64 bit errors. How do I fix my gcc command such that it generates a valid 32 bit pyd file.
You didn't mention if your python is 32bit or 64bit..this kind of behavior usually happens when you try to import 32bit module in 64bit python, or vice versa.
Make sure your python and module, you`re trying to import, are same bit architecture. Easiest way to fix this is by downloading and installing right python.