Develop Reference

fatal error: 'Eigen/Dense' file not found

I am an Ultra Linux newbie, and I am trying to install this program and when I try to build the python wrapper I'd get this
~/Downloads/DeepMimic-master/DeepMimicCore$ make python
clang++ -c -g -std=c++11 -O3 -Wall -fPIC -I./ -I../../libraries/eigen -I../../libraries/bullet3/src -I/usr/include/python3.6m -I/usr/lib/ -lpython3.6m -o objs/Main.o Main.cpp
clang: warning: -lpython3.6m: 'linker' input unused [-Wunused-command-line-argument]
In file included from Main.cpp:3:
In file included from ./DeepMimicCore.h:3:
In file included from ./util/ArgParser.h:6:
./util/MathUtil.h:5:10: fatal error: 'Eigen/Dense' file not found
#include "Eigen/Dense"

You're missing a dependency, Eigen, which is listed under 'Dependencies' in the DeepMimic readme.
I see this problem has been encountered before:
fatal error: Eigen/Dense: No such file or directory

Looks like that program depends on Eigen. Try download Eigen and putting it in the appropriate directory.
Eigen is a template library, so you just have to download it, unzip it, and copy the folder called Eigen inside of the directory of the program.
Eigen website

Issue creating f2py shared object file when linking to libraries and modules

I am able to compile a fortran program normally with these commands:
ifort -O2 -I/work/GSI/include -c create_upperair.f90
ifort -o create_upperair.exe -O2 create_upperair.o -L/work/GSI/lib -lbufr
I am trying to take the same fortran program, but compile it as a python shared object file so that I can call this subroutine from a python script. However the f2py command is giving me issues.
Here is the command I am currently using:
f2py -c -L/work/GSI/lib -lbufr -I/work/GSI/include --f90flags=-O2 --fcompiler=ifort -m create_upperair create_upperair.f90
However that prints out errors at the end of the f2py command with something like:
ld: /work/GSI/lib/libbufr.a(closbf.o): relocation R_X86_64_32S against `nulbfr_' can not be used when making a shared object: recompile with -fPIC
I tried to add --f90flags="-O2 -fPIC" to my f2py command, but it didn't work either. Any ideas of what direction to go with this problem would be much appreciated.
Here is the full error at the end of f2py (note that I changed a few directory names above for simplicity):
ifort:f90: prepbufr_encode_upperair.f90
/glade/apps/opt/modulefiles/ys/cmpwrappers/ifort -shared -shared -nofor_main /tmp/tmpzYKBY5/tmp/tmpzYKBY5/src.linux-x86_64-2.7/prepbufr_encode_upperairmodule.o /tmp/tmpzYKBY5/tmp/tmpzYKBY5/src.linux-x86_64-2.7/fortranobject.o /tmp/tmpzYKBY5/prepbufr_encode_upperair.o - L/glade/p/work/sdegelia/softwares/comGSI_v3.3/lib -L/glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib -lbufr_i4r8 -lpython2.7 -o ./prepbufr_encode_upperair.so
ld: /glade/p/work/sdegelia/softwares/comGSI_v3.3/lib/libbufr_i4r8.a(closbf.o): relocation R_X86_64_32S against `nulbfr_' can not be used when making a shared object; recompile with -fPIC
/glade/p/work/sdegelia/softwares/comGSI_v3.3/lib/libbufr_i4r8.a: could not read symbols: Bad value
ld: /glade/p/work/sdegelia/softwares/comGSI_v3.3/lib/libbufr_i4r8.a(closbf.o): relocation R_X86_64_32S against `nulbfr_' can not be used when making a shared object; recompile with -fPIC
/glade/p/work/sdegelia/softwares/comGSI_v3.3/lib/libbufr_i4r8.a: could not read symbols: Bad value
error: Command "/glade/apps/opt/modulefiles/ys/cmpwrappers/ifort -shared -shared -nofor_main /tmp/tmpzYKBY5/tmp/tmpzYKBY5/src.linux-x86_64-2.7/prepbufr_encode_upperairmodule.o /tmp/tmpzYKBY5/tmp/tmpzYKBY5/src.linux-x86_64-2.7/fortranobject.o /tmp/tmpzYKBY5/prepbufr_encode_upperair.o -L/glade/p/work/sdegelia/softwares/comGSI_v3.3/lib -L/glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib -lbufr_i4r8 -lpython2.7 -o ./prepbufr_encode_upperair.so" failed with exit status 1

How can I wrap a C-Library in SWIG, which has usually to be linked during C-compilation?

Given a C-library, which has to be linked during compilation if I want to use its functions. I want to access these functions in Python using SWIG. I can only find examples and introductions where C-Code (example.c) is wrapped using SWIG, no method how to wrap a dynamic library (example.so).

All you need to do to make the .so (or .a) library case work is to link the library appropriately when you do the compile step of the example build process. You will still have to compile the example_wrap.c that gets generated, this is where you can link against things.
So modified from the SWIG docs that would be:
$ swig -python example.i
$ gcc -O2 -fPIC -c example.c
$ gcc -O2 -fPIC -c example_wrap.c -I/usr/local/include/python2.5
$ gcc -shared example_wrap.o -o _example.so -lmylib.so
In reality you can also skip this at the compile time linker step and use dlopen at runtime instead by injecting some extra code into the Python part of your module that calls dlopen before the shared object from SWIG gets loaded.

'ImportError' in Python extension module wrapping C library

(UPDATE 3 contains the questions I'd like to get answers to. UPDATE 2 refers to corrections I did trying to understand and fix this issue)
I'm trying to get a Python extension module to wrap a C library (in this case, it is only an example described in the book Python Cookbook (3er edition.) The problem is that I'm encountering the classic error
ImportError: dynamic module does not define init function (initsample)
when I'm trying to import the module.
The book itself has the following comment:
For all of the recipes that follow, assume that the preceding code is found in a file named
sample.c, that definitions are found in a file named sample.h and that it has been com‐
piled into a library libsample that can be linked to other C code. The exact details of
compilation and linking vary from system to system, but that is not the primary focus.
It is assumed that if you’re working with C code, you’ve already figured that out.
And I think I'm getting that wrong. This is what I'm doing: First of all, I have the following files:
➜ Sample ls
csample.pxd sample.c sample.h sample.pyx setup.py
The content of the files is according to this github repository containing the files described in the book. Assuming everything is copied correctly. I then compile sample.c
➜ Sample gcc -c -fPIC -I/path/to/python2.7 sample.c
This creates sample.o and I proceed to create a shared library:
➜ Sample gcc -shared sample.o -o libsample.so
It creates a libsample.so and finally run the setup.py file:
➜ Sample python setup.py build_ext --inplace
running build_ext
skipping 'sample.c' Cython extension (up-to-date)
building 'sample' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/include/python2.7 -c sample.c -o build/temp.linux-x86_64-2.7/sample.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/sample.o -L. -L/path/to/lib -lsample -lpython2.7 -o /path/to/sample.so
However, I obtain the same error. What am I missing?
This is my setup.py (eventually, I copied a part to clean previous build):
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import sys
import os
import shutil
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("sample") and not(name.endswith(".pyx") or name.endswith(".pxd") or name.endswith(".h"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
ext_modules = [
Extension('sample',
['sample.pyx'],
libraries=['sample'],
library_dirs=['.'])]
setup(
name = 'Sample extension module',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
UPDATE 2
I was wondering if it is normal that after running
python setup.py build_ext --inplace
the file sample.c gets replaced with a cythonized version.
I assume that running setup.py is using cython sample.c, but I'm asking this because yesterday I was mistakenly compiling the cythonized version sample.c (after removing libsample.so, sample.o, sample.so that were created in a previous run) and after updating sample.pyx, I got this error:
>>> import sample
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./sample.so: undefined symbol: divide
After this, I restored sample.c to its original content, removed all remaining files from previous builds (including temporal files) and crucially, I had to change sample.pyx in some way (e.g. adding a blank line.)
I noticed I have to do this to avoid the
ImportError: dynamic module does not define init function (initsample)
error. Then I proceeded as usual:
➜ Sample gcc -c -fPIC -I/path/to/python2.7 sample.c
➜ Sample gcc -shared sample.o -o libsample.so
➜ Sample python setup.py build_ext --inplace
running build_ext
cythoning sample.pyx to sample.c
warning: sample.pyx:27:42: Use boundscheck(False) for faster access
building 'sample' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/include/python2.7 -c sample.c -o build/temp.linux-x86_64-2.7/sample.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/sample.o -L. -L/path/to/lib -lsample -lpython2.7 -o /path/to/Sample/sample.so
➜ Sample python
>>> import sample
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libsample.so: cannot open shared object file: No such file or directory
This time I get cythoning sample.pyx to sample.c in the output above and a new error. However, libsample.so is in the Sample directory. What could be the issue here?
UPDATE 3:
I finally was able to import this module successfully but the last error
ImportError: libsample.so: cannot open shared object file: No such file or directory
implied that I needed to set the current directory in the environment variable LD_LIBRARY_PATH. However, I did it in a crude way:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/Sample
What is the correct way to do this? Maybe using some parameter in setup.py? Regarding the several issues I found, I'd like to get an answer as to why all of this happened:
sample.c gets replaced when running python setup.py build_ext --inplace Is this okay?
Do I need to restore sample.c every time I'm doing this procedure?
Do I need to change sample.pyx in order to get a correct output (instead of skipping 'sample.c' Cython extension (up-to-date) with a misleading output)
What is the correct way to solve ImportError: libsample.so: cannot open shared object file: No such file or directory?
A possible solution to 4. is to add in setup.py:
runtime_library_dirs=['./'],
However, this requires libsample.so to be located in the same directory as setup.py.

Compile Cython extensions from the command line with gcc (mingw32) on windows

I'm trying to test a small cython module on win32, and I'm having trouble building it.
The file is called linalg_cython.pyx and has these contents:
from __future__ import absolute_import, division, print_function
import numpy as np
cimport numpy as np
import cython
##cython.boundscheck(False)
#np.ndarray[np.float32]
##cython.wraparound(False)
def L2_sqrd_float32(np.ndarray hist1, np.ndarray hist2):
""" returns the squared L2 distance
seealso L2
Test:
hist1 = np.random.rand(4, 2)
hist2 = np.random.rand(4, 2)
out = np.empty(hist1.shape, dtype=hist1.dtype)
"""
return (np.abs(hist1 - hist2) ** 2).sum(-1) # this is faster
L2_sqrd = L2_sqrd_float32
I was able to get this compiling by using a setup.py, but I don't want to have to rely on setup.py to build the extensions. This is because I haven't fully understood the cython compilation process yet. I want to compile it on my own first before I start trusting setup.py. That being said I was able get a good start by looking at the output of "setup.py build_ext":
C:\MinGW\bin\gcc.exe -mdll -O -Wall ^
-IC:\Python27\Lib\site-packages\numpy\core\include ^
-IC:\Python27\include -IC:\Python27\PC ^
-c vtool\linalg_cython.c ^
-o build\temp.win32-2.7\Release\vtool\linalg_cython.o
writing build\temp.win32-2.7\Release\vtool\linalg_cython.def
C:\MinGW\bin\gcc.exe -shared \
-s \
build\temp.win32-2.7\Release\vtool\linalg_cython.o \
build\temp.win32-2.7\Release\vtool\linalg_cython.def \
-LC:\Python27\libs \
-LC:\Python27\PCbuild \
-lpython27 \
-lmsvcr90 \
-o build\lib.win32-2.7\vtool\linalg_cython.pyd
The pyd file that this created seemed to work, but my goal is understanding, not just making it work.
Copying this format (and trying some things myself) I'm currently using these commands to build everything manually.
C:\Python27\Scripts\cython.exe vtool\linalg_cython.pyx
C:\MinGW\bin\gcc.exe -mdll -O -DNPY_NO_DEPRECATED_API -Wall -Wno-unknown-pragmas
-Wno-format -Wno-unused-function -m32 -shared
-IC:\Python27\Lib\site-packages\numpy\core\include -IC:\Python27\include
-IC:\Python27\PC -IC:\Python27\Lib\site-packages\numpy\core\include
-LC:\Python27\libs -LC:\Python27\PCbuild -lpython27 -lmsvcr90 -o
vtool\linalg_cython.pyd -c vtool\linalg_cython.c
The main difference between my command and the setup.py command is that I'm trying to call gcc in one line instead of splitting it up into two lines. I would have called it in two commands, but the def file seems to be autogenerated by setup.py and I'm not sure what its all about.
Its contents seem simple:
LIBRARY linalg_cython.pyd
EXPORTS
initlinalg_cython
but I'd like to know more about what it is before I split my command into two steps and autogenerate this def file myself. Either way shouldn't it be possible to create the .pyd in one call to gcc?
With the command that I'm using I'm able to get a .pyd file in the right place, but when I try to import it I get
<type 'exceptions.ImportError'>: DLL load failed: %1 is not a valid Win32 application.
which is supposed to be a x86/x64 mismatch, hence why I tried adding the flag -m32.
In summary: When trying to comiple a simple cython module my gcc command is giving me 32/64 bit errors. How do I fix my gcc command such that it generates a valid 32 bit pyd file.

You didn't mention if your python is 32bit or 64bit..this kind of behavior usually happens when you try to import 32bit module in 64bit python, or vice versa.
Make sure your python and module, you`re trying to import, are same bit architecture. Easiest way to fix this is by downloading and installing right python.