Develop Reference

How can I force python build to use existing successfully configured CMakeCache?

I successfully build PyTorch using cmake, of course, what's built out is some libxxx.so files, specifically for C/C++ to call.
To continue building a Python version out, I tried the following command, but it looks it's restart everything from scratch, which is NOT expected. Just wonder if there is a method when I try to build a Python version, python setup.py build will be able to force to use the existing CMakeCache.txt as the specified configuration?
Thank you ...
➜ pytorch git:(master) ✗ TORCH_CUDA_ARCH_LIST="7.5" NO_TEST=1 USE_MKLDNN=0 BUILD_CAFFE2=1 python setup.py build -cmake
Building wheel torch-1.12.0a0+git1c5a812
-- Building version 1.12.0a0+git1c5a812
cmake -GNinja -DBUILD_CAFFE2=1 -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=......pytorch/torch -DCMAKE_PREFIX_PATH=/usr/lib/python3.8/site-packages -DCUDA_NVCC_EXECUTABLE=/usr/local/cuda/bin/nvcc -DCUDNN_INCLUDE_DIR=/usr/local/cuda/include -DCUDNN_LIBRARY=/usr/local/cuda/lib64/libcudnn.so -DJAVA_HOME=/usr/lib/jvm/default-java -DNUMPY_INCLUDE_DIR=~/.local/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/usr/bin/python -DPYTHON_INCLUDE_DIR=/usr/include/python3.8 -DPYTHON_LIBRARY=/usr/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=1.12.0a0+git1c5a812 -DUSE_MKLDNN=0 -DUSE_NUMPY=True ......pytorch
CMake Error: Error: generator : Ninja
Does not match the generator used previously: Unix Makefiles
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.

numpy/ndarrayobject.h: No such file or directory

I am trying to compile a project using cmake. This is the relevant CMakeLists.txt code snippet.
set(PYTHON3_NUMPY_INCLUDE_DIRS=/usr/local/lib/python3.6/dist-packages/numpy/core/include/)
set(PYTHON_INCLUDE_DIRS ${PYTHON_INCLUDE_DIRS} /usr/local/lib/python3.6/dist-packages/numpy/core/include)
find_package(Python COMPONENTS Interpreter Development)
find_package(PythonLibs REQUIRED)
...
include_directories( ${PYTHON_INCLUDE_DIRS} )
To ensure that it is working properly, I print the relevant info
execute_process ( COMMAND ${PYTHON_EXECUTABLE} -c "import numpy; print(numpy.get_include())" OUTPUT_VARIABLE NUMPY_INCLUDE OUTPUT_STRIP_TRAILING_WHITESPACE)
message (STATUS "NUMPY_INCLUDE: " ${NUMPY_INCLUDE})
This is how I compile
cmake -DPYTHON_EXECUTABLE=/usr/bin/python3 .
Here is my output of cmake, the relevant bits.
-- Found Python: /usr/bin/python3.6 (found version "3.6.9") found components: Interpreter Development
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found version "3.6.9")
-- NUMPY_INCLUDE: /usr/local/lib/python3.6/dist-packages/numpy/core/include
So far everything is perfect and numpy is found.
However, on running make, I get the following
fatal error: numpy/ndarrayobject.h: No such file or directory
#include <numpy/ndarrayobject.h>
From what I can see I've covered all bases, yet I'm getting this error. What do I do?

OSError: exception: access violation reading with Cyphon basic example

I just followed the simple tutorial on Calling C/C++ from Python?, you can copy paste the code from the original answer over there, then I am just putting here a picture for illustration:
After creating these files, I built the project with:
User#User-PC$ g++ --version (from Cygwin)
g++ (GCC) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
User#User-PC$ g++ -c -fPIC foo.cpp -o foo.o
User#User-PC$ g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o
user#user-PC$ ll
total 162
drwx---r-x+ 1 user None 0 2019-05-15 06:59:29.850994400 -0300 ./
-rwx---r-x+ 1 user None 235 2019-05-15 06:59:02.791270500 -0300 foo.cpp*
-rw-r--r--+ 1 user None 3362 2019-05-15 06:58:54.335017300 -0300 foo.o
-rwx---r-x+ 1 user None 260 2019-05-15 06:59:40.159723600 -0300 foowrapper.py*
-rwxr-xr-x+ 1 user None 137902 2019-05-15 06:59:04.920040300 -0300 libfoo.so*
User#User-PC$ python3 foowrapper.py
Traceback (most recent call last):
File "foowrapper.py", line 12, in <module>
f.bar() #and you will see "Hello" on the screen
File "foowrapper.py", line 9, in bar
lib.Foo_bar(self.obj)
OSError: exception: access violation reading 0x000000636F6C6C61
User#User-PC$ python3 --version (from Windows) https://www.anaconda.com/distribution/
Python 3.7.2
User#User-PC$ python2 --version (from Cygwin)
Python 2.7.16
User#User-PCpython2 foowrapper.py
Hello
User#User-PC$
I am building it with Cygwin g++ and we can see that if I run it with Windows Native Python, it throws the exception OSError: exception: access violation reading, but if I use a Cygwin Python, it works just fine.
Why the Windows Python is throwing such exception? Can it be fixed, so I can build it with Cygwin g++ and run it with Windows Native Python compiler?

Most likely this is because your code compiled in Cygwin depends upon the Cygwin DLL (cygwin1.dll). You can still run that under Windows as long as that DLL is in your path, but there may be interesting effects due to how Cygwin treats paths. If you want to compile an application or dll that does not depend on the Cygwin DLL, you'll need to use the MinGW compilers in Cygwin. You probably don't have those installed. Re-run setup and look for packages named like the following: mingw-x86[_64]-gcc-*. You'll still create your DLL in a similar fashion, but you'll use the mingw version of gcc.

PyCaffe build fails, lboost_python not found

I am on MacOS 10.13.3 (17D47) with Python 2.7.14. I am in the process of building caffe w/ python. The project is CPU only. I am able to build caffe through make run test however, after attempting make pycaffe I get an error. This is the result:
touch python/caffe/proto/__init__.py
CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp
PROTOC (python) src/caffe/proto/caffe.proto
In file included from python/caffe/_caffe.cpp:17:
In file included from ./include/caffe/caffe.hpp:12:
./include/caffe/net.hpp:41:5: warning: unused typedef 'INVALID_REQUESTED_LOG_SEVERITY' [-Wunused-local-typedef]
LOG_EVERY_N(WARNING, 1000) << "DEPRECATED: ForwardPrefilled() "
^
/usr/local/include/glog/logging.h:943:30: note: expanded from macro 'LOG_EVERY_N'
INVALID_REQUESTED_LOG_SEVERITY); \
^
1 warning generated.
ld: library not found for -lboost_python
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [python/caffe/_caffe.so] Error 1
I have a Makefile.config that calls out boost-python from the brew install set as:
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/local/Cellar/boost-python/1.67.0/lib
My two initial questions are:
1)Am I not allowed to link to the boost library using the system python? 2)What is the correct way to tell make to grab boost-python?

I was actually having the same problem and couldn't find any answers for days. However, I found this question: Build caffe with Python ( cannot find -lboost_python3 ) .
That question is for python3, but I'm using python2.7 . Basically, I went into the Makefile (not Makefile.config), searched for boost_python, and changed it to boost_python27 . I did the following steps:
vim Makefile
use vim command :?boost_python (there should only be 1 occurrence)
changed PYTHON_LIBRARIES ?= boost_python python2.7
to PYTHON_LIBRARIES ?= boost_python27 python2.7
ran the command sudo make pycaffe
I hope this helps you!

Goto Makefile (and not the .config file) and change the name of the ld library to python27.

How to remove a compiler option that python disutils includes (language code=pic32 not recognized)

I'm trying to compile the msgpack-python python module with gcc (v4.7) on solaris 10. The python installed is 2.6.8. Distutils is automatically picking up a incorrect compiler option (-xcode=pic32) that I want to remove from command.
The full command that distutils is putting together is:
/opt/csw/bin/gcc-4.7 -DNDEBUG -O -O2 -pipe -mcpu=v9 -I/opt/csw/include -xcode=pic32 -I/opt/csw/include/python2.6 -c msgpack/_msgpack.c -o build/temp.solaris-2.10-sun4v-2.6/msgpack/_msgpack.o
but produces this error:
gcc-4.7: error: language code=pic32 not recognized
then fails. If I remove that -xcode=pic32 option and manually execute the above command the module compiles successfully.
I need to be able to do this in an automated fashion though (using a buildfarm to produce the packages). The question is, Without modifying or changing the current python or distutils, is there a way to "remove" this option that distutils is picking up, so I can have the python setup.py process build the module appropriately (i.e. without the pic32 option)?
Thanks

Do not compile with that gcc. -xcode=pic32 is Sun Studio complier command line parameter. It will lead to linking problems too, even if you compile OK. Compile with SUN CoolTools gcc which can understand such parameter, or use Oracle Solaris Studio for SPARC.
Some hints:
GCC produce very slow code for SPARC, that's for why SUN created Cool Tools.
You haven't to remove -xcode=pic32, but change for -m32 -fpic, when you insist on gcc-4.7
To get mature setup of OSS tool I'm using pkgsrc compiling with Studio Express to particular CPU (-xtarget=native)

You may also find luck by setting the follow env vars:
export CC=$gcc_dir_path # Example: /usr/bin/gcc
export CXX=$gxx_dir_path # Example: /usr/bin/g++
export CFLAGS=''
export CPPFLAGS=''
export CXXFLAGS=''
export LDFLAGS=''
Note: There is a difference between unset env var, and set-as-empty env var. I had build bugs with Python packages when my *FLAGS env vars were unset. (Calling gcc with option -xO2 was the cause.) Setting as empty did the trick.