How to build MultiNEAT from sources? - python

I'm building MultiNEAT from source, on macOS Mojave, with Python3, and I'm hitting a snag when trying "python setup.py build_ext". I get a bunch of errors. Any help is greatly appreciated.
Here is a step by step of what I'm doing from scratch, if you want to follow along in your own terminal. I'm doing all of this so others can do it themselves too, as it has taken me a while to even get here. I've gotten bits and pieces of information here and there, but there is no straight up instructions on how to build this library.
# Change things accordingly for you.
# Define work dir. Should be empty at this point.
WDIR=/Users/luis/Documents/neat
cd $WDIR
# Setup Python virtual environment and requirements.
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install psutil numpy opencv-python
# Get Boost.
# Get the url from here: https://www.boost.org/users/download/
curl -L https://dl.bintray.com/boostorg/release/1.70.0/source/boost_1_70_0.tar.gz | tar -xz
cd boost_1_70_0/
# Get system Python include files with: python3-config --includes
# Put that path into this exported var.
export CPLUS_INCLUDE_PATH=/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m
# Build Boost! This will take a bit.
./bootstrap.sh --prefix=$WDIR/boost
./b2 install
# Get MultiNEAT.
cd $WDIR
git clone https://github.com/peter-ch/MultiNEAT.git
cd MultiNEAT/
# Setup build. (Is this incomplete?)
export MN_BUILD=boost
export PREFIX=$WDIR/boost
# Build MultiNEAT!
python setup.py build_ext
# Supposedly, I'd do "python setup.py install" after, but errors are happening now :(
This is only the top part of when errors start happening, but most are similar:
In file included from src/Innovation.cpp:34:
In file included from src/Innovation.h:37:
src/Genome.h:689:19: error: expected ':'
public void set_children()
^
:
src/Genome.h:691:49: error: indirection requires pointer operand ('std::__1::vector<double,
std::__1::allocator<double> >::size_type' (aka 'unsigned long') invalid)
for(unsigned int ix = 0; ix < 2**coord.size(); ix++){
^~~~~~~~~~~~~


Here is what works:
First install the dependencies you want to use:
Boost 1.49 and above with Boost.Python and Boost.Serialization (optional)
ProgressBar (Python package) (optional)
NumPy (Python package) (optional)
Matplotlib (Python package) (optional)
OpenCV 2.3 and above (with Python bindings) (optional)
Cython (if you want Python bindings)
Then clone this git(this one seems to be older but does not throw those errors you encountered)
And finally enter the source directory and run "sudo python setup.py install"
This worked even with Python 3.7.

Related

Cannot install nlopt python module

I am trying to install nlopt onto macOS 10.15.5. I downloaded the nlopt-2.6.2.tar.gz file from the NLopt docs and ran the following from the nlopt-2.6.2 directory:
mkdir build
cd build
cmake -DNLOPT_OCTAVE=Off -DNLOPT_MATLAB=Off -DNLOPT_GUILE=Off ..
make
sudo make install
I got the following output: cmake.txt.
The header file (nlopt.h) installs correctly to /usr/local/include and the dynamic library (libnlopt.dylib) installs correctly to /usr/local/lib/, but neither the dist-info file nor the nlopt module itself installs.
I have also tried installing via pip, brew, and conda, none of which have worked. I have also tried cloning from this Github, which didn't work either.
I appreciate any help with this, because I am completely lost. I am relatively new to this kind of stuff, and I couldn't find any good answers online.

The official docs are somewhat laconic about the exact steps required for building nlopt with Python bindings. First of all, you'll need SWIG installed:
$ brew install swig
Then, you'll need numpy to be available for the target Python interpreter. It is already preinstalled for the system Python, otherwise install it via Homebrew or pip, depending on your Python installation.
Now run cmake:
$ cmake -DNLOPT_GUILE=OFF -DNLOPT_MATLAB=OFF -DNLOPT_OCTAVE=OFF -DNLOPT_TESTS=OFF
This will build bindings against the default Python 2.7 installation preinstalled on MacOS. If you need to build against custom Python installation (e.g. when you've installed Python 3 via Homebrew or PKG installer from https://www.python.org/downloads), pass it via the PYTHON_EXECUTABLE arg:
$ cmake -DNLOPT_GUILE=OFF -DNLOPT_MATLAB=OFF -DNLOPT_OCTAVE=OFF -DNLOPT_TESTS=OFF -DPYTHON_EXECUTABLE=/usr/local/bin/python3
Inspect the log now - Python, SWIG and numpy headers should be successfully located. Example output snippet (you may have different paths/versions printed):
-- Found PythonInterp: /usr/local/bin/python3.8 (found version "3.8.3")
-- Found PythonLibs: /Library/Frameworks/Python.framework/Versions/3.8/lib/libpython3.8.dylib (found suitable exact version "3.8.3")
-- Found NumPy: /Users/hoefling/Library/Python/3.8/lib/python/site-packages/numpy/core/include (found version "1.19")
-- Found SWIG: /usr/local/bin/swig (found version "4.0.2")
If any of those conditions is not satisfied (e.g. you see Could NOT find NumPy, Could NOT find PythonLibs or Could NOT find SWIG), then stop and make sure the configuration succeeds before proceeding next.
Now compile:
$ make
...
Scanning dependencies of target nlopt_python_swig_compilation
[ 96%] Swig compile nlopt.i for python
[ 96%] Built target nlopt_python_swig_compilation
Scanning dependencies of target nlopt_python
[ 98%] Building CXX object src/swig/CMakeFiles/nlopt_python.dir/CMakeFiles/nlopt_python.dir/nloptPYTHON_wrap.cxx.o
[100%] Linking CXX shared module _nlopt.so
[100%] Built target nlopt_python
Install:
$ make install
...
-- Installing: /usr/local/lib/python3.8/site-packages/nlopt.py
-- Installing: /usr/local/lib/python3.8/site-packages/_nlopt.so
Test the Python bindings are importable:
$ python -c "import nlopt; print(nlopt.__version__)"
2.6.2

R-markdown: the python-logging package seems to cause the problem

I am facing a strange problem which I could track down to the python logging package. Let me shortly explain what I want to do: the goal is to create html reports with python. I am using the R rmarkdown package which runs python code trough reticulate using a local virtualenv.
The Problem:
As soon as I install the python logging package rmarkdown runs into a problem when loading matplotlib. I have written a small test script to reproduce the example.
My system:
Ubuntu 18.04 bionic
Python 2.7.15rc1
Test "1" script (without logging):
Create a new virtualenv (venv).
Use venv/bin/pip to install matplotlib.
Run reticulate::import (at the end via rmarkdown::render).
Test "2" script (with logging):
Create a new virtualenv (venv).
In addition to the first test: install logging via venv/bin/pip.
Use venv/bin/pip to install matplotlib.
Run reticulate::import (at the end via rmarkdown::render).
The modules installed (virtualenv):
backports.functools-lru-cache 1.5
cycler 0.10.0
kiwisolver 1.0.1
logging 0.4.9.6 <- only for "test 2"
matplotlib 2.2.3
numpy 1.15.1
pip 18.0
pkg-resources 0.0.0
pyparsing 2.2.0
python-dateutil 2.7.3
pytz 2018.5
setuptools 40.2.0
six 1.11.0
subprocess32 3.5.2
wheel 0.31.1
The system site packages do have the same module version.
Results:
All tests from test 1 (without logging) work nicely.
The tests from test 2 (with loging) fail when using the virtualenv. When calling rmarkdown::render (see below), when using the system python installation (not virtualenv) they work nice as well.
There seem to be something strange with reticulate when logging is installed in a virtualenenv.
The output of the test script (see below):
The full output including the error:
----------- no logging package installed ------------
Module(matplotlib)
Module(matplotlib)
--------- with logging package installed ------------
Error in py_module_import(module, convert = convert) :
AttributeError: 'module' object has no attribute 'NullHandler'
Detailed traceback:
File "/home/retos/Downloads/venvtest/venv/lib/python2.7/site-packages/matplotlib/__init__.py", line 168, in <module>
_log.addHandler(logging.NullHandler())
Calls: <Anonymous> -> py_module_import -> .Call
Execution halted
Module(matplotlib)
The Module(matplotlib) output is the success message of loading the module via reticulate::import. As one can see only the one test fails where the virtualenv is used with installed logging python module.
Anyone having an idea what could case these problems? I spent quite some time to identify the source of the error, but I am kind of lost now ...
Test script to reproduce the error:
Here is a small bash/shell script to reproduce my tests.
#!/bin/bash
# New virtual environment and install matplotlib
echo " ----------- no logging package installed ------------"
if [ -d venv ] ; then rm -rf venv ; fi
virtualenv venv &>/dev/null > /dev/null
venv/bin/pip install matplotlib > /dev/null
# Print installed packages
Rscript -e "reticulate::use_python('venv/bin/python'); reticulate::import('matplotlib')"
Rscript -e "reticulate::import('matplotlib')"
# New virtual environment and install logging and matplotlib
echo " --------- with logging package installed ------------"
if [ -d venv ] ; then rm -rf venv ; fi
virtualenv venv > /dev/null
venv/bin/pip install logging > /dev/null
venv/bin/pip install matplotlib > /dev/null
# Print installed packages
Rscript -e "reticulate::use_python('venv/bin/python'); reticulate::import('matplotlib')"
Rscript -e "reticulate::import('matplotlib')"
I first thought it is related to the problem "ImportError: cannot import name cbook" but the solution there did not work.
May thanks in advance!
R

Logging became a standard module included in Python library in version 2.3. You must not install it from PyPI. Remove it ASAP:
pip uninstall logging

Pip, pip3, and conflicting python distributions

I am having trouble installing the python packages PyTransit and ktransit to Anaconda 3 (on Linux). It seems like lots of others have had similar errors, but none of their solutions have worked for me. There's a bunch of stuff going on:
pip install ktransit gives:
/usr/bin/ld: /home/name/anaconda3/lib/libgfortran.a(pause.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/home/name/anaconda3/lib/libgfortran.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
I would try to use ./configure with --enable-shared, but there's no config file. I tried adding extra_compile_args = ["-fPIC"], extra_link_args=["-fPIC"] to setup.py (as in https://github.com/h5py/h5py/issues/851), but that did nothing.
pip install pytransit gives different errors:
#warning "Using deprecated NumPy API, disable it by " \
Warning: src/gimenez.f90:35: Illegal preprocessor directive
Warning: src/gimenez.f90:36: Illegal preprocessor directive
Warning: src/gimenez.f90:37: Illegal preprocessor directive
src/gimenez.f90:271.33:
real(8), dimension(CHUNK_SIZE, npol) :: d
1
Error: Symbol 'chunk_size' at (1) has no IMPLICIT type
Here's where it gets interesting. Using pip3 (or sudo pip) allows ktransit to install successfully, but does nothing for pytransit. Pip and pip3 seem to be tied to different places:
which pip
/home/name/anaconda3/bin/pip
which pip3
/usr/local/bin/pip3
which python
/home/name/anaconda3/bin/python
Lastly, When I try to import ktransit in a jupyter notebook through anaconda, it fails, but when I import ktransit in the Jupyter QtConsole, it succeeds, but seems to contain itself and not provide actual functionality. What I mean is that I can't reference anything within the module, and:
Help on package ktransit:
NAME
ktransit
PACKAGE CONTENTS
ktransit (package)
setup
FILE
(built-in)
I'm thinking that whatever version of python is associated with /usr/bin is somehow messing with me...or maybe there's something wrong with my gcc or libgfortran? I have gcc versions 4.8.5 and 6 (gcc --version gives 4.8.5, but -update-alternatives says there's only one version, 6).
Thanks for taking the time to read this far. I would be very grateful for any suggestions or ideas!

Figured it out -- I uninstalled Anaconda, made sure my python was pointing in the right direction, and then used sudo python -m pip install. Thank you to Nagev for pushing me to get rid of Anaconda!

How to install libgpuarray and clBLAS in python on mac os x (10.9)?

I'm trying to install libgpuarray (http://deeplearning.net/software/libgpuarray/installation.html) for python on OSX 10.9. I installed CMake 3.0 and are using the UI. I downloaded the libgpuarray and clBLAS source from git and unpacked under Users/user/libgpuarray .. /clBLAS respectively.
Using CMake UI I get the following warning when generating the build:
MACOSX_RPATH is not specified for the following targets:
gpuarray (clBLAS respectively)
Executing 'python setup.py build' results in the following error:
/Users/i043337/Library/Python/2.7/lib/python/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning:
"Using deprecated NumPy API, disable it by " "#defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
'#warning "Using deprecated NumPy API, disable it by " \
^
pygpu/gpuarray.c:354:10: fatal error: 'gpuarray/types.h' file not found
'#include "gpuarray/types.h"
Tried to figure out how to set the correct path but I'm lost to be honest.
Any help is more than appreciated!
Latest XCode (incl. DeveloperTools) installed. Python version is 2.7 from python.org and under the following paths (Terminal$ type -a python)
python is /Library/Frameworks/Python.framework/Versions/2.7/bin/python
python is /usr/bin/python
python is /usr/local/bin/python

I solved this by running:
python setup.py build_ext -I /usr/local/include -L /usr/local/lib
instead of:
python setup.py build
and that solved it. You could probably add these to your PATH as an alternative.
Also, for anyone googling this in the future, you may have to modify the link.txt in Build/src/CMakeFiles/gpuarray.dir to:
1) omit -arch=i386
2) add -F/Library/Frameworks
after running make in the Build/ directory it has you create.

I think you forgot to run
make
make install
after cmake.

I installed it by following commands:
mkdir build && cd build
cmake ..
make && make install
cd .. && python setup.py install

Cmake is not able to find Python-libraries

Getting this error:
sudo: unable to resolve host coderw#ll
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108
(message):
Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE)
Call Stack (most recent call first):
/usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-2.8/Modules/FindPythonInterp.cmake:139
(FIND_PACKAGE_HANDLE_STANDARD_ARGS)
Code/cmake/Modules/FindNumPy.cmake:10 (find_package)
CMakeLists.txt:114 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/coderwall/Desktop/rdkit/build/CMakeFiles/CMakeOutput.log".
See also "/home/coderwall/Desktop/rdkit/build/CMakeFiles/CMakeError.log".
I have already installed:
sudo apt-get install python-dev
Environment variable are already set as follow:
PYTHON_INCLUDE_DIRS=/usr/include/python2.7
PYTHON_LIBRARIES=/usr/lib/python2.7/config/libpython2.7.so
Location of python.h : /usr/lib/include/python2.7/python.h
Location of python libs: /usr/lib/python2.7/
How to solve this?

You can fix the errors by appending to the cmake command the -DPYTHON_LIBRARY and -DPYTHON_INCLUDE_DIR flags filled with the respective folders.
Thus, the trick is to fill those parameters with the returned information from the python interpreter, which is the most reliable. This may work independently of your python location/version (also for Anaconda users):
$ cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "import sysconfig; print(sysconfig.get_path('include'))") \
-DPYTHON_LIBRARY=$(python -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR'))")
If the version of python that you want to link against cmake is Python3.X and the default python symlink points to Python2.X, python3 -c ... can be used instead of python -c ....
In case that the error persists, you may need to update the cmake to a higher version as stated by #pdpcosta and repeat the process again.

For me this is helpful:
# if using python2
apt-get install python-dev
# if using python3
apt-get install python3-dev

I hit the same issue,and discovered the error message gives misleading variable names. Try setting the following (singular instead of plural):
PYTHON_INCLUDE_DIR=/usr/include/python2.7
PYTHON_LIBRARY=/usr/lib/python2.7/config/libpython2.7.so
The (plural) variables you see error messages about are values that the PythonLibs sets up when it is initialised correctly.

Even after adding -DPYTHON_INCLUDE_DIR and -DPYTHON_LIBRARY as suggested above, I was still facing the error Could NOT find PythonInterp. What solved it was adding -DPYTHON_EXECUTABLE:FILEPATH= to cmake as suggested in https://github.com/pybind/pybind11/issues/99#issuecomment-182071479:
cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
-DPYTHON_LIBRARY=$(python -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))") \
-DPYTHON_EXECUTABLE:FILEPATH=`which python`

I was facing this problem while trying to compile OpenCV 3 on a Xubuntu 14.04 Thrusty Tahr system.
With all the dev packages of Python installed, the configuration process was always returning the message:
Could NOT found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.6", minimum required is "2.7")
Could NOT find PythonLibs (missing: PYTHON_INCLUDE_DIRS) (found suitable exact version "2.7.6")
Found PythonInterp: /usr/bin/python3.4 (found suitable version "3.4", minimum required is "3.4")
Could NOT find PythonLibs (missing: PYTHON_LIBRARIES) (Required is exact version "3.4.0")
The CMake version available on Thrusty Tahr repositories is 2.8.
Some posts inspired me to upgrade CMake.
I've added a PPA CMake repository which installs CMake version 3.2.
After the upgrade everything ran smoothly and the compilation was successful.

Some last version of Ubuntu installs Python 3.4 by default and the CMake version from Ubuntu (2.8) only searches up to Python 3.3.
Try to add set(Python_ADDITIONAL_VERSIONS 3.4) before the find_package statement.
Remember to clean CMakeCache.txt too.

This problem can also happen in Windows. Cmake looks into the registry and sometimes python values are not set. For those with similar problem:
http://ericsilva.org/2012/10/11/restoring-your-python-registry-in-windows/
Just create a .reg file to set the necessary keys and edit accordingly to match your setup.
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Python]
[HKEY_CURRENT_USER\Software\Python\Pythoncore]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6\InstallPath]
#="C:\\python26"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6\PythonPath]
#="C:\\python26;C:\\python26\\Lib\\;C:\\python26\\DLLs\\"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7\InstallPath]
#="C:\\python27"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7\PythonPath]
#="C:\\python27;C:\\python27\\Lib\\;C:\\python27\\DLLs\\"

Note that if you are using cMake version 3.12 or later, variable PythonInterp and PythonLibs has been changed into Python.
So we use:
find_package(Python ${PY_VERSION} REQUIRED)
instead of:
find_package(PythonInterp ${PY_VERSION} REQUIRED)
find_package(PythonLibs ${PY_VERSION} REQUIRED)
see https://cmake.org/cmake/help/v3.12/module/FindPython.html for details.

I had upgraded to python3.8 on my system and had an incomplete install. Managed to fix it by installing the rest of the 3.8 packages:
sudo apt-get install python3.8 python3.8-dev python3.8-distutils python3.8-venv

Paste this into your CMakeLists.txt:
# find python
execute_process(COMMAND python-config --prefix OUTPUT_VARIABLE PYTHON_SEARCH_PATH)
string(REGEX REPLACE "\n$" "" PYTHON_SEARCH_PATH "${PYTHON_SEARCH_PATH}")
file(GLOB_RECURSE PYTHON_DY_LIBS ${PYTHON_SEARCH_PATH}/lib/libpython*.dylib ${PYTHON_SEARCH_PATH}/lib/libpython*.so)
if (PYTHON_DY_LIBS)
list(GET PYTHON_DY_LIBS 0 PYTHON_LIBRARY)
message("-- Find shared libpython: ${PYTHON_LIBRARY}")
else()
message(WARNING "Cannot find shared libpython, try find_package")
endif()
find_package(PythonInterp)
find_package(PythonLibs ${PYTHON_VERSION_STRING} EXACT)

In case that might help, I found a workaround for a similar problem, looking at the cmake doc :
https://cmake.org/cmake/help/v3.0/module/FindPythonLibs.html
You must set two env vars for cmake to find coherent versions.
Unfortunately this is not a generic solution...
cmake -DPYTHON_LIBRARY=${HOME}/.pyenv/versions/3.8.0/lib/libpython3.8.a -DPYTHON_INCLUDE_DIR=${HOME}/.pyenv/versions/3.8.0/include/python3.8/ cern_root/

In Python 3.2 and onward distutils.sysconfig is deprecated in favor of sysconfig.
To get all the variable names in data structure and inspect the situation we can use get_paths function
import sysconfig
sysconfig.get_paths()
which will return us a dict with all the relevant variable names as keys, and corresponding paths as values.
When we know the key we can get the value dynamically,
>>> sysconfig.get_path("include")
'C:\\Program Files\\Python310\\Include'
However, the most convenient feature of sysutils for this situation is that it can list all the variables at once from the command-prompt by invoking python -m sysconfig:
$ python -m sysconfig
Platform: "macosx-10.4-i386"
Python version: "3.2"
Current installation scheme: "posix_prefix"
Paths:
data = "/usr/local"
include = "/Users/tarek/Dev/svn.python.org/py3k/Include"
platinclude = "."
platlib = "/usr/local/lib/python3.2/site-packages"
platstdlib = "/usr/local/lib/python3.2"
purelib = "/usr/local/lib/python3.2/site-packages"
scripts = "/usr/local/bin"
stdlib = "/usr/local/lib/python3.2"
Variables:
AC_APPLE_UNIVERSAL_BUILD = "0"
AIX_GENUINE_CPLUSPLUS = "0"
AR = "ar"
ARFLAGS = "rc"
...

Maybe below command line works for you if all of above methods didn't work.
sudo apt-get install cython cython3

On Ubuntu 20 at least, since Python 2.7 is deprecated, you can:
sudo apt install python2.7-dev
And then PYTHON_INCLUDE_DIR can be set to /usr/include/python2.7

Categories

Resources