Develop Reference

Onnx installation issue on linux

I tried to install onnx on my linux machine and encountered issue when I use python3 -m pip install onnxas
/tmp/pip-install-7ajubmjo/onnx_08cda1ef5f1d460ab7b455210cf3083e/onnx/defs/sequence/defs.cc: In function ‘bool onnx::BuildSequenceMapBodyFunc(const onnx::FunctionBodyBuildContext&, const onnx::OpSchema&, onnx::FunctionProto&)’: /tmp/pip-install-7ajubmjo/onnx_08cda1ef5f1d460ab7b455210cf3083e/onnx/defs/sequence/defs.cc:675:40: error: no match for ‘operator[]’ (operand types are ‘google::protobuf::RepeatedPtrField<onnx::ValueInfoProto>’ and ‘int’)
seq_at_node.add_output(g_inputs[inputIndex].name());
[ 95%] Building CXX object CMakeFiles/onnx.dir/onnx/version_converter/helper.cc.o
CMakeFiles/onnx.dir/build.make:830: recipe for target 'CMakeFiles/onnx.dir/onnx/defs/sequence/defs.cc.o' failed
The protobuf was installed via sudo apt-get install protobuf-compiler libprotobuf-dev and my environment is:
- python=3.6
- ubuntu=18.04
Another related issue seems that the compilation cannot not find pybind11 as:
Generated: /tmp/pip-install-7ajubmjo/onnx_08cda1ef5f1d460ab7b455210cf3083e/.setuptools-cmake-build/onnx/onnx-ml.proto
Generated: /tmp/pip-install-7ajubmjo/onnx_08cda1ef5f1d460ab7b455210cf3083e/.setuptools-cmake-build/onnx/onnx-operators-ml.proto
Generated: /tmp/pip-install-7ajubmjo/onnx_08cda1ef5f1d460ab7b455210cf3083e/.setuptools-cmake-build/onnx/onnx-data.proto
-- Could NOT find pybind11 (missing: pybind11_DIR)
-- pybind11 v2.9.1
--
-- ******** Summary ********
-- CMake version : 3.10.2
-- CMake command : /usr/bin/cmake
-- System : Linux
-- C++ compiler : /usr/bin/c++
-- C++ compiler version : 7.5.0
-- CXX flags : -Wnon-virtual-dtor
even though it's already installed via pip install pybind11.
Any thoughts on how to move forward with this issue ?

Upgrade you protobuf version seems the issue is with default Ubuntu 18.04 repo protobuf package.
simmilar issue was reported on opencv forum.
github issue
Installation of protobuf 3.5.1 fixes this, maybe error in the original
install process. I installed protobuf 3.5.1 from this release
Install steps
./autogen.sh
./configure
make
make install

Python delocate-wheel breaks wheel on MacOS

I'm generating wheel files to support a python distribution on MacOS using delocate. The wheels rely on libraries installed when gcc is installed using Homebrew.
brew install -v gcc#10
/usr/local/Cellar/gcc/10.2.0_3/lib/gcc/10/libstdc++.6.dylib
/usr/local/lib/gcc/10/libgcc_s.1.dylib
When running delocate-wheel with verbose option, the above are the two libraries included in the wheel.
If I use pip to install the original wheel, I can run the program without issue.
After running delicate-wheel and installing the new de-located wheel and trying to run, I produce this error:
python(66449,0x105ea3dc0) malloc: *** error for object 0x1071c3b60: pointer being freed was not allocated
python(66449,0x105ea3dc0) malloc: *** set a breakpoint in malloc_error_break to debug
I have tried using the require-archs=intel, but this errors out, and it appears that I only have single architecture files: Non-fat file: /usr/bin/python is architecture: x86_64
thank you,

Compiling GTK+3 by Using Meson

GTK+3 v3.24 is downloaded from Gitlab and extracted from zip file.
After running Meson, it gives the following error when meson setup --prefix C:\Users\user\Desktop\gtk\gtk-gtk-3-24 builddir command is run:
meson.build:359:0: ERROR: Git program not found.
No Git programs are installed on my machine (Windows 10 x64). But source code is downloaded. Meson and Ninja are installed by using pip command.
What is the problem?

The problem that in addition to GTK sources you need to provide all required dependencies: some dependency is not found -> meson tries to use "fallback", i.e. download some subproject from git, probably this is glib dependency:
glib_dep = dependency('glib-2.0', version: glib_req,
fallback : ['glib', 'libglib_dep'])
Here you can find the list of some of required packages:
You will also need various dependencies, based on the platform you are
building for:
GLib
GdkPixbuf
GObject-Introspection
...
But instead of painful process of setting up all these, why not just trying to use tool that is recommended to setup GTK on Windows? Check gvsbuild project or MSYS2, both described here.

Cmake is not able to find Python-libraries

Getting this error:
sudo: unable to resolve host coderw#ll
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108
(message):
Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE)
Call Stack (most recent call first):
/usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-2.8/Modules/FindPythonInterp.cmake:139
(FIND_PACKAGE_HANDLE_STANDARD_ARGS)
Code/cmake/Modules/FindNumPy.cmake:10 (find_package)
CMakeLists.txt:114 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/coderwall/Desktop/rdkit/build/CMakeFiles/CMakeOutput.log".
See also "/home/coderwall/Desktop/rdkit/build/CMakeFiles/CMakeError.log".
I have already installed:
sudo apt-get install python-dev
Environment variable are already set as follow:
PYTHON_INCLUDE_DIRS=/usr/include/python2.7
PYTHON_LIBRARIES=/usr/lib/python2.7/config/libpython2.7.so
Location of python.h : /usr/lib/include/python2.7/python.h
Location of python libs: /usr/lib/python2.7/
How to solve this?

You can fix the errors by appending to the cmake command the -DPYTHON_LIBRARY and -DPYTHON_INCLUDE_DIR flags filled with the respective folders.
Thus, the trick is to fill those parameters with the returned information from the python interpreter, which is the most reliable. This may work independently of your python location/version (also for Anaconda users):
$ cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "import sysconfig; print(sysconfig.get_path('include'))") \
-DPYTHON_LIBRARY=$(python -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR'))")
If the version of python that you want to link against cmake is Python3.X and the default python symlink points to Python2.X, python3 -c ... can be used instead of python -c ....
In case that the error persists, you may need to update the cmake to a higher version as stated by #pdpcosta and repeat the process again.

For me this is helpful:
# if using python2
apt-get install python-dev
# if using python3
apt-get install python3-dev

I hit the same issue,and discovered the error message gives misleading variable names. Try setting the following (singular instead of plural):
PYTHON_INCLUDE_DIR=/usr/include/python2.7
PYTHON_LIBRARY=/usr/lib/python2.7/config/libpython2.7.so
The (plural) variables you see error messages about are values that the PythonLibs sets up when it is initialised correctly.

Even after adding -DPYTHON_INCLUDE_DIR and -DPYTHON_LIBRARY as suggested above, I was still facing the error Could NOT find PythonInterp. What solved it was adding -DPYTHON_EXECUTABLE:FILEPATH= to cmake as suggested in https://github.com/pybind/pybind11/issues/99#issuecomment-182071479:
cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
-DPYTHON_LIBRARY=$(python -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))") \
-DPYTHON_EXECUTABLE:FILEPATH=`which python`

I was facing this problem while trying to compile OpenCV 3 on a Xubuntu 14.04 Thrusty Tahr system.
With all the dev packages of Python installed, the configuration process was always returning the message:
Could NOT found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.6", minimum required is "2.7")
Could NOT find PythonLibs (missing: PYTHON_INCLUDE_DIRS) (found suitable exact version "2.7.6")
Found PythonInterp: /usr/bin/python3.4 (found suitable version "3.4", minimum required is "3.4")
Could NOT find PythonLibs (missing: PYTHON_LIBRARIES) (Required is exact version "3.4.0")
The CMake version available on Thrusty Tahr repositories is 2.8.
Some posts inspired me to upgrade CMake.
I've added a PPA CMake repository which installs CMake version 3.2.
After the upgrade everything ran smoothly and the compilation was successful.

Some last version of Ubuntu installs Python 3.4 by default and the CMake version from Ubuntu (2.8) only searches up to Python 3.3.
Try to add set(Python_ADDITIONAL_VERSIONS 3.4) before the find_package statement.
Remember to clean CMakeCache.txt too.

This problem can also happen in Windows. Cmake looks into the registry and sometimes python values are not set. For those with similar problem:
http://ericsilva.org/2012/10/11/restoring-your-python-registry-in-windows/
Just create a .reg file to set the necessary keys and edit accordingly to match your setup.
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Python]
[HKEY_CURRENT_USER\Software\Python\Pythoncore]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6\InstallPath]
#="C:\\python26"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6\PythonPath]
#="C:\\python26;C:\\python26\\Lib\\;C:\\python26\\DLLs\\"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7\InstallPath]
#="C:\\python27"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7\PythonPath]
#="C:\\python27;C:\\python27\\Lib\\;C:\\python27\\DLLs\\"

Note that if you are using cMake version 3.12 or later, variable PythonInterp and PythonLibs has been changed into Python.
So we use:
find_package(Python ${PY_VERSION} REQUIRED)
instead of:
find_package(PythonInterp ${PY_VERSION} REQUIRED)
find_package(PythonLibs ${PY_VERSION} REQUIRED)
see https://cmake.org/cmake/help/v3.12/module/FindPython.html for details.

I had upgraded to python3.8 on my system and had an incomplete install. Managed to fix it by installing the rest of the 3.8 packages:
sudo apt-get install python3.8 python3.8-dev python3.8-distutils python3.8-venv

Paste this into your CMakeLists.txt:
# find python
execute_process(COMMAND python-config --prefix OUTPUT_VARIABLE PYTHON_SEARCH_PATH)
string(REGEX REPLACE "\n$" "" PYTHON_SEARCH_PATH "${PYTHON_SEARCH_PATH}")
file(GLOB_RECURSE PYTHON_DY_LIBS ${PYTHON_SEARCH_PATH}/lib/libpython*.dylib ${PYTHON_SEARCH_PATH}/lib/libpython*.so)
if (PYTHON_DY_LIBS)
list(GET PYTHON_DY_LIBS 0 PYTHON_LIBRARY)
message("-- Find shared libpython: ${PYTHON_LIBRARY}")
else()
message(WARNING "Cannot find shared libpython, try find_package")
endif()
find_package(PythonInterp)
find_package(PythonLibs ${PYTHON_VERSION_STRING} EXACT)

In case that might help, I found a workaround for a similar problem, looking at the cmake doc :
https://cmake.org/cmake/help/v3.0/module/FindPythonLibs.html
You must set two env vars for cmake to find coherent versions.
Unfortunately this is not a generic solution...
cmake -DPYTHON_LIBRARY=${HOME}/.pyenv/versions/3.8.0/lib/libpython3.8.a -DPYTHON_INCLUDE_DIR=${HOME}/.pyenv/versions/3.8.0/include/python3.8/ cern_root/

In Python 3.2 and onward distutils.sysconfig is deprecated in favor of sysconfig.
To get all the variable names in data structure and inspect the situation we can use get_paths function
import sysconfig
sysconfig.get_paths()
which will return us a dict with all the relevant variable names as keys, and corresponding paths as values.
When we know the key we can get the value dynamically,
>>> sysconfig.get_path("include")
'C:\\Program Files\\Python310\\Include'
However, the most convenient feature of sysutils for this situation is that it can list all the variables at once from the command-prompt by invoking python -m sysconfig:
$ python -m sysconfig
Platform: "macosx-10.4-i386"
Python version: "3.2"
Current installation scheme: "posix_prefix"
Paths:
data = "/usr/local"
include = "/Users/tarek/Dev/svn.python.org/py3k/Include"
platinclude = "."
platlib = "/usr/local/lib/python3.2/site-packages"
platstdlib = "/usr/local/lib/python3.2"
purelib = "/usr/local/lib/python3.2/site-packages"
scripts = "/usr/local/bin"
stdlib = "/usr/local/lib/python3.2"
Variables:
AC_APPLE_UNIVERSAL_BUILD = "0"
AIX_GENUINE_CPLUSPLUS = "0"
AR = "ar"
ARFLAGS = "rc"
...

Maybe below command line works for you if all of above methods didn't work.
sudo apt-get install cython cython3

On Ubuntu 20 at least, since Python 2.7 is deprecated, you can:
sudo apt install python2.7-dev
And then PYTHON_INCLUDE_DIR can be set to /usr/include/python2.7

How to deal with Linux/Python dependencies?

Due to lack of support for some libraries I want to use, I moved some Python development from Windows to Linux development. I've spent most of the day messing about getting nowhere with dependencies.
The question
Whenever I pick up Linux, I usually run into some kind of dependency issue, usually with development libraries, whether they're installed via apt-get, easy_install or pip. I can waste days on what should be simple tasks, spending longer on getting libraries to work than writing code. Where can I learn about strategy for dealing with these kind of issues rather than aimlessly googling for someone who's come across the same problem before?
An example
Just one example: I wanted to generate some QR codes. So, I thought I'd use github.com/bitly/pyqrencode which is based on pyqrcode.sourceforge.net but supposedly without the Java dependencies. There are others (pyqrnative, github.com/Arachnid/pyqrencode) but that one seemed like the best bet for my needs.
So, I found the package on pypi and thought using that would make life easier:
(I've perhaps made life more difficult for myself by using virtualenv to keep things neat and tidy.)
(myenv3)mat#ubuntu:~/myenv3$ bin/pip install pyqrencode
Downloading/unpacking pyqrencode
Downloading pyqrencode-0.2.tar.gz
Running setup.py egg_info for package pyqrencode
Installing collected packages: pyqrencode
Running setup.py install for pyqrencode
building 'qrencode' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c qrencode.c -o build/temp.linux-i686-2.7/qrencode.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions build/temp.linux-i686-2.7/qrencode.o -lqrencode -o build/lib.linux-i686-2.7/qrencode.so
Successfully installed pyqrencode
Cleaning up...
(I guess I probably sudo apt-get install libqrencode-dev at some point prior to that too.)
So then I tried to run the test script:
(myenv3)mat#ubuntu:~/myenv3$ python test_qr.py
Traceback (most recent call last):
File "test_qr.py", line 1, in <module>
from qrencode import Encoder
File "qrencode.pyx", line 1, in init qrencode (qrencode.c:1520)
ImportError: No module named ImageOps
:(
Well, investigations revealed that ImageOps appears to be part of PIL...
(myenv3)mat#ubuntu:~/myenv3$ pip install pil
Downloading/unpacking pil
Downloading PIL-1.1.7.tar.gz (506Kb): 122Kb downloaded
Operation cancelled by user
Storing complete log in /home/mat/.pip/pip.log
(myenv3)mat#ubuntu:~/myenv3$ bin/pip install pil
Downloading/unpacking pil
Downloading PIL-1.1.7.tar.gz (506Kb): 506Kb downloaded
Running setup.py egg_info for package pil
WARNING: '' not a valid package name; please use only.-separated package names in setup.py
Installing collected packages: pil
Running setup.py install for pil
WARNING: '' not a valid package name; please use only.-separated package names in setup.py
building '_imaging' extension
gcc ...
building '_imagingmath' extension
gcc ...
--------------------------------------------------------------------
PIL 1.1.7 SETUP SUMMARY
--------------------------------------------------------------------
version 1.1.7
platform linux2 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2]
--------------------------------------------------------------------
*** TKINTER support not available
*** JPEG support not available
*** ZLIB (PNG/ZIP) support not available
*** FREETYPE2 support not available
*** LITTLECMS support not available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.
To check the build, run the selftest.py script.
...
Successfully installed pil
Cleaning up...
Hmm, PIL's installed but hasn't picked up the libraries I installed with sudo apt-get install libjpeg62 libjpeg62-dev libpng12-dev zlib1g zlib1g-dev earlier. I'm not sure how to tell pip to feed the library locations to setup.py. Googling suggests a variety of ideas which I've tried, but none of them seem to help much other than to send me round in circles.
Ubuntu 11.04: Installing PIL into a virtualenv with PIP suggests using the pillow package instead, so let's try that:
(myenv3)mat#ubuntu:~/myenv3$ pip install pillow
Downloading/unpacking pillow
Downloading Pillow-1.7.5.zip (637Kb): 637Kb downloaded
Running setup.py egg_info for package pillow
...
Installing collected packages: pillow
Running setup.py install for pillow
building '_imaging' extension
gcc ...
--------------------------------------------------------------------
SETUP SUMMARY (Pillow 1.7.5 / PIL 1.1.7)
--------------------------------------------------------------------
version 1.7.5
platform linux2 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2]
--------------------------------------------------------------------
*** TKINTER support not available
--- JPEG support available
--- ZLIB (PNG/ZIP) support available
--- FREETYPE2 support available
*** LITTLECMS support not available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.
To check the build, run the selftest.py script.
...
Successfully installed pillow
Cleaning up...
Well, we seem to have the JPEG and PNG support this time, yay!
(myenv3)mat#ubuntu:~/myenv3$ python test_qr.py
Traceback (most recent call last):
File "test_qr.py", line 1, in <module>
from qrencode import Encoder
File "qrencode.pyx", line 1, in init qrencode (qrencode.c:1520)
ImportError: No module named ImageOps
Still no ImageOps though. Now I'm stumped, is ImageOps missing from pillow, or is it a different problem that was also there with pil.

I see two separate problems here:
Keeping track of all the python modules you need for your project.
Keeping track of all the dynamic libraries you need for the python modules in your project.
For the first problem, I have found that buildout is good help, althought it takes a litle while to grasp.
In your case, I would start by creating a directory for my new project. I would then go into that directory and download bootstrap.py
wget http://python-distribute.org/bootstrap.py
I would then create a buildout.cfg file:
[buildout]
parts = qrproject
python
eggs = pyqrencode
[qrproject]
recipe = z3c.recipe.scripts
eggs = ${buildout:eggs}
entry-points= qrproject=qrprojectmodule:run
extra-paths = ${buildout:directory}
# This is a simple way of creating an interpreter that will have
# access to all the eggs / modules that this project uses.
[python]
recipe = z3c.recipe.scripts
interpreter = python
eggs = ${buildout:eggs}
extra-paths = ${buildout:directory}
In this buildout.cfg I'm referencing the module qrprojectmodule (in entry-points under [qrproject]. This will create a bin/qrproject that runs the function run in the module qrprojectmodule. So I will also create the file qrprojectmodule.py
import qrencode
def run():
print "Entry point for qrproject. Happily imports qrencode module"
Now it's time to run bootstrap.py with the python binary you want to use:
python bootstrap.py
Then run the generated bin/buildout
bin/buildout
This will create two additional binaries in the bin/ directory - bin/qrproject and bin/python. The former is your project's main binary. It will be created automatically each time you run buildout and will have all the modules and eggs you want loaded. The second is simply a convenient way to get a python prompt where all your modules and eggs are loaded, for easy debugging. The fine thing here is that bin/buildout will automatically install any python eggs that the eggs (in your case pyqrencode) have specified as dependencies.
Actually, you will probably get a compilation error in the step where you run bin/buildout. This is because you need to address problem 2: All dynamic libraries being available on your system. On Linux, it's usually best to get help from your packaging system. I'm going to assume you're using a Debian derivate such as Ubuntu here.
The pyqrencode web site specifies that you need the libqrencode library for pyqrencode to work. So I used my package manager to search for that:
$ apt-cache search libqrencode
libqrencode-dev - QR Code encoding library -- development
libqrencode3 - QR Code encoding library
qrencode - QR Code encoder into PNG image
In this case, I want the -dev package, as that installs linkable libraries and header files required to compile python C-modules. Also, the dependency system in the package manager will make sure that if I install libqrencode-dev, I will also get libqrencode3, as that is required at runtime, i.e. after compilation of the module.
So, I install the package:
sudo apt-get install libqrencode-dev
Once that has completed, rerun bin/buildout and the pyqrencode module will (hopefully) compile and install successfully. Now try to run bin/qrproject
$ bin/qrproject
Entry point for qrproject. Happily imports qrencode module
Success! :-)
So, in summary:
Use buildout to automatically download and install all the python modules/eggs you need for your project.
Use your system's package manager to install any dynamic (C) libraries required by the python modules you use.
Be aware that in many cases there are already packaged versions of your python modules available in the package system. For example, pil is available by installing the python-imaging package on Ubuntu. In this case, you don't need to install it via buildout, and you don't need to worry about libraries being available - the package manager will install all dependencies required for the module to run. Doing it via buildout can however make it easier to distribute your project and make it run on other systems.

Your story reminds me of my early experiences with Linux, and why I love APT.
There is no universal solution to your general problem; the best you can do is to take advantage of the work or others. The Debian packagers do a great job of flagging the dependencies of packages, so apt-get will pull in what you need. So, my strategy is simply to avoid building and installing stuff on my own, and use apt-get wherever possible.
Note that Ubuntu is based on Debian and thus gains the benefit of the work of the Debian packagers. I haven't used Fedora but I hear that the packages are not as well-organized as the ones from Debian.