Develop Reference

How to debug python wheel build on macos remote server (azure pipelines)

I try to build wheel for imagecodecs python package. Previous version I build without any problem.
For current revision I got wheels for linux (manylinux2010) and macos, but without openmp support.
I would like to build also with openmp support. For this purpose I use gcc from brew.
From previous version of imagecodecs to dependency there is added snappy, zopflipy and changed version of jxrlib.
I use cibuildweel to build wheel so it is build with python installed from official pkg files.
Build pass, then when test start it fail with (on azure pipelines)
python(53258,0x7fffa20fb380) malloc: *** error for object 0x109c85c00: pointer being freed was not allocated
When I download it on my computer whole test set pass.
On azure pipelines there is no option to login on machine. I try to debug it with
lldb --batch -o "br set -n malloc_error_break" -o "run" -o "bt" -o "c" -- python -m pytest {project}/tests/
Then I got:
+ lldb --batch -o "br set -n malloc_error_break" -o "run" -o "bt" -o "c" -- python -m pytest /Users/runner/runners/2.163.1/work/1/s/tests/
(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) settings set -- target.run-args "-m" "pytest" "/Users/runner/runners/2.163.1/work/1/s/tests/"
(lldb) br set -n malloc_error_break
Breakpoint 1: where = libsystem_malloc.dylib`malloc_error_break, address = 0x00000000000119e6
(lldb) run
============================= test session starts ==============================
platform darwin -- Python 2.7.17, pytest-4.6.8, py-1.8.0, pluggy-0.13.1
python(53294,0x7fff97d45380) malloc: *** error for object 0x10fc3ec00: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Process 53294 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x00007fff5f7d89e6 libsystem_malloc.dylib`malloc_error_break
libsystem_malloc.dylib`malloc_error_break:
-> 0x7fff5f7d89e6 <+0>: pushq %rbp
0x7fff5f7d89e7 <+1>: movq %rsp, %rbp
0x7fff5f7d89ea <+4>: nop
0x7fff5f7d89eb <+5>: nopl (%rax)
Target 0: (python) stopped.
Process 53294 launched: '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp_g9gpq37/bin/python' (x86_64)
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
* frame #0: 0x00007fff5f7d89e6 libsystem_malloc.dylib`malloc_error_break
frame #1: 0x00007fff5f7ca82d libsystem_malloc.dylib`free + 400
frame #2: 0x000000010face5bd libstdc++.6.dylib`std::locale::_Impl::~_Impl() + 109
frame #3: 0x00007fff97d24998 libsystem_c.dylib`__stdinp + 8
(lldb) c
Whole code is here https://github.com/Czaki/imagecodecs/tree/2019.12.16
Pipelines are here https://dev.azure.com/bokota/open%20source%20contrib/_build?definitionId=3&_a=summary
Any idea how I can debug this? How to find where pointer is created?
EDIT
In this case problem is connected with compiling zopflipy with gcc-9/g++-9. I need to compile test dependencies with clang

You can check Enable system diagnostics to get verbose logs when you Run pipeline.
You can also click add variable to add a variable named system.debug and set its value to true. Please refer to troubleshoot build and release.
If you would like to login on the agent machine. You can configure your self-hosted agent, which will allow you to login and troubleshoot your builds.

uwsgi can't build plugin.so for python3.6

I was trying to build uwsgi+django server. I need to build uwsgi python36_plugin.so for python3.6 using this command:
make PROFILE=nolang
PYTHON=python3 ./uwsgi --build-plugin "plugins/python python36"
It didn't work:
[root#izuf64a9gck81cz uwsgi-2.0.15]# PYTHON=python ./uwsgi --build-plugin "plugins/python python36"
*** uWSGI building and linking plugin from plugins/python ***
[gcc -pthread] python36_plugin.so
/usr/bin/ld: /usr/python3/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.a(abstract.o): relocation R_X86_64_32S against `_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC
/usr/python3/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
*** unable to build python36 plugin ***
When i use build python2.7 version,it turns out right
PYTHON=python2.7 ./uwsgi --build-plugin "plugins/python python27"

My problem is I have two version of python 3, the default python3 is python3.4, and the second one is python3.6, and I want to use python3.6 for some of my django project so I need to build the plugins and I got same problem you got.
It took me couple hours to solved this issue. Here is what I did from here:
Reconfigure my python3.6
cd Python-3.6.4/
./configure --enable-shared --enable-optimizations --prefix=/usr/local LDFLAGS="-Wl,--rpath=/usr/local/lib"
Get in to your uwsgi folder and try to build again
PYTHON=python3.6 ./uwsgi --build-plugin "plugins/python python36"
Goodluck! May it solve your prob too!

Raspberry Pie 3 Why do I get an error when installing wxpython?

I am trying to install wxpython on raspberry pi3.
raspbian2.4.1 / Python 3.4 pip3 is version 9.0.1.
Firmware, Updates, Upgrades
Apt-get install libgtk2.0-dev
Apt-get install libgtk-3-dev
Apt-get install gstreamer1.0 *
Apt-get install libgstreamer-plugins-base1.0-dev
Apt-get install freeglut3-dev
If you install pip3 install wxpython after this pre-installation, it will not continue to install as shown in the picture. What's wrong?
Cpu and ram usage soar to 100%.
---- 2017-07-17 edit ----
Thank you all for your reply. I've tried a lot of the answers but I still can not install it.
I did not have any problems with Ubuntu 16.04, but I do not know why Raspberry Pie 3 is doing this in Razbian. Is this because Python 3 is not fully supported by RazBian?
I unzipped tar.gz and installed setup.py inside with the following command. I think the error is on the webkit gateways.
Sudo apt-get install libwebkitgtk-3.0-dev
Sudo apt-get install libwebkitgtk-dev
I have also tried installing Raspberry Pie 3 python3, wxpython phoenix version I am doing wrong how to install?
pi#raspberrypi:~/Downloads/wxPython-4.0.0a3 $ python3 setup.py install --user
running install
running build
WARNING: Building this way assumes that all generated files have been
generated already. If that is not the case then use build.py directly
to generate the source and perform the build stage. You can use
--skip-build with the bdist_* or install commands to avoid this
message and the wxWidgets and Phoenix build steps in the future.
"/usr/bin/python3" -u build.py build
Will build using: "/usr/bin/python3"
3.4.2 (default, Oct 19 2014, 13:31:11)
[GCC 4.9.1]
Python's architecture is 32bit
cfg.VERSION: 4.0.0a3
Running command: build
Running command: build_wx
wxWidgets build options: ['--wxpython', '--unicode', '--no_config']
Configure options: ['--enable-unicode', '--with-opengl', '--enable-sound', '--enable-graphics_ctx', '--enable-mediactrl', '--enable-display', '--enable-geometry', '--enable-debug_flag', '--enable-optimise', '--disable-debugreport', '--enable-uiactionsim', '--with-sdl']
/usr/bin/make
make --jobs=4
(if test -f utils/wxrc/Makefile ; then cd utils/wxrc && make all ; fi)
make[1]: Entering directory '/home/pi/Downloads/wxPython-4.0.0a3/build/wxbld/utils/wxrc'
make[1]: Nothing to be done for 'all'.
........
4.0.0a3/ext/wxWidgets/locale'
Finished command: build_wx (0m3.803s)
Running command: build_py
Checking for /home/pi/Downloads/wxPython-4.0.0a3/bin/waf-1.7.15-p1...
"/usr/bin/python3" /home/pi/Downloads/wxPython-4.0.0a3/bin/waf-1.7.15-p1 --wx_config=/home/pi/Downloads/wxPython-4.0.0a3/build/wxbld/wx-config --python="/usr/bin/python3" --out=build/waf/3.4 configure build
Setting top to : /home/pi/Downloads/wxPython-4.0.0a3
Setting out to : /home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4
Checking for 'gcc' (c compiler) : /usr/bin/gcc
Checking for 'g++' (c++ compiler) : /usr/bin/g++
Checking for program python : /usr/bin/python3
Checking for python version : (3, 4, 2, 'final', 0)
Checking for library python3.4 in LIBDIR : not found
Checking for library python3.4 in python_LIBPL : yes
Checking for program /usr/bin/python3-config,python3.4-config,python-config-3.4,python3.4m-config : /usr/bin/python3-config
Checking for header Python.h : yes
......
Waf: Leaving directory `/home/pi/Downloads/wxPython-.0.0a3/build/waf/3.4'
Build failed
-> task in '_html2' failed (exit status 1):
{task 1943459824: cxx sip_html2wxWebViewEvent.cpp -> sip_html2wxWebViewEvent.cpp.17.o}
['/usr/bin/g++', '-fPIC', '-pthread', '-pthread', '-UNDEBUG', '-g', '-pthread', '-fwrapv', '-fstack-protector-strong', '-flto', '-fuse-linker-plugin', '-ffat-lto-objects', '-fno-strict-aliasing', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/wxbld/lib/wx/include/gtk2-unicode-3.0', '-I/home/pi/Downloads/wxPython-4.0.0a3/ext/wxWidgets/include', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4/sip/siplib', '-I/home/pi/Downloads/wxPython-4.0.0a3/sip/siplib', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4/src', '-I/home/pi/Downloads/wxPython-4.0.0a3/src', '-I/usr/include/python3.4m', '-DPYTHONDIR="/usr/local/lib/python3/dist-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python3/dist-packages"', '-DHAVE_PYTHON_H=1', '-DHAVE_WX=1', '-DHAVE_WXADV=1', '-DHAVE_WXSTC=1', '-DHAVE_WXHTML=1', '-DHAVE_WXGL=1', '-DHAVE_WXWEBVIEW=1', '-DHAVE_WXXML=1', '-DHAVE_WXXRC=1', '-DHAVE_WXRICHTEXT=1', '-DHAVE_WXMEDIA=1', '-DHAVE_WXRIBBON=1', '-DHAVE_WXPROPGRID=1', '-DHAVE_WXAUI=1', '-D_FILE_OFFSET_BITS=64', '-DWXUSINGDLL', '-D__WXGTK__', '../../../sip/cpp/sip_html2wxWebViewEvent.cpp', '-c', '-o', 'sip/cpp/sip_html2wxWebViewEvent.cpp.17.o']
-> task in '_html2' failed (exit status 1):
{task 1943459632: cxx sip_html2cmodule.cpp -> sip_html2cmodule.cpp.17.o}
['/usr/bin/g++', '-fPIC', '-pthread', '-pthread', '-UNDEBUG', '-g', '-pthread', '-fwrapv', '-fstack-protector-strong', '-flto', '-fuse-linker-plugin', '-ffat-lto-objects', '-fno-strict-aliasing', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/wxbld/lib/wx/include/gtk2-unicode-3.0', '-I/home/pi/Downloads/wxPython-4.0.0a3/ext/wxWidgets/include', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4/sip/siplib', '-I/home/pi/Downloads/wxPython-4.0.0a3/sip/siplib', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4/src', '-I/home/pi/Downloads/wxPython-4.0.0a3/src', '-I/usr/include/python3.4m', '-DPYTHONDIR="/usr/local/lib/python3/dist-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python3/dist-packages"', '-DHAVE_PYTHON_H=1', '-DHAVE_WX=1', '-DHAVE_WXADV=1', '-DHAVE_WXSTC=1', '-DHAVE_WXHTML=1', '-DHAVE_WXGL=1', '-DHAVE_WXWEBVIEW=1', '-DHAVE_WXXML=1', '-DHAVE_WXXRC=1', '-DHAVE_WXRICHTEXT=1', '-DHAVE_WXMEDIA=1', '-DHAVE_WXRIBBON=1', '-DHAVE_WXPROPGRID=1', '-DHAVE_WXAUI=1', '-D_FILE_OFFSET_BITS=64', '-DWXUSINGDLL', '-D__WXGTK__', '../../../sip/cpp/sip_html2cmodule.cpp', '-c', '-o', 'sip/cpp/sip_html2cmodule.cpp.17.o']
-> task in '_html2' failed (exit status 1):
{task 1943459760: cxx sip_html2wxWebViewArchiveHandler.cpp -> sip_html2wxWebViewArchiveHandler.cpp.17.o}
['/usr/bin/g++', '-fPIC', '-pthread', '-pthread', '-UNDEBUG', '-g', '-pthread', '-fwrapv', '-fstack-protector-strong', '-flto', '-fuse-linker-plugin', '-ffat-lto-objects', '-fno-strict-aliasing', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/wxbld/lib/wx/include/gtk2-unicode-3.0', '-I/home/pi/Downloads/wxPython-4.0.0a3/ext/wxWidgets/include', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4/sip/siplib', '-I/home/pi/Downloads/wxPython-4.0.0a3/sip/siplib', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4/src', '-I/home/pi/Downloads/wxPython-4.0.0a3/src', '-I/usr/include/python3.4m', '-DPYTHONDIR="/usr/local/lib/python3/dist-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python3/dist-packages"', '-DHAVE_PYTHON_H=1', '-DHAVE_WX=1', '-DHAVE_WXADV=1', '-DHAVE_WXSTC=1', '-DHAVE_WXHTML=1', '-DHAVE_WXGL=1', '-DHAVE_WXWEBVIEW=1', '-DHAVE_WXXML=1', '-DHAVE_WXXRC=1', '-DHAVE_WXRICHTEXT=1', '-DHAVE_WXMEDIA=1', '-DHAVE_WXRIBBON=1', '-DHAVE_WXPROPGRID=1', '-DHAVE_WXAUI=1', '-D_FILE_OFFSET_BITS=64', '-DWXUSINGDLL', '-D__WXGTK__', '../../../sip/cpp/sip_html2wxWebViewArchiveHandler.cpp', '-c', '-o', 'sip/cpp/sip_html2wxWebViewArchiveHandler.cpp.17.o']
-> task in '_html2' failed (exit status 1):
{task 1943459696: cxx sip_html2wxWebView.cpp -> sip_html2wxWebView.cpp.17.o}
['/usr/bin/g++', '-fPIC', '-pthread', '-pthread', '-UNDEBUG', '-g', '-pthread', '-fwrapv', '-fstack-protector-strong', '-flto', '-fuse-linker-plugin', '-ffat-lto-objects', '-fno-strict-aliasing', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/wxbld/lib/wx/include/gtk2-unicode-3.0', '-I/home/pi/Downloads/wxPython-4.0.0a3/ext/wxWidgets/include', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4/sip/siplib', '-I/home/pi/Downloads/wxPython-4.0.0a3/sip/siplib', '-I/home/pi/Downloads/wxPython-4.0.0a3/build/waf/3.4/src', '-I/home/pi/Downloads/wxPython-4.0.0a3/src', '-I/usr/include/python3.4m', '-DPYTHONDIR="/usr/local/lib/python3/dist-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python3/dist-packages"', '-DHAVE_PYTHON_H=1', '-DHAVE_WX=1', '-DHAVE_WXADV=1', '-DHAVE_WXSTC=1', '-DHAVE_WXHTML=1', '-DHAVE_WXGL=1', '-DHAVE_WXWEBVIEW=1', '-DHAVE_WXXML=1', '-DHAVE_WXXRC=1', '-DHAVE_WXRICHTEXT=1', '-DHAVE_WXMEDIA=1', '-DHAVE_WXRIBBON=1', '-DHAVE_WXPROPGRID=1', '-DHAVE_WXAUI=1', '-D_FILE_OFFSET_BITS=64', '-DWXUSINGDLL', '-D__WXGTK__', '../../../sip/cpp/sip_html2wxWebView.cpp', '-c', '-o', 'sip/cpp/sip_html2wxWebView.cpp.17.o']
Command '"/usr/bin/python3" /home/pi/Downloads/wxPython-4.0.0a3/bin/waf-1.7.15-p1 --wx_config=/home/pi/Downloads/wxPython-4.0.0a3/build/wxbld/wx-config --python="/usr/bin/python3" --out=build/waf/3.4 configure build ' failed with exit code 1.
Finished command: build_py (0m18.606s)
Finished command: build (0m22.411s)
Command '"/usr/bin/python3" -u build.py build' failed with exit code 1.
In raspbian, add software-wxpython search. After installing all wxwidgets, the wxpython module is installed.
I checked wxpython (4.0.0a3) with pip3 list and imported it in python3.
I do not know what wxwidgets installed, but thanks to everyone who answered.

freeze_graph.py throws an error during build

I'm trying to build freeze_graph.py as specified on the readme part:
bazel build tensorflow/python/tools:freeze_graph && \
bazel-bin/tensorflow/python/tools/freeze_graph \
--input_graph=some_graph_def.pb \
--input_checkpoint=model.ckpt-8361242 \
--output_graph=/tmp/frozen_graph.pb --output_node_names=softmax
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py
But after the first command, it throws this error:
ERROR:
/home/bernardo/.cache/bazel/_bazel_bernardo/6ec3db0a5b478824b301f4851c10d381/external/boringssl/BUILD:91:1:
C++ compilation of rule '#boringssl//:crypto' failed: gcc failed:
error executing command /usr/bin/gcc -U_FORTIFY_SOURCE
'-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now
-B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-canonical-system-headers ... (remaining 42 argument(s) skipped):
com.google.devtools.build.lib.shell.BadExitStatusException: Process
exited with status 1. In file included from
/usr/include/stdint.h:25:0,
from /usr/lib/gcc/x86_64-pc-linux-gnu/6.2.1/include/stdint.h:9,
from external/boringssl/src/include/openssl/base.h:60,
from external/boringssl/src/include/openssl/chacha.h:18,
from external/boringssl/src/crypto/chacha/chacha.c:17:
/usr/include/features.h:331:4: error: #warning _FORTIFY_SOURCE
requires compiling with optimization (-O) [-Werror=cpp] # warning
_FORTIFY_SOURCE requires compiling with optimization (-O)
^~~~~~~ cc1: all warnings being treated as errors Target //tensorflow/python/tools:freeze_graph failed to build Use
--verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.485s, Critical Path: 0.24s [bernardo#foobar-pc
tensorflow]$
I'm trying to run it to create a new .pb file to import in an Android project.

From the error it's saying it wants you to bazel build with "-O" option to get the optimized build.

I have solved this issue by adding --copt=-O to the build command.
The whole command used to build freeze_graph was:
bazel build --copt=-O --config=cuda tensorflow/python/tools:freeze_graph

Python missing or unusable while compiling gdb under Cent OS 5

I checked this question and it is not applicable.
Python missing or unusable error while cross compiling GDB
I'm trying to build gdb 7.5 with python support on Cent OS 5.
configure --with-expat --with-python
Does not log any errors and the Python devel files are installed.
I set LDFLAGS="L/usr/local/expat-2.1 -L/opt/python27/lib
/opt/python27/lib contains:
rwxrwxrwx 1 root root 19 Mar 14 15:35 libpython2.7.so -> libpython2.7.so.1.0
-r-xr-xr-x 1 root root 1893448 Aug 30 2012 libpython2.7.so.1.0
drwxr-xr-x 2 root root 4096 Mar 14 15:35 pkgconfig
drwxr-xr-x 27 root root 20480 Mar 14 21:49 python2.7
Running make produces this error:
checking whether to use python... yes
checking for python... /opt/python27/bin/python
checking for python2.7... no
configure: error: python is missing or unusable
/opt/python27/bin/python exists has the correct permissions and is executable.
Why can't the link process find python?

The next line AFTER the configure: error: line should be something like:
make[1]: [configure-gdb]: ....
telling you that this error comes from running configure in the gdb subdirectory. So you should look in the config.log file there. This should show you something that configure tried to do that failed -- probably a chunk of C code that failed to compile, with an error message from the compiler something like 'can't find include file "Python.h"'. This tells you what the problem is -- it couldn't find the python header file used to embed python in gdb.
Now your problem may be slightly different, but its probably related to not having the python embedding components properly installed.

Why can't the link process find python?
It's not the link process; it's the configure process in one of the subdirectories.
Find out which subdirectory it is (make should say Leaving directory XXXX), and look in config.log in that subdirectory for what went wrong.
The reason I posted is that config.log does not point to a problem, yet when I run make I receive the error from the original post
The make process runs additional configure processes. Your error is clearly from that additional process. You claim to have checked all config.logs in all subdirectories, but that appears unlikely given the symptoms you've described.

Sorry for making you wait 10 years for an answer.
In my case I looked under the gdb/config.log file and found this:
configure:10369: checking for python2.7
configure:10387: gcc -o conftest -g -O2 -I/home/user/stevie-bsp/toolchain-build/builds/destdir/aarch64-host-linux-gnu/include -I/usr/include/python2.7 -I/usr/include/python2.7 -static-libstdc++ -static-libgcc conftest.c -ldl -lncursesw -lm -ldl -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions >&5
conftest.c:54:10: fatal error: Python.h: No such file or directory
54 | #include "Python.h"
| ^~~~~~~~~~
compilation terminated.
Looks like it is a misleading error message when the Python 2 headers aren't installed. I fixed that on Ubuntu with apt install python2-dev; on CentOS the package name would vary.
Secondly, the command python must start the Python 2 interpreter. The build process will not give a clear error about this and silently fails to query the include path for the Python headers. I resolved it by using the python-is-python2 package on Ubuntu. I'm not sure about CentOS but in the worst case you can create a symlink.