Many undefined references when using go-python3 library - python

Describe what happened:
For context, i am currently trying to embed python into golang. I came across this module go-python3 and decided to use it. I was originally using the python3.7 variant of the module then decided to upgrade my python version from 3.7 to 3.8.
Read: https://github.com/go-python/cpy3/issues/29#issuecomment-1321199010
I replaced python3.pc in pkgconfig with python-3.8.pc and got the following error when trying to run:
protoc -Isrc/proto --go_opt=module=grpc-golang --go_out=. --go-grpc_opt=module=grpc-golang --go-grpc_out=. src/proto/*.proto
go build -o dest/server ./src/server && PYTHONPATH=.
# grpc-golang/src/server
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x78): undefined reference to `PyBool_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x80): undefined reference to `PyByteArray_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x88): undefined reference to `PyBytes_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x90): undefined reference to `PyComplex_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x98): undefined reference to `PyDict_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xa0): undefined reference to `PyExc_ArithmeticError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xa8): undefined reference to `PyExc_AssertionError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xb0): undefined reference to `PyExc_AttributeError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xb8): undefined reference to `PyExc_BaseException'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xc0): undefined reference to `PyExc_BlockingIOError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xc8): undefined reference to `PyExc_BrokenPipeError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xd0): undefined reference to `PyExc_BufferError'
:
This are some of the details:
>> python3 --version
3.8.10
>> pkg-config --cflags --libs python3
-I/usr/include/python3.8 -I/usr/include/x86_64-linux-gnu/python3.8
>> echo $PKG_CONFIG_PATH
/usr/lib/x86_64-linux-gnu/pkgconfig/
>> pkg-config --list-all
python-3.8 Python - Build a C extension for Python
zlib zlib - zlib compression library
protobuf-lite Protocol Buffers - Google's Data Interchange Format
bash-completion bash-completion - programmable completion for the bash shell
xkeyboard-config XKeyboardConfig - X Keyboard configuration data
protobuf Protocol Buffers - Google's Data Interchange Format
systemd systemd - systemd System and Service Manager
libdmmp libdmmp - Device mapper multipath management library
python3 Python - Build a C extension for Python
libxcrypt libxcrypt - Extended crypt library for DES, MD5, Blowfish and others
udev udev - udev
shared-mime-info shared-mime-info - Freedesktop common MIME database
python-3.8-embed Python - Embed Python into an application
expat expat - expat XML parser
python3-embed Python - Embed Python into an application
libcrypt libxcrypt - Extended crypt library for DES, MD5, Blowfish and others
Describe what you expected:
Should be able to work fine, like its 3.7 variant.

I ran into a similar problem just recently. After searching through a few forums on embedding python, I ran into this one: "Embedding Python in C, linking fails with undefined reference to `Py_Initialize'" 1. I wasn't entirely sure how to do this at first in golang, but I had been doing other online searches on embedded python in golang. Going back through these, I landed at an article call, "Python and Go : Part IV - Using Python in Memory" 2, where it describes in the "Building" section how to specify specific flags to be use in the C building process. Basically, it shows that you can define flags in one of files as such:
/*
#cgo pkg-config: python3
#cgo LDFLAGS: -L/usr/local/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/local/lib -lpython3.8 -lcrypt -ldl -lm -lm
#include "Python.h"
*/
import "C"
Although, according to Python's documentation on compiling a python interpreter embedded in another application, you should get the LDFLAGS via,
python3.7-config --ldflags
for python versions <= 3.7 (I don't know how far back if goes. Also, replace the "python3.7-config" accordingly), or,
python3.10-config --ldflags --embed
for python versions > 3.7 (And once again, replace the "python3.10-config" part with the desired python version)
I then thought to myself, "A version of this header is probably located somewhere in the go-python project and it just needs to be changed", so after looking around, I found this header in the high_level_layer.go file in the project and modified it to match what's shown above.
What's exactly going on here is that golang has native functionality with C, so libraries written in C can work in golang, and the opposite is also true (with enough finagling). There's plenty of articles on this functionality. Here's one from go's official website, 3. And since the base python interpreter is written in C, its header and object files can be referenced at compile time by the golang compiler if the linker flags and other flags are set correctly, so that's exactly what's being specified above.
Also, on a different note, I've tried compiling with a few different go compilers. I've had success with the compiler for versions 1.15.5 and 1.18.1, but haven't with 1.19.5. I don't know what's exactly happening, but I'm not going to try and figure it out right now since all I need currently are features in compiler versions >= 1.18. Lastly, I actually had basically the same problems with python3.7, and these changes fixed these problems too...

Related

Embeded python3.6 with MinGW in C++ fail on linking

Try to embed python3.6 in C++ on Win10 64bit system.
Python is installed by python-3.6.1-amd64.exe.
The CMakeList.txt is showing below
cmake_minimum_required(VERSION 3.8)
project(EmbedPython)
set(CMAKE_CXX_STANDARD 11)
set(SOURCE_FILES main.cpp)
find_package(PythonLibs REQUIRED)
include_directories(${PYTHON_INCLUDE_DIRS})
add_executable(EmbedPython ${SOURCE_FILES} ${PYTHON_INCLUDE_DIRS})
target_link_libraries(EmbedPython ${PYTHON_LIBRARIES})
And CMake found the Python package.
PYTHON_INCLUDE_DIR C:/Program Files (x86)/Python36/include
PYTHON_LIBRARY C:/Program Files (x86)/Python36/libs/python36.lib
Find a comment on github it's talking about the API problem.
# used to embed python script
find_package(PythonLibs 2.7 REQUIRED) # this has to be 2.7 because the 3.0 API requires argv to be wchar_t** rather than the default char**
include_directories(${PYTHON_INCLUDE_DIRS})
target_link_libraries( assignment ${PYTHON_LIBRARIES})
The errors below is what I'm facing.
[ 50%] Building CXX object CMakeFiles/EmbedPython.dir/main.cpp.obj
[100%] Linking CXX executable EmbedPython.exe
CMakeFiles\EmbedPython.dir/objects.a(main.cpp.obj): In function `main':
D:/Projects/EmbedPython/main.cpp:16: undefined reference to `_imp__Py_Initialize'
D:/Projects/EmbedPython/main.cpp:17: undefined reference to `_imp__PyUnicode_DecodeFSDefault'
D:/Projects/EmbedPython/main.cpp:20: undefined reference to `_imp__PyImport_Import'
D:/Projects/EmbedPython/main.cpp:24: undefined reference to `_imp__PyObject_GetAttrString'
D:/Projects/EmbedPython/main.cpp:27: undefined reference to `_imp__PyCallable_Check'
D:/Projects/EmbedPython/main.cpp:28: undefined reference to `_imp__PyTuple_New'
D:/Projects/EmbedPython/main.cpp:30: undefined reference to `_imp__PyLong_FromLong'
D:/Projects/EmbedPython/main.cpp:38: undefined reference to `_imp__PyTuple_SetItem'
D:/Projects/EmbedPython/main.cpp:40: undefined reference to `_imp__PyObject_CallObject'
D:/Projects/EmbedPython/main.cpp:43: undefined reference to `_imp__PyLong_AsLong'
D:/Projects/EmbedPython/main.cpp:49: undefined reference to `_imp__PyErr_Print'
D:/Projects/EmbedPython/main.cpp:55: undefined reference to `_imp__PyErr_Occurred'
D:/Projects/EmbedPython/main.cpp:56: undefined reference to `_imp__PyErr_Print'
D:/Projects/EmbedPython/main.cpp:63: undefined reference to `_imp__PyErr_Print'
D:/Projects/EmbedPython/main.cpp:67: undefined reference to `_imp__Py_FinalizeEx'
collect2.exe: error: ld returned 1 exit status
mingw32-make.exe[3]: *** [EmbedPython.exe] Error 1
mingw32-make.exe[2]: *** [CMakeFiles/EmbedPython.dir/all] Error 2
mingw32-make.exe[1]: *** [CMakeFiles/EmbedPython.dir/rule] Error 2
mingw32-make.exe: *** [EmbedPython] Error 2
CMakeFiles\EmbedPython.dir\build.make:96: recipe for target 'EmbedPython.exe' failed
CMakeFiles\Makefile2:66: recipe for target 'CMakeFiles/EmbedPython.dir/all' failed
CMakeFiles\Makefile2:78: recipe for target 'CMakeFiles/EmbedPython.dir/rule' failed
Makefile:117: recipe for target 'EmbedPython' failed
Not sure what to do to make the project run.Any help would be appreciated.
EDIT:
the result from make VERBOSE=1
D:\Projects\EmbedPython\cmake-build-debug>make VERBOSE=1
"C:\Program Files\JetBrains\CLion 172.2273.4\bin\cmake\bin\cmake.exe" -HD:\Projects\EmbedPython -BD:\Projects\EmbedPython\cmake-build-debug --check-build-system CMakeFiles\Makefile.cmake 0
"C:\Program Files\JetBrains\CLion 172.2273.4\bin\cmake\bin\cmake.exe" -E cmake_progress_start D:\Projects\EmbedPython\cmake-build-debug\CMakeFiles D:\Projects\EmbedPython\cmake-build-debug\CMakeFiles\progress.marks
make -f CMakeFiles\Makefile2 all
make[1]: Entering directory 'D:/Projects/EmbedPython/cmake-build-debug'
make -f CMakeFiles\EmbedPython.dir\build.make CMakeFiles/EmbedPython.dir/depend
make[2]: Entering directory 'D:/Projects/EmbedPython/cmake-build-debug'
"C:\Program Files\JetBrains\CLion 172.2273.4\bin\cmake\bin\cmake.exe" -E cmake_depends "MinGW Makefiles" D:\Projects\EmbedPython D:\Projects\EmbedPython D:\Projects\EmbedPython\cmake-build-debug D:\Projects\EmbedPython\cmake-build-debug D:\Projects\EmbedPython\cmake-build-debug\CMakeFiles\EmbedPython.dir\DependInfo.cmake --color=
make[2]: Leaving directory 'D:/Projects/EmbedPython/cmake-build-debug'
make -f CMakeFiles\EmbedPython.dir\build.make CMakeFiles/EmbedPython.dir/build
make[2]: Entering directory 'D:/Projects/EmbedPython/cmake-build-debug'
[ 50%] Linking CXX executable EmbedPython.exe
"C:\Program Files\JetBrains\CLion 172.2273.4\bin\cmake\bin\cmake.exe" -E cmake_link_script CMakeFiles\EmbedPython.dir\link.txt --verbose=1
"C:\Program Files\JetBrains\CLion 172.2273.4\bin\cmake\bin\cmake.exe" -E remove -f CMakeFiles\EmbedPython.dir/objects.a
C:\PROGRA~2\MINGW-~1\I686-6~1.0-P\mingw32\bin\ar.exe cr CMakeFiles\EmbedPython.dir/objects.a #CMakeFiles\EmbedPython.dir\objects1.rsp
C:\PROGRA~2\MINGW-~1\I686-6~1.0-P\mingw32\bin\G__~1.EXE -g -Wl,--whole-archive CMakeFiles\EmbedPython.dir/objects.a -Wl,--no-whole-archive -o EmbedPython.exe -Wl,--out-implib,libEmbedPython.dll.a -Wl,--major-image-version,0,--minor-image-version,0 #CMakeFiles\EmbedPython.dir\linklibs.rsp
CMakeFiles\EmbedPython.dir/objects.a(main.cpp.obj): In function `main':
D:/Projects/EmbedPython/main.cpp:16: undefined reference to `_imp__Py_Initialize'
D:/Projects/EmbedPython/main.cpp:17: undefined reference to `_imp__PyUnicode_DecodeFSDefault'
D:/Projects/EmbedPython/main.cpp:20: undefined reference to `_imp__PyImport_Import'
D:/Projects/EmbedPython/main.cpp:24: undefined reference to `_imp__PyObject_GetAttrString'
D:/Projects/EmbedPython/main.cpp:27: undefined reference to `_imp__PyCallable_Check'
D:/Projects/EmbedPython/main.cpp:28: undefined reference to `_imp__PyTuple_New'
D:/Projects/EmbedPython/main.cpp:30: undefined reference to `_imp__PyLong_FromLong'
D:/Projects/EmbedPython/main.cpp:38: undefined reference to `_imp__PyTuple_SetItem'
D:/Projects/EmbedPython/main.cpp:40: undefined reference to `_imp__PyObject_CallObject'
D:/Projects/EmbedPython/main.cpp:43: undefined reference to `_imp__PyLong_AsLong'
D:/Projects/EmbedPython/main.cpp:49: undefined reference to `_imp__PyErr_Print'
D:/Projects/EmbedPython/main.cpp:55: undefined reference to `_imp__PyErr_Occurred'
D:/Projects/EmbedPython/main.cpp:56: undefined reference to `_imp__PyErr_Print'
D:/Projects/EmbedPython/main.cpp:63: undefined reference to `_imp__PyErr_Print'
D:/Projects/EmbedPython/main.cpp:67: undefined reference to `_imp__Py_FinalizeEx'
collect2.exe: error: ld returned 1 exit status
CMakeFiles\EmbedPython.dir\build.make:97: recipe for target 'EmbedPython.exe' failed
make[2]: *** [EmbedPython.exe] Error 1
make[2]: Leaving directory 'D:/Projects/EmbedPython/cmake-build-debug'
CMakeFiles\Makefile2:66: recipe for target 'CMakeFiles/EmbedPython.dir/all' failed
make[1]: *** [CMakeFiles/EmbedPython.dir/all] Error 2
make[1]: Leaving directory 'D:/Projects/EmbedPython/cmake-build-debug'
Makefile:82: recipe for target 'all' failed
make: *** [all] Error 2
I had the exactly same problem, just with Python 3.7 while trying to build the example from the Python manual.
My system is a 64-bit Windows, the installed Python is a 64-bit 3.7 (from Anaconda3), and the compiler is a 32-bit MinGW 5.3.0 (from Qt 5.11).
The simplest solution that worked for me is the following:
Download Windows x86 embeddable zip file from this page.
Put the content of that archive to a folder inside the project you're building.
Do not install anything from that archive and do not put it on the system PATH.
Add the above folder to the gcc library search path, using: -L"path/to/folder/"
Add the python library to the linker, using -l"python37" ( replace 37 with whatever version you download)
As noted here (this was a hard one to find!), add a definition, using -DPy_BUILD_CORE_BUILTIN=1 (It is also possible to write #define DPy_BUILD_CORE_BUILTIN 1 just before #include-ing the Python header, though I would not recommend it)
Provide the path to the Python headers, using -I"path/to/include". I used the one from Anaconda, e.g. -I"c:/Users/user/Anaconda3/include/". Any installed header will do, just make sure its from the same version (headers from an older version might also be used, but this may cause a rupture in the space-time continuum)
Make sure that #include <Python.h> is before any other include, as stated in the manual.
As a side note, the MinGW linker (ld) is perfectly content, for many years now, if you just give it the "dll" file. No "lib", nor "a", is needed if you don't have one and only perform dynamic linking. This is why I just told the linker to use python37.
It it actually smart enough to look for files like python37.lib, python37.dll & libpython37.a.
Also note that the "embedable zip" is just the bare minimum needed for, well, embedding Python.
And as a last comment, contrary to what is stated here, there is no problem whatsoever linking from GCC with a library compiled using MSVS (and vice versa), as long as the library uses the plain C interface, since that interface is pretty much the only thing that is compatible across different compilers for any given platform.
This is pretty much the exact reason why Python is using the C interface (and nothing else, not even C++) for extending & embedding - to allow code written in any language, using any tool set, to interface with Python.
I had the same problem.
probably the problem is because of using 32 bit MinGW and 64 bit Python. you should install 32 bit version of python
Hope it will be useful to you :-)

f2py is not able to link with NetCDF library -

I am trying install some Python modules in my system using f2py, which are related to an ocean model (in Fortran 90). I am facing some problems with f2py. To be specific, the f2py is unable to link with the NetCDF library even though I have the required library and include files installed. I am using Python 2.7 with Anaconda 2 on Ubuntu16.04 on a 64-bit machine. I'm using gfortran.
To test it's working, I wrote a small code - an f90 module that contains a small subroutine. The subroutine performs a basic math task, and calls a NetCDF routine that prints the NetCDF version installed. The module (testsub.f90) is as follows:
module testsub
implicit none
contains
subroutine f_sum(a, b, s)
!#include 'netcdf.inc'
use netcdf
real(8) :: a, b, s
s = a+b;
!Calls a function that prints the netcdf version
write(*,*) trim(nf90_inq_libvers())
end subroutine f_sum
end module
The makefile for testsub is:
#Fortran compiler
FC=gfortran
NCLIB = -L/home/sonaljit/anaconda2/lib -lnetcdf -lnetcdff -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7
NCINC = -I/home/sonaljit/anaconda2/include
#f2py and flags
F2PY = /home/sonaljit/anaconda2/bin/f2py
PYFLAGS = "-fPIC -g -O2 -fdefault-real-8"
pytest : testsub.f90
$(F2PY) --fcompiler=$(FC) --f90flags=$(PYFLAGS) -c $(NCINC) -m testpymod testsub.f90 $(NCLIB)
clean :
rm testpymod.so
I have the NetCDF library and include files installed in the given paths. When I run the makefile using make pytest, I get the following error:
/usr/bin/ld: /home/sonaljit/anaconda2/lib/libnetcdf.a(netcdf.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/home/sonaljit/anaconda2/lib/libnetcdf.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
However, I am not seeing this error when I am commenting out the NetCDF lines in the module. It seems f2py is not able to link to NetCDF routines. What can be the error here? Is it due to the structure of the code? Or, do I need to include some other library?
You are compiling a shared (dynamic) library and you should use a shared library version of NetCDF.
If you installed NetCDF yourself (as suggested by the path in your /home/sonaljit), you should install the .so version and link with this version.

Python.h compiling error: collect2.exe Id returned 1 exit status

So, I've searched all around the web and I couldn't find anything.
I'm started with python, and I made a TKinter application. At first, I tried to compile it to an .exe file, but that didn't work. So now I'm embedding python in C++. Every single time I try to compile it (using Dev-C++), I get the error:
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x10): undefined reference to `__imp_Py_Initialize'
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x25): undefined reference to `__imp_PyRun_SimpleStringFlags'
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x2e): undefined reference to `__imp_Py_Finalize'
F:\Documents\Videos\[program-name]\program\collect2.exe [Error] ld returned 1 exit status
F: stands for the flash drive.
In the Dev-c++ options, I added:
-Wall -I\C:\Users\*****\AppData\Local\Programs\Python\Python35\include
So this is the full command:
g++.exe "F:\Documents\Videos\[program-name]\program\[program-name].cpp" -o "F:\Documents\Videos\[program-name]\program\[program-name].exe" -Wall -I\C:\Users\*****\AppData\Local\Programs\Python\Python35\include -I"C:\Programma's\Dev-C++\MinGW64\include" -I"C:\Programma's\Dev-C++\MinGW64\x86_64-w64-mingw32\include" -I"C:\Programma's\Dev-C++\MinGW64\lib\gcc\x86_64-w64-mingw32\4.9.2\include" -I"C:\Programma's\Dev-C++\MinGW64\lib\gcc\x86_64-w64-mingw32\4.9.2\include\c++" -I"C:\Users\*****\AppData\Local\Programs\Python\Python35\include" -L"C:\Programma's\Dev-C++\MinGW64\lib" -L"C:\Programma's\Dev-C++\MinGW64\x86_64-w64-mingw32\lib" -static-libgcc
My question is: how do I deal with this?
And please note: I haven't got administrator privileges, I'm just a standard user.
Python version: 3.5.1
Dev-C++ version: 5.11
GCC version: I don't know, but I'll find it out if needed :)
EDIT: Because of the duplicate mark by NathanOliver, I'll ask it like this: which command line variables do I have to add to compile it?
The linker is complaining bacause it can't find a group of references about some python stuff, as a matter of fact I don't see any python library in the g++ invocation.
You need to had something like this -lpython3.5.1 to tell g++ to link against the python library (assuming python is installed in your system, else you will have to add the path to the library using the -L option).
More resources on g++ makefiles and linking:
https://www3.ntu.edu.sg/home/ehchua/programming/cpp/gcc_make.html

Using libspotify .dll/.lib files in MinGW32 compiling pySpotify

Using MinGW32 on a Windows PC I am trying to compile pySpotify. The first error was that libspotify/api.h was missing. I fixed this by copying the appropriate folder from libspotify into C:\MinGW\include. However now dllwrap is now failing with ld linking. The binaries Spotify distribute are libspotify.dll and libspotify.lib. No matter where I put them (pySpotify folder/subfolders, temp build folder/subfolders and MinGW folder/subfolders) or what I name them (.a, .o & .so) it still shows the same error messages.
The pertinent error is:
C:\MinGW\bin\dllwrap.exe -mdll -static --output-lib build\temp.win32-2.7\Release\src\lib_spotify.a --def build\temp.win32-2.7\Release\src\_spotify.def -s build\temp.win32-2.7\Release\src\module.o build\temp.win32-2.7\Release\src\session.o build\temp.win32-2.7\Release\src\link.o build\temp.win32-2.7\Release\src\track.obuild\temp.win32-2.7\Release\src\album.o build\temp.win32-2.7\Release\src\albumbrowser.o build\temp.win32-2.7\Release\src\artist.o build\temp.win32-2.7\Release\src\artistbrowser.o build\temp.win32-2.7\Release\src\search.o build\temp.win32-2.7\Release\src\playlist.o build\temp.win32-2.7\Release\src\playlistcontainer.o build\temp.win32-2.7\Release\src\playlistfolder.o build\temp.win32-2.7\Release\src\image.o build\temp.win32-2.7\Release\src\user.o build\temp.win32-2.7\Release\src\pyspotify.o build\temp.win32-2.7\Release\src\toplistbrowser.o -LC:\Python26\libs -LC:\Python26\PCbuild -lspotify -lpython26 -lmsvcr90 -o build\lib.win32-2.7\spotify\_spotify.pyd
c:/mingw/bin/../lib/gcc/mingw32/4.7.2/../../../../mingw32/bin/ld.exe: cannot find -lspotify
collect2.exe: error: ld returned 1 exit status
dllwrap: gcc exited with status 1
error: command 'dllwrap' failed with exit status 1
bok says on github that:
You need to add the API headers in the include path (add something like -I~\libspotify\include to your compiler options) and the shared library in the library path (add -L~\libspotify\lib to your linker options). This will allow the compiler to find the necessary include files, and your linker to find the necessary binary objects.
However the distutils Extension class seems to be deprecated and hard to find documentation for (I believe this is where custom compiler options need to go). Appreciate that ~ may need changed to %USERPROFILE% or similar. Similarly %PYTHONPATH%\Lib\distutils\distutils.cfg has little documentation beyond the [build] compiler=mingw32 stanza. This makes editing the compile/link commands and their options impossible to change.
How do you compile pySpotify on Windows?
EDIT:
By using Python 2.6 and copying libspotify.dll/libspotify.lib to C:\Python26\PCbuild and renaming them to spotify.dll/libspotify.lib I now receive another error message from ld:
C:\MinGW\bin\dllwrap.exe -mdll -static --output-lib build\temp.win32-2.6\Release\src\lib_spotify.a --def build\temp.win32-2.6\Release\src\_spotify.def -s build\temp.win32-2.6\Release\src\module.o build\temp.win32-2.6\Release\src\session.o build\temp.win32-2.6\Release\src\link.o build\temp.win32-2.6\Release\src\track.obuild\temp.win32-2.6\Release\src\album.o build\temp.win32-2.6\Release\src\albumbrowser.o build\temp.win32-2.6\Release\src\artist.o build\temp.win32-2.6\Release\src\artistbrowser.o build\temp.win32-2.6\Release\src\search.o build\temp.win32-2.6\Release\src\playlist.o build\temp.win32-2.6\Release\src\playlistcontainer.o build\temp.win32-2.6\Release\src\playlistfolder.o build\temp.win32-2.6\Release\src\image.o build\temp.win32-2.6\Release\src\user.o build\temp.win32-2.6\Release\src\pyspotify.o build\temp.win32-2.6\Release\src\toplistbrowser.o -LC:\Python26\libs -LC:\Python26\PCbuild -lspotify -lpython26 -lmsvcr90 -o build\lib.win32-2.6\spotify\_spotify.pyd
_spotify.exp: file not recognized: File format not recognized
collect2.exe: error: ld returned 1 exit status
dllwrap: gcc exited with status 1
error: command 'dllwrap' failed with exit status 1
Not having access to a mingw installation at the moment, I can suggest a few things.
First, ld is known to be picky about order of arguments. Strangely, when I googled for "ld argument order", I got a bunch of pages suggesting that the order doesn't matter, but I have been burned several times by this. I've had the most success with the following argument order:
Switches to ld (ie, -Wall)
Library search paths (ie, -LPATH)
Object files
Libraries (ie, -lspotify)
I noticed in your linker output some references to amd64. I'm not sure how you compiled the other object files that you have, but since libspotify is 32-bit on Windows, I'm guessing that mixing 32/64 bit here is not going to work out too well.
The last thing I can think of is that perhaps the dll extension is confusing to ld, have you tried changing the filename to libspotify.so? I know this is kind of a shot in the dark, but otherwise I'm not sure how to help you further.
I guess you are trying to link a 64bit version of pyspotify against a 32bit version of libspotify. Try to rebuild pyspotify adding the -m32 to CFLAGS and CXXFLAGS
I've added some instructions on how I got it to compile on the actual ticket for the issue. Let me Know if it helped you!
https://github.com/mopidy/pyspotify/issues/63

How can I build my C extensions with MinGW-w64 in Python?

So I have a few Python C extensions I have previously built for and used in 32 bit Python running in Win7. I have now however switched to 64 bit Python, and I am having issues building the C extension with MinGW-w64.
I made the changes to distutils as per this post, but I am getting some weird errors suggesting something is wrong:
$ python setup.py build
running build
running build_ext
building 'MyLib' extension
c:\MinGW64\bin\x86_64-w64-mingw32-gcc.exe -mdll -O -Wall -Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python27\include -Ic:\Python27\PC -c MyLib.c -o build\temp.win-amd64-2.7\Release\mylib.o
MyLib.c: In function 'initMyLib':
MyLib.c:631:5: warning: implicit declaration of function 'Py_InitModule4_64' [-Wimplicit-function-declaration]
writing build\temp.win-amd64-2.7\Release\MyLib.def
c:\MinGW64\bin\x86_64-w64-mingw32-gcc.exe -shared -s build\temp.win-amd64-2.7\Release\mylib.o build\temp.win-amd64-2.7\Release\MyLib.def -Lc:\Python27\libs -Lc:\Python27\PCbuild\amd64 -lpython27 -o build\lib.win-amd64-2.7\MyLib.pyd
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x13d): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1275): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1eef): undefined reference to `__imp_PyExc_ImportError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f38): undefined reference to `__imp_PyExc_AttributeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f4d): undefined reference to `__imp_PyCObject_Type'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f61): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1fc7): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1ffe): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x2042): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x206c): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x208a): more undefined references to `__imp_PyExc_RuntimeError' follow
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x20a7): undefined reference to `__imp_PyExc_ImportError'
collect2.exe: error: ld returned 1 exit status
error: command 'x86_64-w64-mingw32-gcc' failed with exit status 1
I have googled around quite a bit to find information, but it's not easy to find a definite answer. Could someone shed some light on this? What further changes should I do to be able to successfully build C extensions for 64 bit Python in Win7?
EDIT:
After some helpful pointers in cgohlke's comments below I managed to generate libpython27.a. However after following the advice on this post (2nd to last) I still had a the __imp_Py_InitModule4_64 error. After some serious Google-fu I managed to trip over this post telling me to rename the Py_InitModule4 line to Py_InitModule4_64. After that everything worked swimmingly.
This worked for me with Python 3.3 :
create static python lib from dll
python dll is usually in C:/Windows/System32; in msys shell:
gendef.exe python33.dll
dlltool.exe --dllname python33.dll --def python33.def --output-lib libpython33.a
mv libpython33.a C:/Python33/libs
use swig to generate wrappers
e.g., swig -c++ -python myExtension.i
wrapper MUST be compiled with MS_WIN64, or your computer will crash when you import the class in Python
g++ -c myExtension.cpp -I/other/includes
g++ -DMS_WIN64 -c myExtension_wrap.cxx -IC:/Python33/include
shared library
g++ -shared -o _myExtension.pyd myExtension.o myExtension_wrap.o -lPython33 -lOtherSharedLibs -LC:/Python33/libs -LC:/path/to/other/shared/libs
make sure all shared libs (gdal, OtherSharedLibs) are in your PATH
(windows does not use LD_LIBRARY_PATH or PYTHONPATH)
in Python, just: import myExtension
voila!
I realize this is an old question, but it is still the top search result. Today, in 2019, I was able to do this:
https://github.com/PetterS/quickjs/commit/67bc2428b8c0716538b4583f4f2b0a2a5a49106c
In short:
Make sure a 64-bit version of mingw-w64 is in the PATH.
Monkey-patch distutils:
import distutils.cygwinccompiler
distutils.cygwinccompiler.get_msvcr = lambda: []
Some differences in the shell w.r.t. escaping.
extra_link_args = ["-Wl,-Bstatic", "-lpthread"] in order to link statically and not have extra runtime deps.
pipenv run python setup.py build -c mingw32 now works.
Here is a example code for VC++ Build Tools
https://github.com/starnight/python-c-extension/tree/master/00-HelloWorld
You could try:
python setup.py -c mingw32
However this is not work for me.
My Solution is:
install Anaconda 64bit python 3.6
install mingw64
add mingw64/bin to PATH
compile dll from c file by
gcc -c libmypy.c -IC:\Users\{user_name}\Anaconda3\pkgs\python-3.6.4-h6538335_1\include
gcc -shared -o libmypy.dll libmypy.o -LC:\Users\{user_name}\Anaconda3\pkgs\python-3.6.4-h6538335_1\libs -lPython36
load dll file in .py script
from ctypes import *
m = cdll.LoadLibrary(r"C:\{path_to_dll}\libmypy.dll")
print(m.hello())
I created a monkey-patch for setuptools to let you to build_ext with mingw64 on Windows easily. See https://github.com/imba-tjd/mingw64ccompiler
I used this thread to wade through learning how to make a C extension, and since most of what I learned is in it, I thought I'd put the final discovery here too, so that someone else can find it if they are looking.
I wasn't trying to compile something big, just the example in Hetland's Beginning Python. Here is what I did (the example C pgm is called palindrome.c). I'm using Anaconda with python 3.7 in it, and the TDM-GCC version of MinGW64. I put all of the tools used into my Path, and all of the paths needed in PYTHONPATH, and the ..\Anaconda3 directory into PYTHON_HOME. I still ended up using explicit paths on some things.
I created the libpython37.a library with gendef.exe and dlltool.exe as Mark said above, and put it in ..\Anaconda3\libs.
I followed the prescription in Hetland:
gcc -c palindrome.c
gcc -I$PYTHON_HOME -I$PYTHON_HOME/Include -c palindrome_wrap.c
The second failed, the compiler couldn't find Python.h, the following worked:
gcc -I[somedirectories]\Anaconda3\Include -c palindrome_wrap.c
I then did, as many have said, including Hetland 3rd ed.,
gcc -shared palindrome.o palindrome_wrap.o [somedirectories]/Anaconda3/libs/libpython37.a -o _palindrome.dll
This did not work. Even with the Load Library cswu used (which I found elsewhere, too).
So I gendef'd _palindrome.dll and couldn't find the function in it, "is_palindrome" in the exports. I went through some of the SWIG documentation, and declared the function both in the %{ %} section and below it, both extern, that finally got the function extern'd in palindrome_wrap.c as it should have been. But no export, so I went back into palindrome.c and redeclared the function as:
declspec(dllexport) extern int __stdcall is_palindrome(char* text)
and redeclared it in palindrome.i in both places as above with this signature.
Partial success! It got listed in the Export section when I gendef'd _palindrome.dll and I could do cswu's call using Load Library. But still not do what Hetland says and do
import _palindrome
in Python.
Going back to all the sources again, I could not figure this out. I finally started reading the SWIG documentation from the beginning leaving no stone unturned -- Searching through the manual doesn't produce the place found.
At the end of Introduction Sec. 2.7 Incorporating Into a Build System, under the sample Make process, it says:
"The above example will generate native build files such as makefiles, nmake files and Visual Studio projects which will invoke SWIG and compile the generated C++ files into _example.so (UNIX) or _example.pyd (Windows). For other target languages on Windows a dll, instead of a .pyd file, is usually generated."
And that's the answer to the last problem:
The compile step for the dll should read:
gcc -shared palindrome.o palindrome_wrap.o [somedirectories]/Anaconda3/libs/libpython37.a -o _palindrome.pyd
(I didn't go back and change out my declspec declarations so I don't know whether they were necessary, so they were still there too).
I got a file, _palindrome.pyd
Which if in the PYTHONPATH (mine was local) works, and one can then do
import _palindrome
from _palindrome import is_palindrome
and use the exported, properly wrapped and packaged C function, compiled with TDM-GCC, in python as promised. gcc, which is MinGW64 in a different installation, knows how to do the .pyd file. I diffed the dll and pyd since they were the same byte length. They are not the same at hundreds of points.
Hope this helps someone else.

Categories

Resources