Develop Reference

Many undefined references when using go-python3 library

Describe what happened:
For context, i am currently trying to embed python into golang. I came across this module go-python3 and decided to use it. I was originally using the python3.7 variant of the module then decided to upgrade my python version from 3.7 to 3.8.
Read: https://github.com/go-python/cpy3/issues/29#issuecomment-1321199010
I replaced python3.pc in pkgconfig with python-3.8.pc and got the following error when trying to run:
protoc -Isrc/proto --go_opt=module=grpc-golang --go_out=. --go-grpc_opt=module=grpc-golang --go-grpc_out=. src/proto/*.proto
go build -o dest/server ./src/server && PYTHONPATH=.
# grpc-golang/src/server
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x78): undefined reference to `PyBool_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x80): undefined reference to `PyByteArray_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x88): undefined reference to `PyBytes_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x90): undefined reference to `PyComplex_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0x98): undefined reference to `PyDict_Type'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xa0): undefined reference to `PyExc_ArithmeticError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xa8): undefined reference to `PyExc_AssertionError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xb0): undefined reference to `PyExc_AttributeError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xb8): undefined reference to `PyExc_BaseException'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xc0): undefined reference to `PyExc_BlockingIOError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xc8): undefined reference to `PyExc_BrokenPipeError'
/usr/bin/ld: /tmp/go-link-2733155445/go.o:(.data+0xd0): undefined reference to `PyExc_BufferError'
:
This are some of the details:
>> python3 --version
3.8.10
>> pkg-config --cflags --libs python3
-I/usr/include/python3.8 -I/usr/include/x86_64-linux-gnu/python3.8
>> echo $PKG_CONFIG_PATH
/usr/lib/x86_64-linux-gnu/pkgconfig/
>> pkg-config --list-all
python-3.8 Python - Build a C extension for Python
zlib zlib - zlib compression library
protobuf-lite Protocol Buffers - Google's Data Interchange Format
bash-completion bash-completion - programmable completion for the bash shell
xkeyboard-config XKeyboardConfig - X Keyboard configuration data
protobuf Protocol Buffers - Google's Data Interchange Format
systemd systemd - systemd System and Service Manager
libdmmp libdmmp - Device mapper multipath management library
python3 Python - Build a C extension for Python
libxcrypt libxcrypt - Extended crypt library for DES, MD5, Blowfish and others
udev udev - udev
shared-mime-info shared-mime-info - Freedesktop common MIME database
python-3.8-embed Python - Embed Python into an application
expat expat - expat XML parser
python3-embed Python - Embed Python into an application
libcrypt libxcrypt - Extended crypt library for DES, MD5, Blowfish and others
Describe what you expected:
Should be able to work fine, like its 3.7 variant.

I ran into a similar problem just recently. After searching through a few forums on embedding python, I ran into this one: "Embedding Python in C, linking fails with undefined reference to `Py_Initialize'" 1. I wasn't entirely sure how to do this at first in golang, but I had been doing other online searches on embedded python in golang. Going back through these, I landed at an article call, "Python and Go : Part IV - Using Python in Memory" 2, where it describes in the "Building" section how to specify specific flags to be use in the C building process. Basically, it shows that you can define flags in one of files as such:
/*
#cgo pkg-config: python3
#cgo LDFLAGS: -L/usr/local/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/local/lib -lpython3.8 -lcrypt -ldl -lm -lm
#include "Python.h"
*/
import "C"
Although, according to Python's documentation on compiling a python interpreter embedded in another application, you should get the LDFLAGS via,
python3.7-config --ldflags
for python versions <= 3.7 (I don't know how far back if goes. Also, replace the "python3.7-config" accordingly), or,
python3.10-config --ldflags --embed
for python versions > 3.7 (And once again, replace the "python3.10-config" part with the desired python version)
I then thought to myself, "A version of this header is probably located somewhere in the go-python project and it just needs to be changed", so after looking around, I found this header in the high_level_layer.go file in the project and modified it to match what's shown above.
What's exactly going on here is that golang has native functionality with C, so libraries written in C can work in golang, and the opposite is also true (with enough finagling). There's plenty of articles on this functionality. Here's one from go's official website, 3. And since the base python interpreter is written in C, its header and object files can be referenced at compile time by the golang compiler if the linker flags and other flags are set correctly, so that's exactly what's being specified above.
Also, on a different note, I've tried compiling with a few different go compilers. I've had success with the compiler for versions 1.15.5 and 1.18.1, but haven't with 1.19.5. I don't know what's exactly happening, but I'm not going to try and figure it out right now since all I need currently are features in compiler versions >= 1.18. Lastly, I actually had basically the same problems with python3.7, and these changes fixed these problems too...

How to create C-extension/embed Python with MinGW-w64 on Windows

Is it possible (and how) to use MinGW-w64 for building of C-extensions for Python or embeding Python on Windows?
Let's take as example the following cython-extension foo.pyx:
print("foo loaded")
from which the C-code can be generated either via cython -3 foo.pyx or cython -3 --embed foo.pyx if interpreter should be embedded.

While mingw-w64-compiler is not really supported (the only supported windows compiler is MSVC), it can be used to create C-extensions or to embed Python. There are however no guarantee, this won't break in the future versions.
distutils does not support mingw-w64, so there is no gain in setting up a setup.py-file - the steps must be performed manually.
First we need some information usually provided by distutils:
Headers: We need the path to the Python includes. For a way to find them see this SO-post.
DLL: mingw-w64's linker works differently than MSVC's: python-dll and not python-lib is needed. So we need the path to the pythonXY.dll which is usually next the the python.exe.
Once the C-code is created/generated, the extension can be build via
x86_64-w64-mingw32-gcc -shared foo.c -DMS_WIN64 -O2 <other_options> -I <path_to_python_include> -L <path_to_python_dll> -lpython37 -o foo.pyd
The important details are:
it is probably Ok to use only use -O2 for optimization and leave <other_options> empty-
It is important to define MS_WIN64-macro (e.g. via -DMS_WIN64). In order to build for x64 on windows it must be set, but it works out of the box only for MSVC (defining _WIN64 could have slightly different outcomes):
#ifdef _WIN64
#define MS_WIN64
#endif
if it is not done, at least for files generated by Cython the following error message will be generated by the compiler:
error: enumerator value for ‘__pyx_check_sizeof_voidp’ is not an integer constant
201 | enum { __pyx_check_sizeof_voidp = 1 / (int)(SIZEOF_VOID_P == sizeof(void*)) };
pyd is just a dll in disguise, thus we need the -shared option, which means a dynamic library (i.e. shared-object in Linux-world) will be created.
It is important, that the python-library (pythonXY) should be the dll itself and not the lib (see this SO-post). Thua we use the path to pythonXY.dll (in my case python37) and not pythonXY.lib, as it would be the case for MSVC.
One probably should add the proper suffix to the resulting pyd-file, I use the old convention for simplicity here.
Embeded Python:
In this case an executable should be build (e.g. the C-file is generated by Cython with --embed option: cython -3 --embed foo.pyx) and thus the command line looks as follows:
x86_64-w64-mingw32-gcc foo.c -DMS_WIN64 -O2 <other_options> -I <path_to_python_include> -L <path_to_python_dll> -lpython37 -o foo.exe -municode
There are two important differences:
-shared should no longer be used, as the result is no longer a dynamic library (that is what *.pyd-file is after all) but an executable.
-municode is needed, because for Windows, Cython defines int wmain(int argc, wchar_t **argv) instead of int main(int argc, char** argv). Without this option, an error message like
/build/mingw-w64-_1w3Xm/mingw-w64-4.0.4/mingw-w64-crt/crt/crt0_c.c:18: undefined reference to 'WinMain'
collect2: error: ld returned 1 exit status
would appear (see this SO-post for more information).
Note: for the resulting executable to run, a whole python-distribution (and not only the dll) is needed (see also this SO-post), otherwise the resulting executable will abort with error (either the python dll wasn't found or the python installation or the site packages - depending on the configuration of the machine on which the exe has to run).
mingw-w64 can also be used on Linux for cross-compilation for Windows, see this SO-post.

Python.h compiling error: collect2.exe Id returned 1 exit status

So, I've searched all around the web and I couldn't find anything.
I'm started with python, and I made a TKinter application. At first, I tried to compile it to an .exe file, but that didn't work. So now I'm embedding python in C++. Every single time I try to compile it (using Dev-C++), I get the error:
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x10): undefined reference to `__imp_Py_Initialize'
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x25): undefined reference to `__imp_PyRun_SimpleStringFlags'
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x2e): undefined reference to `__imp_Py_Finalize'
F:\Documents\Videos\[program-name]\program\collect2.exe [Error] ld returned 1 exit status
F: stands for the flash drive.
In the Dev-c++ options, I added:
-Wall -I\C:\Users\*****\AppData\Local\Programs\Python\Python35\include
So this is the full command:
g++.exe "F:\Documents\Videos\[program-name]\program\[program-name].cpp" -o "F:\Documents\Videos\[program-name]\program\[program-name].exe" -Wall -I\C:\Users\*****\AppData\Local\Programs\Python\Python35\include -I"C:\Programma's\Dev-C++\MinGW64\include" -I"C:\Programma's\Dev-C++\MinGW64\x86_64-w64-mingw32\include" -I"C:\Programma's\Dev-C++\MinGW64\lib\gcc\x86_64-w64-mingw32\4.9.2\include" -I"C:\Programma's\Dev-C++\MinGW64\lib\gcc\x86_64-w64-mingw32\4.9.2\include\c++" -I"C:\Users\*****\AppData\Local\Programs\Python\Python35\include" -L"C:\Programma's\Dev-C++\MinGW64\lib" -L"C:\Programma's\Dev-C++\MinGW64\x86_64-w64-mingw32\lib" -static-libgcc
My question is: how do I deal with this?
And please note: I haven't got administrator privileges, I'm just a standard user.
Python version: 3.5.1
Dev-C++ version: 5.11
GCC version: I don't know, but I'll find it out if needed :)
EDIT: Because of the duplicate mark by NathanOliver, I'll ask it like this: which command line variables do I have to add to compile it?

The linker is complaining bacause it can't find a group of references about some python stuff, as a matter of fact I don't see any python library in the g++ invocation.
You need to had something like this -lpython3.5.1 to tell g++ to link against the python library (assuming python is installed in your system, else you will have to add the path to the library using the -L option).
More resources on g++ makefiles and linking:
https://www3.ntu.edu.sg/home/ehchua/programming/cpp/gcc_make.html

Using libspotify .dll/.lib files in MinGW32 compiling pySpotify

Using MinGW32 on a Windows PC I am trying to compile pySpotify. The first error was that libspotify/api.h was missing. I fixed this by copying the appropriate folder from libspotify into C:\MinGW\include. However now dllwrap is now failing with ld linking. The binaries Spotify distribute are libspotify.dll and libspotify.lib. No matter where I put them (pySpotify folder/subfolders, temp build folder/subfolders and MinGW folder/subfolders) or what I name them (.a, .o & .so) it still shows the same error messages.
The pertinent error is:
C:\MinGW\bin\dllwrap.exe -mdll -static --output-lib build\temp.win32-2.7\Release\src\lib_spotify.a --def build\temp.win32-2.7\Release\src\_spotify.def -s build\temp.win32-2.7\Release\src\module.o build\temp.win32-2.7\Release\src\session.o build\temp.win32-2.7\Release\src\link.o build\temp.win32-2.7\Release\src\track.obuild\temp.win32-2.7\Release\src\album.o build\temp.win32-2.7\Release\src\albumbrowser.o build\temp.win32-2.7\Release\src\artist.o build\temp.win32-2.7\Release\src\artistbrowser.o build\temp.win32-2.7\Release\src\search.o build\temp.win32-2.7\Release\src\playlist.o build\temp.win32-2.7\Release\src\playlistcontainer.o build\temp.win32-2.7\Release\src\playlistfolder.o build\temp.win32-2.7\Release\src\image.o build\temp.win32-2.7\Release\src\user.o build\temp.win32-2.7\Release\src\pyspotify.o build\temp.win32-2.7\Release\src\toplistbrowser.o -LC:\Python26\libs -LC:\Python26\PCbuild -lspotify -lpython26 -lmsvcr90 -o build\lib.win32-2.7\spotify\_spotify.pyd
c:/mingw/bin/../lib/gcc/mingw32/4.7.2/../../../../mingw32/bin/ld.exe: cannot find -lspotify
collect2.exe: error: ld returned 1 exit status
dllwrap: gcc exited with status 1
error: command 'dllwrap' failed with exit status 1
bok says on github that:
You need to add the API headers in the include path (add something like -I~\libspotify\include to your compiler options) and the shared library in the library path (add -L~\libspotify\lib to your linker options). This will allow the compiler to find the necessary include files, and your linker to find the necessary binary objects.
However the distutils Extension class seems to be deprecated and hard to find documentation for (I believe this is where custom compiler options need to go). Appreciate that ~ may need changed to %USERPROFILE% or similar. Similarly %PYTHONPATH%\Lib\distutils\distutils.cfg has little documentation beyond the [build] compiler=mingw32 stanza. This makes editing the compile/link commands and their options impossible to change.
How do you compile pySpotify on Windows?
EDIT:
By using Python 2.6 and copying libspotify.dll/libspotify.lib to C:\Python26\PCbuild and renaming them to spotify.dll/libspotify.lib I now receive another error message from ld:
C:\MinGW\bin\dllwrap.exe -mdll -static --output-lib build\temp.win32-2.6\Release\src\lib_spotify.a --def build\temp.win32-2.6\Release\src\_spotify.def -s build\temp.win32-2.6\Release\src\module.o build\temp.win32-2.6\Release\src\session.o build\temp.win32-2.6\Release\src\link.o build\temp.win32-2.6\Release\src\track.obuild\temp.win32-2.6\Release\src\album.o build\temp.win32-2.6\Release\src\albumbrowser.o build\temp.win32-2.6\Release\src\artist.o build\temp.win32-2.6\Release\src\artistbrowser.o build\temp.win32-2.6\Release\src\search.o build\temp.win32-2.6\Release\src\playlist.o build\temp.win32-2.6\Release\src\playlistcontainer.o build\temp.win32-2.6\Release\src\playlistfolder.o build\temp.win32-2.6\Release\src\image.o build\temp.win32-2.6\Release\src\user.o build\temp.win32-2.6\Release\src\pyspotify.o build\temp.win32-2.6\Release\src\toplistbrowser.o -LC:\Python26\libs -LC:\Python26\PCbuild -lspotify -lpython26 -lmsvcr90 -o build\lib.win32-2.6\spotify\_spotify.pyd
_spotify.exp: file not recognized: File format not recognized
collect2.exe: error: ld returned 1 exit status
dllwrap: gcc exited with status 1
error: command 'dllwrap' failed with exit status 1

Not having access to a mingw installation at the moment, I can suggest a few things.
First, ld is known to be picky about order of arguments. Strangely, when I googled for "ld argument order", I got a bunch of pages suggesting that the order doesn't matter, but I have been burned several times by this. I've had the most success with the following argument order:
Switches to ld (ie, -Wall)
Library search paths (ie, -LPATH)
Object files
Libraries (ie, -lspotify)
I noticed in your linker output some references to amd64. I'm not sure how you compiled the other object files that you have, but since libspotify is 32-bit on Windows, I'm guessing that mixing 32/64 bit here is not going to work out too well.
The last thing I can think of is that perhaps the dll extension is confusing to ld, have you tried changing the filename to libspotify.so? I know this is kind of a shot in the dark, but otherwise I'm not sure how to help you further.

I guess you are trying to link a 64bit version of pyspotify against a 32bit version of libspotify. Try to rebuild pyspotify adding the -m32 to CFLAGS and CXXFLAGS

I've added some instructions on how I got it to compile on the actual ticket for the issue. Let me Know if it helped you!
https://github.com/mopidy/pyspotify/issues/63

How to link C lib against python for embedding under Windows?

I am working on an application written in C. One part of the application should embed python and there is my current problem. I try to link my source to the Python library but it does not work.
As I use MinGW I have created the python26.a file from python26.lib with dlltool and put the *.a file in C:/Program Files (x86)/python/2.6/libs.
Therefore, I compile the file with this command:
gcc -shared -o mod_python.dll mod_python.o "-LC:\Program Files (x86)\python\2.6\libs" -lpython26 -Wl,--out-implib,libmod_python.a -Wl,--output-def,mod_python.def
and I get those errors:
Creating library file: libmod_python.a
mod_python.o: In function `module_init':
mod_python.c:34: undefined reference to `__imp__Py_Initialize'
mod_python.c:35: undefined reference to `__imp__PyEval_InitThreads'
... and so on ...
My Python "root" folder is C:\Program Files (x86)\python\2.6
The Devsystem is a Windows Server 2008
GCC Information: Reading specs from C:/Program Files (x86)/MinGW/bin/../lib/gcc/mingw32/3.4.5/specs
Configured with: ../gcc-3.4.5-20060117-3/configure --with-gcc --with-gnu-ld --with-gnu-as --host=mingw32 --target=mingw32 --prefix=/mingw --enable-threads --disable-nls --enable-languages=c,c++,f77,ada,objc,java --disable-win32-registry --disable-shared --enable-sjlj-exceptions --enable-libgcj --disable-java-awt --without-x --enable-java-gc=boehm --disable-libgcj-debug --enable-interpreter --enable-hash-synchronization --enable-libstdcxx-debug
Thread model: win32
gcc version 3.4.5 (mingw-vista special r3)
What I do wrong? How I get it compiled and linked :-)?
Cheers, gregor
Edit:
I forgot to write information about my Python installation: It's the official python.org installation 2.6.1
... and how I created the python.a file:
dlltool -z python.def --export-all-symbols -v c:\windows\system32\python26.dll
dlltool --dllname c:\Windows\system32\python26.dll --def python.def -v --output-lib python26.a

Well on Windows the python distribution comes already with a libpython26.a in the libs subdir so there is no need to generate .a files using dll tools.
I did try a little example with a single C file toto.c:
gcc -shared -o ./toto.dll ./toto.c -I/Python26/include/ -L/Python26/libs -lpython26
And it works like a charm. Hope it will help :-)

Python (at least my distribution) comes with a "python-config" program that automatically creates the correct compiler and linker options for various situations. However, I have never used it on Windows. Perhaps this tool can help you though?

IIRC, dlltool does not always work. Having python 2.6 + Wow makes things even more less likely to work. For numpy, here is how I did it. Basically, I use obdump.exe to build the table from the dll, which I parse to generate the .def. You should check whether your missing symbols are in the .def, or otherwise it won't work.