Develop Reference

Waf configure error, cannot find pyembed for some reason

I'm trying to configure Planck Likelihood Code using ./waf, but I keep getting the following error after calling ./waf configure CC=gcc python=/opt/local/bin/python cfitsio_lib=/opt/local/lib cfitsio_include=/opt/local/include --lapack_mkl=${MKLROOT} --install_all_deps:
Testing pyembed configuration: Could not build a python embedded interpreter
No suitable python distribution found
Cause: 'The configuration failed'
I have no idea how to proceed from here... Any suggestions would be greatly appreciated.
Here is the relevant part from the config.log file for the configuration attempt also. I'm using Python 2.7 on Mac OS, and it seems to be installed fine (as far as I can tell).
python-config
/opt/local/bin/python2.7-config
find program=['/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python-config', 'python2.7-config', 'python-config-2.7', 'python2.7m-config'] paths=['/Users/USER/Desktop/montepython_public-3.3/montepython/', '/opt/local/bin', '/opt/local/sbin', '/usr/local/bin', '/usr/bin', '/bin', '/usr/sbin', '/sbin', '/Library/TeX/texbin', '/Library/Apple/usr/bin'] var='PYTHON_CONFIG' -> ['/opt/local/bin/python2.7-config']
--------------------------------------------------------------------------
Asking python-config for pyembed '--cflags --libs --ldflags --embed' flags
['/opt/local/bin/python2.7-config', '--cflags', '--libs', '--ldflags', '--embed']
err: Usage: /opt/local/bin/python2.7-config [--prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--help]
not found
from /Users/USER/Desktop/code/plc_3.0/plc-3.01: The configuration failed
--------------------------------------------------------------------------
Asking python-config for pyembed '--cflags --libs --ldflags' flags
['/opt/local/bin/python2.7-config', '--cflags', '--libs', '--ldflags']
out: -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -fno-strict-aliasing -fno-common -dynamic -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
-lpython2.7 -lintl -ldl -framework CoreFoundation
-L/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -lintl -ldl -framework CoreFoundation
Unhandled flag '-pipe'
Unhandled flag '-Wall'
Unhandled flag '-Wstrict-prototypes'
yes
--------------------------------------------------------------------------
Testing pyembed configuration
==>
#include <Python.h>
#ifdef __cplusplus
extern "C" {
#endif
void Py_Initialize(void);
void Py_Finalize(void);
#ifdef __cplusplus
}
#endif
int main(int argc, char **argv)
{
(void)argc; (void)argv;
Py_Initialize();
Py_Finalize();
return 0;
}
<==
[1/2] Compiling [32mbuild/.conf_check_e02afcfd7f3c9a900cd99ee408c985e3/test.c[0m
['gcc', '-fno-strict-aliasing', '-fno-common', '-dynamic', '-Os', '-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7', '-DPYTHONDIR="/Users/USER/Desktop/code/plc_3.0/plc-3.01/lib/python2.7/site-packages"', '-DPYTHONARCHDIR="/Users/USER/Desktop/code/plc_3.0/plc-3.01/lib/python2.7/site-packages"', '../test.c', '-c', '-o/Users/USER/Desktop/code/plc_3.0/plc-3.01/build/.conf_check_e02afcfd7f3c9a900cd99ee408c985e3/testbuild/test.c.1.o']
[2/2] Linking [33mbuild/.conf_check_e02afcfd7f3c9a900cd99ee408c985e3/testbuild/testprog[0m
['gcc', '-arch', 'x86_64', 'test.c.1.o', '-o/Users/USER/Desktop/code/plc_3.0/plc-3.01/build/.conf_check_e02afcfd7f3c9a900cd99ee408c985e3/testbuild/testprog', '-L/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib', '-L/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config', '-lm', '-ldl', '-lutil', '-lpthread', '-lpython2.7', '-lintl', '-ldl', '-lpython2.7', '-lintl', '-ldl']
err: ld: library not found for -lintl
collect2: error: ld returned 1 exit status
from /Users/USER/Desktop/code/plc_3.0/plc-3.01: Test does not build: Traceback (most recent call last):
File "/Users/USER/Desktop/code/plc_3.0/plc-3.01/.waf-2.0.17-c5414c9b1eca70e1add79159e16494fe/waflib/Configure.py", line 325, in run_build
bld.compile()
File "/Users/USER/Desktop/code/plc_3.0/plc-3.01/.waf-2.0.17-c5414c9b1eca70e1add79159e16494fe/waflib/Build.py", line 176, in compile
raise Errors.BuildError(self.producer.error)
BuildError: Build failed
-> task in 'testprog' failed with exit status 1 (run with -v to display more information)
Could not build a python embedded interpreter
from /Users/USER/Desktop/code/plc_3.0/plc-3.01: The configuration failed

The linker did not find libintl.
Are you sure It is installed On your system? If yes the library must be installed in a standard library directory otherwise you must use -L"lib_path"

Linking error trying to compile C code embedded with Python

I'm trying to import and run this python function inside C code:
import requests
import json
url = 'http://127.0.0.1:5000/'
def verify():
code = input('Show-me the code: ')
r = requests.post(url, data=json.dumps({'code': code}))
return r.json()
Here is the C code:
#include <Python.h>
int main(void){
PyObject *myModuleString, *myModule, *myFunction, *myResult;
Py_Initialize();
myModuleString = PyUnicode_FromString((char*)"verify");
myModule = PyImport_Import(myModuleString);
myFunction = PyObject_GetAttrString(myModule,(char*)"verify");
myResult = PyObject_CallObject(myFunction, NULL);
const char* s = PyUnicode_AsUTF8(myResult);
printf("REPR: %s\n", s);
Py_Finalize();
return 0;
}
Create the object file works fine:
$ gcc -c `python3.7-config --cflags --ldflags` source_code.c
$
But it does not work at the end:
$ gcc source_code.o -o source_code.bin
/usr/bin/ld: source_code.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: final link failed: nonrepresentable section on output
collect2: error: ld returned 1 exit status
The -fPIE argument also does not help:
$ gcc -c `python3.7-config --cflags --ldflags` -fPIE source_code.c
$
$ gcc source_code.o -o source_code.bin
/usr/bin/ld: source_code.o: in function `main':
/home/user/source_code.c:7: undefined reference to `Py_Initialize'
/usr/bin/ld: /home/user/source_code.c:9: undefined reference to `PyUnicode_FromString'
/usr/bin/ld: /home/user/source_code.c:10: undefined reference to `PyImport_Import'
/usr/bin/ld: /home/user/source_code.c:12: undefined reference to `PyObject_GetAttrString'
/usr/bin/ld: /home/user/source_code.c:13: undefined reference to `PyObject_CallObject'
/usr/bin/ld: /home/user/source_code.c:15: undefined reference to `PyUnicode_AsUTF8'
/usr/bin/ld: /home/user/source_code.c:18: undefined reference to `Py_Finalize'
collect2: error: ld returned 1 exit status
How can I define the references to generate the binary file?

I had a similar error. Adding the -no-pie argument to the compile statement worked for me.
You could try this:
gcc -c `python3.7-config --cflags --ldflags` source_code.c -no-pie

Embedding Python in C: Error in linking - undefined reference to PyString_AsString

I am trying to embed a python program inside a C program. My OS is Ubuntu 14.04
I try to embed python 2.7 and python 3.4 interpreter in the same C code base (as separate applications). The compilation and linking works when embedding python 2.7 but not for the python 3.4. It fails during the linker stage.
Here is my C code (just an example not real code)
simple.c
#include <stdio.h>
#include <Python.h>
int main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pFunc, *pValue;
char module[] = "get_version";
char func[] = "get_version";
char module_path[] = ".";
Py_Initialize();
PyObject *sys_path = PySys_GetObject("path");
PyList_Append(sys_path, PyUnicode_FromString(module_path));
pName = PyUnicode_FromString(module);
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if(pModule != NULL)
{
pFunc = PyObject_GetAttrString(pModule, func);
if (pFunc && PyCallable_Check(pFunc))
{
pValue = PyObject_CallObject(pFunc, NULL);
if (pValue != NULL) {
printf("Python version: %s\n", PyString_AsString(pValue));
Py_DECREF(pValue);
}
else {
Py_DECREF(pFunc);
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr,"Call failed\n");
return 1;
}
}
}
Py_Finalize();
return 0;
}
get_version.py
import sys
def get_version():
version = '.'.join(str(v) for v in sys.version_info[:3])
print("version: ", version)
return version
I compile the program using gcc. First with compiling and linking flags set to python 2.7 I run the compilation and linking by using following command:
gcc `python-config --cflags` simple.c `python-config --ldflags`
The flags expand as:
python-config --cflags: -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
python-config --ldflags: -L/usr/lib/python2.7/config-x86_64-linux-gnu -L/usr/lib -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
It works fine without any issues. When I try to compile the same with python3.4 flags it fails:
gcc `python3-config --cflags` simple.c `python3-config --ldflags`
The flags expand as:
python-config --cflags: -I/usr/include/python3.4m -I/usr/include/python3.4m -Wno-unused-result -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
python-config --ldflags: -L/usr/lib/python3.4/config-3.4m-x86_64-linux-gnu -L/usr/lib -lpython3.4m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
Error message:
simple.c: In function ‘main’:
simple.c:27:17: warning: implicit declaration of function ‘PyString_AsString’ [-Wimplicit-function-declaration]
printf("Python version: %s\n", PyString_AsString(pValue));
^
simple.c:27:17: warning: format ‘%s’ expects argument of type ‘char *’, but argument 2 has type ‘int’ [-Wformat=]
/tmp/ccaoMdTo.o: In function `main':
/home/vagrant/c_python_api/simple.c:27: undefined reference to `PyString_AsString'
collect2: error: ld returned 1 exit status
I tried by changing the order in which linker objects are specified. But no luck. Any idea why this would be the case?
Thanks for the help!!

Python 3 does not have PyString_AsString any more; the Python 3 str correspond to Python 2 unicode objects; the names of the functions for handling str in Python 3 are PyUnicode_-prefixed in the C-API.
Thus this line:
printf("Python version: %s\n", PyString_AsString(pValue));
could be changed to use PyUnicode_AsUTF8 on Python 3:
#if PY_MAJOR_VERSION >= 3
printf("Python version: %s\n", PyUnicode_AsUTF8(pValue));
#else
printf("Python version: %s\n", PyString_AsString(pValue));
#endif
(Not that passing NULL to printf %s will have undefined behaviour, so you'd want to check that a non-NULL pointer was returned)

Python C binding error

I have written a Python API in C code and saved the file as foo.c.
Code:
#include <Python.h>
#include <stdio.h>
static PyObject *foo_add(PyObject *self, PyObject *args)
{
int a;
int b;
if (!PyArg_ParseTuple(args, "ii", &a, &b))
{
return NULL;
}
return Py_BuildValue("i", a + b);
}
static PyMethodDef foo_methods[] = {
{ "add", (PyCFunction)foo_add, METH_VARARGS, NULL },
{ NULL, NULL, 0, NULL }
};
PyMODINIT_FUNC initfoo()
{
Py_InitModule3("foo", foo_methods, "My first extension module.");
}
When i try to compile using the below mentioned command i am getting compilation error.
Command: gcc -shared -I/usr/include/python2.7 foo.c -o foo.so
Error:
gcc -shared -I/usr/include/python2.7 foo.c -o foo.so
/usr/bin/ld: /tmp/ccd6XiZp.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/tmp/ccd6XiZp.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
If i give compilation command with "-c" option, its getting compiled successfully and created the object file foo.so (This is the executable file).
I have to create a object file (without using -c option in compilation command) and import them in Python shell to verify it.
Please let me know what am i doing wrong here.

In your compilation flags you should include -fPIC to compile as position independent code. This is required for dynamically linked libraries.
e.g.
gcc -c -fPIC foo.c -o foo.o
gcc -shared foo.o -o foo
or in a single step
gcc -shared -fPIC foo.c -o foo.so

Python extension: symbol(s) not found for architecture x86_64 error

I want to write a python extension in c. I work on Mac, I took a code from here:
#include <Python.h>
static PyObject* say_hello(PyObject* self, PyObject* args)
{
const char* name;
if (!PyArg_ParseTuple(args, "s", &name))
return NULL;
printf("Hello %s!\n", name);
Py_RETURN_NONE;
}
static PyMethodDef HelloMethods[] =
{
{"say_hello", say_hello, METH_VARARGS, "Greet somebody."},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC
inithello(void)
{
(void) Py_InitModule("hello", HelloMethods);
}
I compile it:
gcc -c -o py_module.o py_module.c -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/
gcc -o py_module py_module.o -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/ -lm
But I get this error:
Undefined symbols for architecture x86_64:
"_PyArg_ParseTuple", referenced from:
_say_hello in py_module.o
"_Py_InitModule4_64", referenced from:
_inithello in py_module.o
"__Py_NoneStruct", referenced from:
_say_hello in py_module.o
"_main", referenced from:
start in crt1.10.6.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
make: *** [py_module] Error 1
How comes python doesn't support X86_64 architecture?

Two things:
You need to link your extension as a shared object (you're attempting to link an executable, which is why the linker is looking for main());
You need to link against the Python static library (-lpython).

Query the python env path with these commands:
$python-config --includes
-I/usr/include/python2.6 -I/usr/include/python2.6
$python-config --ldflags
-lpthread -ldl -lutil -lm -lpython2.6
generate .o file:
$ g++ -fPIC -c -I/usr/include/python2.6 -I/usr/include/python2.6 xx.cpp
generate .so file:
g++ -shared xx.o -o xx.so

Thanks to #NPE #glglgl and anatoly here is my Makefile:
DIR=/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/
CC=gcc
CFLAGS=-I$(DIR)
ODIR=.
LIBS_DIR=/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/
LIBS=-lpython2.7
_DEPS =
DEPS = $(patsubst %,$(IDIR)/%,$(_DEPS))
_OBJ = py_module.o
OBJ = $(patsubst %,$(ODIR)/%,$(_OBJ))
$(ODIR)/%.o: %.c $(DEPS)
$(CC) -c -o $# $< $(CFLAGS)
py_module: $(OBJ)
gcc -shared $^ $(CFLAGS) -I$(LIBS_DIR) $(LIBS) -o $#
.PHONY: clean
clean:
rm -f $(ODIR)/*.o *~ core $(INCDIR)/*~
makefile template had been taken from here.
In order to find the paths, one may use python-config --ldflags
and python-config --includes