I'm trying to use pybind11 for the first time by following the tutorial documentation.
As the documentation says, I created the following example.cpp file with the following content:
#include <pybind11/pybind11.h>
int add(int i, int j) {
return i + j;
}
PYBIND11_MODULE(example, m) {
m.doc() = "pybind11 example plugin"; // optional module docstring
m.def("add", &add, "A function which adds two numbers");
}
And since I'm on macOS, based on this tutorial, I compiled the example file with the following command: (not sure if this is correct but seems to work)
c++ -O3 -Wall -shared -std=c++11 -undefined dynamic_lookup -I /usr/local/Cellar/pybind11/2.4.3/include example.cpp -o example`python3-config --extension-suffix --includes`
This command generates the example.cpython-37m-darwin.so file in the same directory.
And then I created example.py with the following content:
import example
example.add(1, 2)
But when I run the script with python example.py, it shows the following error:
Traceback (most recent call last):
File "example.py", line 1, in <module>
import example
File "/Users/cuinjune/pybind11/build/example.py", line 2, in <module>
example.add(1, 2)
AttributeError: 'module' object has no attribute 'add'
What is wrong and how to fix this?
EDIT: Previously, it somehow generated example.pyc file but I don't know how the file was generated and now it just says: ImportError: No module named example.
This line:
/Users/cuinjune/pybind11/build/example.py
says that there is (or was) another file in your work directory (or perhaps /Users/cuinjune/pybind11/build is in your PYTHONPATH) that is a pure Python file, and unrelated to your built example.cpython-37m-darwin.so. When that example.py was imported, presumably some time in the past, the example.pyc you saw was generated. The .pyc remains fully functional standalone, even when the originating example.py was removed. The .pyc file will have the full directory to the original file hardcoded in it, and thus will continue to refer to that file, even when it is removed.
From your update, now all other example.py, .pyc, .pyo are gone, right?
First make sure that the python interpreter you are running matches the tag that you used to build. For example by running:
python3 -c 'import distutils.sysconfig as ds; print(ds.get_config_var("EXT_SUFFIX"))'
(where I'm assuming you used python3 as the command to start the Python interpreter; if not, then change it to the relevant command).
Second, verify the PYTHONPATH envar to see whether the local directory (either fully spelled out, or .) is in it. If not, add it.
With those two covered, it should work ...
Personally, however, I'd first simplify my life by renaming the module to something more unique (rename both example in the .cpp as well as the target on the CLI).
Related
I'm on MacOs and I'm trying to embed python inside a c shared library. Disclaimer: there are two installations of python involved with this question. One is my "main" python, so to speak, installed at /Library/Frameworks/Python.framework/Versions/3.10/bin/python3, that will be referred as "main python"; and the other one is a subdirectory of my current working directory containing python source code that i will build and embed, and it will be referred as embedded python.
For clarity, I have reproduced a minimalistic example of my problem that gives the same results.
Consider that the current working directory contains:
python310 (a directory containing python source code)
I configure it using the following command:
./configure --prefix=__path_to_the_directory_containing_python_source__ --exec-prefix=__path_to_the_directory_containing_python_source__ --enable-shared --enable-optimizations --with-lto
and compiled it using make && make altinstall
I have used altinstall because I will use this distribution only inside my application and I didn't want to replace my normal python installation.
test_interpreter.c:
#include "__path_to_the_directory_containing_python_source__/Include/Python.h"
int test() {
Py_SetProgramName(L"__path_to_the_directory_containing_python_source__/python.exe");
Py_Initialize();
// adds the current directory to sys.path to make test_import.py importable
PyObject* sysPath = PySys_GetObject("path"); // borrowed reference
PyObject* cur_dir = PyUnicode_FromString(""); // new reference
PyList_Append(sysPath, cur_dir);
PyRun_SimpleString("import test_import");
Py_DECREF(cur_dir);
Py_Finalize();
return 0;
}
test_import.py:
# will be called by embedded python
print("Inside python")
...
test_interpreter.py
# will be called by main python
import ctypes
lib = ctypes.CDLL("/__path_to_current_directory__/libtest.so")
tp = ctypes.CFUNCTYPE(ctypes.c_int)
test = tp(("test", lib))
test()
Now the problem is that, when test_import.py imports some builtin modules (note it doesn't happen with every module but just some), i get errors like segfault (e.g. when importing ctypes) or abort trap (e.g. when importing sqlite3). The most interesting part is that the errors do not occur if the interpreter is embedded inside an executable instead than a shared library.
So, if I compile test_interpreter.c into libtest.so using: gcc -L__path_to_the_directory_containing_python_source__ -lpython3.10 -shared -install_name __current_directory_path__/libtest.so -o libtest.so test_interpreter.c,
then modify test_import.py to for example
# will be called by embedded python
print("Inside python")
import decimal
and execute python3 test_interpreter.py (using main python; version is still 3.10) i get:
Inside python
Segmentation fault: 11
Other modules that give me the same error message are:
tkinter
ctypes
Also, if it can be usefull, I managed to understand that when importing ctypes inside the embedded interpreter the error occures when the line from _ctypes import Union, Structure, Array (of ctypes' __ init __.py) is executed.
If i modify test_interpreter.py to:
print("Inside python")
import sqlite3
and run the same command i get:
Inside python
Python(86563,0x10a7105c0) malloc: *** error for object 0x101d56360: pointer being freed was not allocated
Python(86563,0x10a7105c0) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6
Other modules that give me the same error message are:
dbm
distutils
random
json
zoneinfo
base64
csv
calendar
pickle
Note that if I compile test_interpreter.c as an executable (after changing test function name to main) using gcc -L__path_to_the_directory_containing_python_source__ -o test test_interpreter.c, or if I run the python executable (without embedding it) and try to import those module, everything works fine.
Thanks in advance to everyone who will try to understand what's going on.
I'm using some example code I found on the web to try to set up libboost so I can call into cpp routines with Python code. (I intend to use python to write my UI and cpp for my backend for this application) Boost seems simple enough to use, but it's currently not exposing any functionality.
#include <boost/python.hpp>
char const* greet()
{
return "hello, world";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
I compile this using the line g++ -c hello.cpp -I/usr/include/python3.6/
(that last include is necessary because I'm on ubuntu, where g++ doesn't locate python correctly, and I'm too lazy to add it to my path)
import hello_ext
print(hello_ext.greet())
I run this using python3, and I get the following output
File "hello.py", line 1, in <module>
import hello_ext
ModuleNotFoundError: No module named 'hello_ext'
This implies to me that Boost is not properly exposing the C++ functionality I created a module for. What am I missing here? I've already tried exposing the functionality to python in a header file instead of in the cpp file, and that has the same result.
Also, if anyone looking at this post is having issues accessing functionality within their module, but it seems like the module is being exposed, make sure python doesn't already have a default module with the same name which would take precedence over your module.
According to boost::python docs your code should be compiled as a shared library to be used in python:
g++ hello.cpp -I /usr/include/python3.6 -lboost_python-py36 -shared -fPIC -o hello_ext.so
Note that the name of the shared library must be the same as the name of your python module. You also forgot to link your code with boost lib.
This question already has an answer here:
Are executables produced with Cython really free of the source code?
(1 answer)
Closed 2 years ago.
I'm experimenting with Cython and possibilities of code obfuscation (article). In that article especially noted:
When the compilation is done there’s no way to reverse compiled libraries back to readable Python source code!
I use this question info to compile my code in standalone executable.
In my understanding and as mentioned in article 1, Cython translates Python code into C code, with correspond calls of Python library (is this correct?). In other wolds, we have only C file as output, and it can't be de-compiled back like .pyc files.
My test code is very simple:
def my_crashing_function():
x = "1"
y = 2
test = x + y # we will crash here
if __name__ == "__main__":
my_crashing_function()
But when I run this executable (after cython --embed -o test.c main.py and gcc -Os -I /usr/include/python3.5m -o test test.c -lpython3.5m -lpthread -lm -lutil -ldl -s)
I get error like this:
user#debian:~# ./hello
Traceback (most recent call last):
File "main.py", line 7, in init main
my_crashing_function()
File "main.py", line 4, in main.my_crashing_function
test = x + y # we will crash here
TypeError: Can't convert 'int' object to str implicitly
As you see, we have traceback with all method names, correct variable names and lines, even with original source code comments. If we rename variable or change comment - it will be also changed in traceback.
I don't understand how traceback can display all this info. It can work that way only if the full source code is also stored in the executable file.
Please explain me, where I'm wrong?
Update. Traceback in my situation was generated from original .py file. This file was in the same folder as compiled executable, and only because of this I got all source code and comments in traceback. After deletion of original .py file traceback will contain only exception type and line numbers, without other info.
No, it does not embed the code. It relies on being able to find the .pyx file - if you move that file then you will get a traceback but without the original code.
If you inspect the generated C source you'll find that the error handling routine goes through __Pyx_AddTraceback, then __Pyx_CreateCodeObjectForTraceback, which creates a PyCodeObject linked to your .pyx source file.
Under some circumstances (I'm not sure what though) it links to your .c file instead. The same rules will apply though - if it can't find the source it won't show that line.
Even without the .pyx file you will still get a traceback with useful method names - these are preserved in the compiled executable.
First I'm on Mac OSX 10.12.6.
I want to embed a python environment in my C application.
I take the github python project at https://github.com/python/cpython and manage to compile it with it's configure file and the command lines :
env LINKFORSHARED="-Wl,-stack_size,1000000 -framework CoreFoundation"
LDFLAGS="-Wl,-syslibroot,$SDK -arch i386"
CFLAGS="-Os -isysroot $SDK -arch i386"
./configure MACOSX_DEPLOYMENT_TARGET=10.6 --disable-shared --prefix=$PYTHON_PATH --exec-prefix=$PYTHON_PATH -build=i386-darwin
make
make altinstall
($SDK point to MacOSX.sdk inside Xcode.app)
it gives me a libPython.a for 32bits then I redo it for 64bits and merge both with
lipo -create libPython32.A libPython64.a libPython32-64.a
In my XCode project, on .xconfig file, I import the lib and headers files with :
OTHER_LDFLAGS = $(inherited) -lpython32-64;
USER_HEADER_SEARCH_PATHS = $(inherited) $(LIBS)/cpython35/include
LIBRARY_SEARCH_PATHS = $(inherited) $(LIBS)/cpython35/lib;
Everything is found and my project compiles, no problem seen.
And then in my python.c file (I reduced my path for this article, that's why '...') :
wchar_t *stdProgramName = L".../LIBs/cpython35/bin/python3.5";
Py_SetProgramName(stdProgramName);
wchar_t *stdPythonHome = L".../LIBs/cpython35";
Py_SetPythonHome(stdPythonHome);
wchar_t *stdlib = L".../LIBs/cpython35/lib/python3.5.zip:.../LIBs/cpython35/lib/python3.5:.../LIBs/cpython35/lib/python3.5/plat-darwin:.../LIBs/cpython35/lib/python3.5/lib-dynload:.../LIBs/cpython35/lib/python3.5/site-packages";
Py_SetPath(stdlib);
Py_Initialize();
// Run something
PyRun_SimpleString("import sys; print(sys.path)");
//To this line it's work fine, all path are correcte but then
PyRun_SimpleString("import math;");
Py_Finalize();
the import of the math librarie does not work, it gives me :
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: dlopen(.../LIBs/cpython35/lib/python3.5/lib-dynload/math.cpython-35m-darwin.so, 2): Symbol not found: _PyExc_MemoryError
Referenced from: .../LIBs/cpython35/lib/python3.5/lib-dynload/math.cpython-35m-darwin.so
Expected in: flat namespace
in .../LIBs/cpython35/lib/python3.5/lib-dynload/math.cpython-35m-darwin.so
I'm kinda stuck at this point searching a solution for this import !!!
In the end, I didn't need my python lib to be in a static library
So '--disable-shared' was the problem, I switched it with '--enable-shared'.
With this change, I get a libpython32.dylib, and that lib can access math.cpython-35m-darwin.so...
My process didn't change that much. In my project, I deleted -lpython32-64 in OTHER_LDFLAGS, and include libpython.dylib to "Link Binary With Libraries".
And it works.
So I'd like to call some python code from c via cython. I've managed to call cython code from c. And I can also call python code from cython. But when I add it all together, some things are missing.
Here is my python code (quacker.pyx):
def quack():
print "Quack!"
Here is my cython "bridge" (caller.pyx):
from quacker import quack
cdef public void call_quack():
quack()
And here is the c code (main.c):
#include <Python.h>
#include "caller.h"
int main() {
Py_Initialize();
initcaller();
call_quack();
Py_Finalize();
return 0;
}
When I run this I get this exception:
Exception NameError: "name 'quack' is not defined" in 'caller.call_quack' ignored
The missing pieces I'm suspecting:
I haven't called initquacker()
I haven't included quacker.h
Cython didn't produce any quacker.h - only quacker.c
caller.c doesn't import quacker.h or call initquacker()
I'm not really sure that it's even possible to do what I'm trying to do, but it seems to me that it ought to be. I'd love to hear any input you might have.
Edit:
This is how I cythonize / compile / link / run:
$ cython *.pyx
$ cc -c *.c -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
$ cc -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl *.o -o main
$ ./main
If you rename the quacker.pyx to quacker.py, everything is actually correct. The only problem is that your program won't search for python modules in the current directory, resulting in the output:
Exception NameError: "name 'quack' is not defined" in 'caller.call_quack' ignored
If you add the current directory to the PYTHONPATH environment variable however, the output becomes the one you'd expect:
$ PYTHONPATH=".:$PYTHONPATH" ./main
Quack!
When running the python shell, according to the documentation the current directory (or the directory containing the script) is added to the sys.path variable automatically, but when creating a simple program using Py_Initialize and Py_Finalize this does not seem to happen. Since the PYTHONPATH variable is also used to populate the sys.path python variable, the workaround above produces the correct result.
Alternatively, below the Py_Intialize line, you could add an empty string to sys.path as follows by just executing some python code, specified as a string:
PyRun_SimpleString("import sys\nsys.path.insert(0,'')");
After recompiling, just running ./main should then work.
Edit
It's actually interesting to see what's going on if you run the code as specified in the question, so without renaming the quacker.pyx file. In that case, the initcaller() function tries to import the quacker module, but since no quacker.py or quacker.pyc exists, the module cannot be found, and the initcaller() function produces an error.
Now, this error is reported the python way, by raising an exception. But the code in the main.c file doesn't check for this. I'm no expert in this, but in my tests adding the following code below initcaller() seemed to work:
if (PyErr_Occurred())
{
PyErr_Print();
return -1;
}
The output of the program then becomes the following:
Traceback (most recent call last):
File "caller.pyx", line 1, in init caller (caller.c:836)
from quacker import quack
ImportError: No module named quacker
By calling the initquacker() function before initcaller(), the module name quacker already gets registered so the import call that's done inside initcaller() will detect that it's already loaded and the call will succeed.
In case there's anyone wondering how would it work in Python 3, here's my solution after struggling a bit as a Cython newbie.
main.c
#include <Python.h>
#include "caller.h"
int
main()
{
PyImport_AppendInittab("caller", PyInit_caller);
Py_Initialize();
PyImport_ImportModule("caller");
call_quack();
Py_Finalize();
return 0;
}
caller.pyx
# cython: language_level=3
import sys
sys.path.insert(0, '')
from quacker import quack
cdef public void call_quack():
quack()
quacker.py
def quack():
print("Quack!")
Finally, here's the Makefile that compiles everything:
target=main
cybridge=caller
CC=gcc
CFLAGS= `python3-config --cflags`
LDFLAGS=`python3-config --ldflags`
all:
cython $(cybridge).pyx
$(CC) $(CFLAGS) -c *.c
$(CC) $(LDFLAGS) *.o -o $(target)
clean:
rm -f $(cybridge).{c,h,o} $(target).o $(target)
rm -rf __pycache__
Maybe this is not what you want but I got it working by the following changes:
in quacker.pyx I added
cdef public int i
To force Cython to generate the .h file.
An then in the main:
#include <Python.h>
#include "caller.h"
#include "quacker.h"
int main() {
Py_Initialize();
initquacker();
initcaller();
call_quack();
Py_Finalize();
return 0;
}
I needed to do this using CMake and ended up recreating this sample. You can find the repository with complete working example here.
You can build and run the example using either Docker on the CLI or a Visual Studio devcontainer.