I currently have a source files written in C++ and wrapped into a Python module (uses Boost and Parser libs).
I ported the C++ folder under go/src and .so files along with main.go
Program Structure
src
/main
main.go
network.so
/network
file1.cpp (this has a function **object DBdata::getTable())
file1.hpp (#define FILE1_H_)
main.go
package main
// #cgo pkg-config: python3
// #cgo CFLAGS : -I./ -I/usr/include/python3.6
// #cgo LDFLAGS: -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6 -lpthread -ldl -lutil -lm
// #include <Python.h>
import "C"
import "fmt"
func main() {
cmd := exec.Command("python", "-c", "import network; network.getTable()")
cmd.Dir = "/home/username/go/src/network" //directory where is my python code
out,err := cmd.CombinedOutput()
}
After building this main.go, I get the error as
/usr/include/boost/config/no_tr1/memory.hpp: fatal error: memory: No
such file or directory
> # include
> ^~~~~~~~
> compilation terminated.
How to import .so as Python modules in Go?
Can Swig be used in this place?
What is the better approach to expose Python module to Go?
Issue fixed. The .so placed in /bin and the Go was build and could access the functions under network
Related
So I am trying to make a C/C++ dll for a project but any C++ library I include in my header, ends up causing loading problems when I try to load it using ctypes in python. I'm guessing maybe ctypes doesn't have c++ libs paths included? I made a simple demonstration to my problem.
init2.h: Generic Header file for dll and execs using the dll
#ifndef INIT2_H
#define INIT2_H
#ifdef __cplusplus
extern "C" {
#endif
#include <stdio.h>
void func11();
#ifdef __cplusplus
}
#endif
#endif
init2.cc: dll code (loads fine)
// This works fine
#include "init2.h"
void func11() {
printf("Func11\n");
}
init.c: exe code that loads dll
#include "init2.h"
int main() {
func11();
return 0;
}
Makefile
CFLAGS = -Wall -std=c11 -g
CCFLAGS = -Wall -std=c++11 -g
all: libmain.dll main
main.o: init2.cc
g++ $(CCFLAGS) init2.cc -fpic -c -o main.o
libmain.dll: main.o
g++ $(CCFLAGS) -shared main.o -o libmain.dll
main: init.c
gcc $(CFLAGS) -L. -lmain init.c -o main
clean:
del *.o *.exe *.dll *.txt
script.py
from ctypes import *
lib = CDLL('D:\Projects\Learning-C++\libmain.dll')
lib.func11()
The above compiles, links and loads fine in the generated main.exe and script.py. The problem occurs when I add a c++ lib in my dll.
init2.cc: With iostream (loads in main.exe only)
#include "init2.h"
#include <iostream>
void func11() {
std::cout << "Func11\n";
}
Compiling that into the dll, while everything stays the same loads fine into the main.exe (init.c code) but gives me the following loading error in script.py.
Traceback (most recent call last):
File "D:\Projects\Learning-C++\script.py", line 3, in <module>
lib = CDLL('D:\Projects\Learning-C++\libmain.dll')
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'D:\Projects\Learning-C++\libmain.dll' (or one of its dependencies). Try using the full path with constructor syntax.
How can I fix this? My bad if these seems obvious, but I'm fairly new to C++ and haven't got the integration down yet. Thanks in advance!
The problem was as I expected due to cdll not able to find the C++ library. The file name for my case was libstdc++-6.dll. Which is located in the bin folder in my compiler's directory. To make cdll search for the dependencies in that folder I did the following:
import os
# add dependency directory
os.add_dll_directory('C:\\Program Files\\CodeBlocks\\MinGW\\bin')
# load the dll
lib = CDLL('D:\\Projects\\Learning-C++\\libmain.dll')
lib.func11()
The dll may have many dependencies located in other directories e.g. You defined libDLL.dll and bin/libDependency.dll. You will need to add the /path/to/dir/bin directory using the above method if you're trying to load libDLL.dll in python. You can find all the dependencies of a dll using dependency walker or
dumpbin /dependents libDLL.dll
in the VS dev powershell.
I'm currently trying to write a 'setup.py' script that when the user installs the python package automatically compiles my C++ extension bound with 'pybind11'. In Windows, I haven't got any problem making it happen with the 'VS19 MSVC' compiler. But I'm trying to make it happen if the user has installed 'MinGW-w64' instead.
These are the package files:
**main.cpp**
#include <pybind11/pybind11.h>
int add(int i, int j) {
return i + j;
}
namespace py = pybind11;
PYBIND11_MODULE(pybind11_example, m) {
m.def("add", &add);
}
**setup.py**
from setuptools import setup, Extension
import pybind11
ext_modules = [
Extension(
'pybind11_example',
sources = ['main.cpp'],
include_dirs=[pybind11.get_include()],
language='c++'
),
]
setup(
name='pybind11_example',
ext_modules=ext_modules
)
Having the two files in the same folder and running from the command prompt:
python setup.py build
If the user has VS19 MSVC compiler installed it successfully generates **pybind11_example.pyd** that can be tested to work running with python:
import pybind11_example as m
print(m.add(1, 2))
But if the user has a Mingw-w64 compiler installed raises an error saying that Visual Studio 2015 is required.
Note that I can easily compile **main.cpp** in to **pybind11_example.pyd** manually with Mingw-w64 running:
g++ -static -shared -std=c++11 -DMS_WIN64 -fPIC -I C:\...\Python\Python38\Lib\site-packages\pybind11\include -I C:\ ... \Python\Python38\include -L C:\ ... \Python\Python38\libs main.cpp -o pybind11_example.pyd -lPython38
Is there a way to write **setup.py** in a way that if the user has Windows with a MinGW-w64 compiler automatically compile **main.cpp** into **pybind11_example.pyd** when installing the package without needing to make it manually?
Chek the answer to this question. They try to solve the opposite case, force msvc instead of mingw, but the approach with setup.cfg might help you.
And here the answer demonstrates how to specify command line parameters depending on the choice made by setup tools: if it is msvc then one set of parameters, and another set for mingw.
I belive the second approach should suite your needs - whichever compier is installled, you have the proper command line to build your module.
I just recently discovered Boost.Python and I am trying to figure out how it works. I tried to go through the tutorial on the official website. However, I got
link.jam: No such file or directory
when running bjam as in the example (which appears to be just a warning),
and
Traceback (most recent call last):
File "hello.py", line 7, in <module>
import hello_ext
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
when running python hello.py.
I also tried to compile a module as described in another tutorial with similar results. I am running Ubuntu14.04 with boost1.55 compiled myself.
I tried to compile the following:
#include <boost/python.hpp>
char const* greet()
{
return "hello, world";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
with the following command from command line:
g++ -o hello_ext.so hello.cpp -I /usr/include/python2.7/ -I /home/berardo/boost_1_55_0/ -L /usr/lib/python2.7/ -L /home/berardo/boost/lib/ -lboost_python -lpython2.7 -Wl, -fPIC -expose-dynamic
which still gives me a:
/usr/bin/ld: impossibile trovare : File o directory non esistente
collect2: error: ld returned 1 exit status.
Finally, I was able to make it work. First, I fixed the linker issues, as suggested by Dan. It finally compiled but I still got:
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
The problem was that the python module was not able to load correctly so I needed to add another linker option. Here, I report the final Makefile:
# location of the Python header file
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = ${HOME}/boost/include
BOOST_LIB = ${HOME}/boost/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,-rpath,$(BOOST_LIB) -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr/lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).C
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).C
Notice the -Wl,-rpath, option, which apparently makes the new created shared library available to the python script.
#Dan: Thanks for the valuable hints.
I am trying to wrap c++ functions into python using swig. I am using following commands
swig -c++ -python helloworld.i
g++ -O2 -fPIC -c hello.cpp
g++ -O2 -fPIC -c helloworld_wrap.cxx -I//anaconda/include/python2.7
g++ -lpython -dynamclib hello.o helloworld_wrap.o -o _helloworld.so
with hello.cpp being initial file with functions and helloworld.i being file with wrapper. These commands creates the library helloworldbut I can only import it through default python in /usr/bin/python
If I try to import it through python installed through anaconda it gives following error:
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6
Can you tell me how can I wrap the codes with python from anaconda?
Found a solution : Python.h not found using swig and Anaconda Python
In above question, the top answer gives an explanation of using disutilsand a set up file in python to build the library. This works wonders :)
The next problem I am having for wrapping simple class:
my class code from [example] (http://web.mit.edu/svn/src/swig-1.3.25/Examples/python/class/index.html)
/* File : example.h */
class Shape {
public:
Shape() {
nshapes++;
}
virtual ~Shape() {
nshapes--;
};
double x, y;
void move(double dx, double dy);
virtual double area() = 0;
virtual double perimeter() = 0;
static int nshapes;
};
class Circle : public Shape {
private:
double radius;
public:
Circle(double r) : radius(r) { };
virtual double area();
virtual double perimeter();
};
class Square : public Shape {
private:
double width;
public:
Square(double w) : width(w) { };
virtual double area();
virtual double perimeter();
};
My setup.py file :
#setup.py file:
from setuptools import setup, Extension
setup(name='example',
version='0.1',
ext_modules=[Extension('_example', ['example.h', 'example.i'],
swig_opts=['-c++'],
)],
)
Code I am using to wrap :
python setup.py build_ext --inplace
Error message:
running build_ext
building '_example' extension
swigging example.i to example_wrap.cpp
swig -python -c++ -o example_wrap.cpp example.i
error: unknown file type '.h' (from 'example.h')
Can you suggest what is wrong here. I suppose it is not recognizing '.h' file but as it is header file, I thought it could be kept as it is. Also if my setup.py file is correct or not? I am just trying to follow example for simple wrapping, there is no simple tutorial online apparently.
I can also ask this question on other different question but thought just continuing here for now.
Answer by Warren Weckesser in similar question
. I used the setup.py file as suggested in the answer and added the path to the library to sys.path and it work wonders :)
Use the option -I/Users/myuser/anaconda/include/python2.7 in the gcc command. (That's assuming you are using python 2.7. Change the name to match the version of python that you are using.) You can use the command python-config --cflags to get the full set of recommended compilation flags:
$ python-config --cflags
-I/Users/myuser/anaconda/include/python2.7 -I/Users/myuser/anaconda/include/python2.7 -fno-strict-aliasing -I/Users/myuser/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
However, to build the extension module, I recommend using a simple setup script, such as the following setup.py, and let distutils figure out all the compiling and linking options for you.
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_example', sources=['example_wrap.c', 'example.c'])
setup(name='example', ext_modules=[example_module], py_modules=["example"])
Then you can run:
$ swig -python example.i
$ python setup.py build_ext --inplace
(Take a look at the compiler commands that are echoed to the terminal when setup.py is run.)
distutils knows about SWIG, so instead of including example_wrap.c in the list of source files, you can include example.i, and swig will be run automatically by the setup script:
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_example', sources=['example.c', 'example.i'])
setup(name='example', ext_modules=[example_module], py_modules=["example"])
With the above version of setup.py, you can build the extension module with the single command
$ python setup.py build_ext --inplace
Once you've built the extension module, you should be able to use it in python:
>>> import example
>>> example.fact(5)
120
If you'd rather not use the script setup.py, here's a set of commands that worked for me:
$ swig -python example.i
$ gcc -c -I/Users/myuser/anaconda/include/python2.7 example.c example_wrap.c
$ gcc -bundle -undefined dynamic_lookup -L/Users/myuser/anaconda/lib example.o example_wrap.o -o _example.so
Note: I'm using Mac OS X 10.9.4:
$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.3.0
Thread model: posix
Can I use cython to create a shared library with exported C functions that have python code as the core? Like wrapping Python with C??
It is to be used in plugins.
tk
Using Cython, you can write function declared as C ones with the cdef keyword (and public... important!), with Python inner code:
yourext.pyx
cdef int public func1(unsigned long l, float f):
print(f) # some python code
Note: in the following is assumed that we are working in the root of drive D:\
Building (setup.py)
from distutils.core import setup
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
name = 'My app',
ext_modules = cythonize("yourext.pyx"),
)
Then run python setup.py build_ext --inplace
After running the setup.py (if you are using distutils), you'll get 2 files of interest:
yourext.h
yourext.c
Looking into the .c will show you that func1 is a C function, in the end.
Those two files are all we need to do the rest.
C main program for testing
// test.c
#include "Python.h"
#include "yourext.h"
main()
{
Py_Initialize(); // start python interpreter
inityourext(); // run module yourext
func1(12, 3.0); // Lets use shared library...
Py_Finalize();
}
As we don't use the extension (.pyd) by itself, we need to make a little trick/hack in the header file to disable the "DLL behavior". Add the following at the beginning of "yourext.h":
#undef DL_IMPORT # Undefines DL_IMPORT macro
#define DL_IMPORT(t) t # Redefines it to do nothing...
__PYX_EXTERN_C DL_IMPORT(int) func1(unsigned long, float);
Compiling "yourext" as a shared library
gcc -shared yourext.c -IC:\Python27\include -LC:\Python27\libs -lpython27 -o libyourext.dll
Then compiling our test program (linking to the DLL)
gcc test.c -IC:\Python27\include -LC:\Python27\libs -LD:\ -lpython27 -lyourext -o test.exe
Finally, run the program
$ test
3.0
This is not obvious, and there is many other ways to achieve the same thing, but this works (have a look to boost::python, ..., other solutions may better fit your needs).
I hope this answers a little bit your question or, at least, gave you an idea...