All stackoverflow/github issues I've seen were about speeding up functions calls from Python in case of marshalling objects.
But my problem is about the working time of pure c++ function inside pybind11 C++ module function.
I have the training function that loads dataset and calls train method from the native C++ library class:
void runSvm()
// read smvlight file into required sparse representation
auto problem = read_problem( "random4000x20.train.svml" );
CSvmBinaryClassifierBuilder::CParams params( CSvmKernel::KT_Linear );
params.Degree = 3;
params.Gamma = 1/20;
params.Coeff0 = 0;
// measure time here
using namespace std::chrono;
system_clock::time_point startTime = high_resolution_clock::now();
CSvmBinaryClassifierBuilder( params ).Train( *problem ); // measure time only for this line
nanoseconds delay = duration_cast<nanoseconds>( high_resolution_clock::now() - startTime );
std::cout << setprecision(3) << delay / 1e6 << std::endl;
}
I bound this function to Python via pybind11:
PYBIND11_MODULE(PythonWrapper, m) {
m.def( "runSvm", &runSvm );
}
Then compiled pybind11 module library and called it from Python. The timer value was over 3000ms.
But when I call this function from pure C++, the timer value is around 800ms.
Of course, I was expecting some overhead but not in this place and not so much.
I ran it in one thread and both cases it 100% loaded one core.
Where the issue can be? Who faced the same and how did you handle this?
When I was working on a reproducible example, I found out that I compare different svm kernels in C++ example (it was based on libsvm params proved 'rbf') and in pybind11 lib (it was hardcoded 'linear'). After fixing it and comparing the same algorithms there was no difference in time.
I want to use the Delphi 4 Python components from here https://github.com/pyscripter/python4delphi
but I don't want to drop the components on a form, I want everything in code , my code goes like this :
var
PythonEngine_netA: TPythonEngine;
PythonInputOutput_netA: TPythonInputOutput;
begin
PythonEngine_netA := TPythonEngine.Create(Self);
PythonInputOutput_netA := TPythonInputOutput.Create(Self);
try
/// configure the components
PythonEngine_netA.DllName:='python39.dll';
PythonEngine_netA.IO := PythonInputOutput_netA;
PythonEngine_netA.UseLastKnownVersion := True;
PythonInputOutput_netA.OnSendUniData := PythonInputOutput_SendUniData;
PythonInputOutput_netA.UnicodeIO := True;
PythonInputOutput_netA.RawOutput := True;
/// execute the script
PythonEngine_netA.ExecString(UTF8Encode(mmo_pythoncode.text));
finally
PythonEngine_netA.free;
PythonInputOutput_netA.free;
end;
execution of this code fails, error msg : "Python is not properly initialized",
what did I miss to use this code ?
One quick look at PythonEngine.pas (or even better: always search all files for the error message to find out where and why an error is returned) tells me you missed calling PythonEngine_netA.Initialize().
Also note that /Demos describes:
Demo34 Dynamically creating, destroying and recreating PythonEngine. Uses PythonVersions
So please have a look at /Demos/Demo34/Unit1.pas how it is done there with (almost) no components. Or run the whole project in general, preferably in debug mode single stepping thru it be aware which method does what.
You just forgot to load the Dll:
PythonEngine_netA.UseLastKnownVersion:= True;
//PythonEngine_netA.opendll(PYDLL)
PythonEngine_netA.LoadDll;
PythonEngine_netA.IO:= PythonInputOutput_netA;
In python, you can learn memory location of variables by using id function, so:
X = "Hello world!"
print(id(X)) # Output is equal to 139806692112112 (0x7F27483876F0)
I'm tried to access to variable with pointers in C (Surely the other program still alive):
#include <stdio.h>
int main(void){
char *x = (char *) 0x7F27483876F0;
printf("%s\n", x);
return 0;
}
I compile the code, no errors or warnings but when i tried the running program OS giving a Segmentation error. How i can solve this problem?
Or is it possible?
Doing something like this is more and more impossible these days. With features like address space layout randomization you can't really tell where a given program, let alone variable will load in actual memory.
Best bet is to use some type of message passing. Not sure why all the downvotes on your question, but it seems like a reasonably put question, even if not technically feasible these days.
Is there any equivalent of DocTest for Delphi. I use DUnit but I like the Python DocTest idea. I saw some answer like here but I think that, for simple functions, a DocTest like could be OK.
My goal is to define my tests in the comment header when I write the function.
Like :
function Plus(i1, i2 : integer) : integer;
//>>> Check( Plus(1, 3) = 4)
begin
result := i1 + i2;
end;
The idea is that you can use a "console" to output some results to testing code, then compare the output text content to an expected value.
Take a look for instance at the regression tests available with the great DWSScript Open Source project. You'll find out some .pas files and some related .txt files.
For instance abs.pas:
var vf = 1.5;
var vi = 2;
var i : Integer := Abs(-vi);
PrintLn(i);
PrintLn(Abs(vi));
var f : Float := Abs(-vf);
PrintLn(f);
PrintLn(Abs(vf));
And the corresponding abs.txt content:
2
2
1.5
1.5
AFAIK there is no already existing solution by now integrate in the Delphi world.
Writing the test in the comment will lack for IDE auto-completion, and somewhat break the object pascal design. It would be something easy with DWS, but require to call the command-line Delphi compiler. Honestly, I do not see what is wrong having your own set of units dedicated to tests. A small piece of code with a for..to loop with fixed and random values will have a much better test coverage than a fixed set of parameters.
I want to extend a large C project with some new functionality, but I really want to write it in Python. Basically, I want to call Python code from C code. However, Python->C wrappers like SWIG allow for the OPPOSITE, that is writing C modules and calling C from Python.
I'm considering an approach involving IPC or RPC (I don't mind having multiple processes); that is, having my pure-Python component run in a separate process (on the same machine) and having my C project communicate with it by writing/reading from a socket (or unix pipe). my python component can read/write to socket to communicate. Is that a reasonable approach? Is there something better? Like some special RPC mechanism?
Thanks for the answer so far - however, i'd like to focus on IPC-based approaches since I want to have my Python program in a separate process as my C program. I don't want to embed a Python interpreter. Thanks!
I recommend the approaches detailed here. It starts by explaining how to execute strings of Python code, then from there details how to set up a Python environment to interact with your C program, call Python functions from your C code, manipulate Python objects from your C code, etc.
EDIT: If you really want to go the route of IPC, then you'll want to use the struct module or better yet, protlib. Most communication between a Python and C process revolves around passing structs back and forth, either over a socket or through shared memory.
I recommend creating a Command struct with fields and codes to represent commands and their arguments. I can't give much more specific advice without knowing more about what you want to accomplish, but in general I recommend the protlib library, since it's what I use to communicate between C and Python programs (disclaimer: I am the author of protlib).
Have you considered just wrapping your python application in a shell script and invoking it from within your C application?
Not the most elegant solution, but it is very simple.
See the relevant chapter in the manual: http://docs.python.org/extending/
Essentially you'll have to embed the python interpreter into your program.
I haven't used an IPC approach for Python<->C communication but it should work pretty well. I would have the C program do a standard fork-exec and use redirected stdin and stdout in the child process for the communication. A nice text-based communication will make it very easy to develop and test the Python program.
If I had decided to go with IPC, I'd probably splurge with XML-RPC -- cross-platform, lets you easily put the Python server project on a different node later if you want, has many excellent implementations (see here for many, including C and Python ones, and here for the simple XML-RPC server that's part the Python standard library -- not as highly scalable as other approaches but probably fine and convenient for your use case).
It may not be a perfect IPC approach for all cases (or even a perfect RPC one, by all means!), but the convenience, flexibility, robustness, and broad range of implementations outweigh a lot of minor defects, in my opinion.
This seems quite nice http://thrift.apache.org/, there is even a book about it.
Details:
The Apache Thrift software framework, for scalable cross-language
services development, combines a software stack with a code generation
engine to build services that work efficiently and seamlessly between
C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa,
JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages.
I've used the "standard" approach of Embedding Python in Another Application. But it's complicated/tedious. Each new function in Python is painful to implement.
I saw an example of Calling PyPy from C. It uses CFFI to simplify the interface but it requires PyPy, not Python. Read and understand this example first, at least at a high level.
I modified the C/PyPy example to work with Python. Here's how to call Python from C using CFFI.
My example is more complicated because I implemented three functions in Python instead of one. I wanted to cover additional aspects of passing data back and forth.
The complicated part is now isolated to passing the address of api to Python. That only has to be implemented once. After that it's easy to add new functions in Python.
interface.h
// These are the three functions that I implemented in Python.
// Any additional function would be added here.
struct API {
double (*add_numbers)(double x, double y);
char* (*dump_buffer)(char *buffer, int buffer_size);
int (*release_object)(char *obj);
};
test_cffi.c
//
// Calling Python from C.
// Based on Calling PyPy from C:
// http://doc.pypy.org/en/latest/embedding.html#more-complete-example
//
#include <stdio.h>
#include <assert.h>
#include "Python.h"
#include "interface.h"
struct API api; /* global var */
int main(int argc, char *argv[])
{
int rc;
// Start Python interpreter and initialize "api" in interface.py using
// old style "Embedding Python in Another Application":
// https://docs.python.org/2/extending/embedding.html#embedding-python-in-another-application
PyObject *pName, *pModule, *py_results;
PyObject *fill_api;
#define PYVERIFY(exp) if ((exp) == 0) { fprintf(stderr, "%s[%d]: ", __FILE__, __LINE__); PyErr_Print(); exit(1); }
Py_SetProgramName(argv[0]); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString(
"import sys;"
"sys.path.insert(0, '.')" );
PYVERIFY( pName = PyString_FromString("interface") )
PYVERIFY( pModule = PyImport_Import(pName) )
Py_DECREF(pName);
PYVERIFY( fill_api = PyObject_GetAttrString(pModule, "fill_api") )
// "k" = [unsigned long],
// see https://docs.python.org/2/c-api/arg.html#c.Py_BuildValue
PYVERIFY( py_results = PyObject_CallFunction(fill_api, "k", &api) )
assert(py_results == Py_None);
// Call Python function from C using cffi.
printf("sum: %f\n", api.add_numbers(12.3, 45.6));
// More complex example.
char buffer[20];
char * result = api.dump_buffer(buffer, sizeof buffer);
assert(result != 0);
printf("buffer: %s\n", result);
// Let Python perform garbage collection on result now.
rc = api.release_object(result);
assert(rc == 0);
// Close Python interpreter.
Py_Finalize();
return 0;
}
interface.py
import cffi
import sys
import traceback
ffi = cffi.FFI()
ffi.cdef(file('interface.h').read())
# Hold references to objects to prevent garbage collection.
noGCDict = {}
# Add two numbers.
# This function was copied from the PyPy example.
#ffi.callback("double (double, double)")
def add_numbers(x, y):
return x + y
# Convert input buffer to repr(buffer).
#ffi.callback("char *(char*, int)")
def dump_buffer(buffer, buffer_len):
try:
# First attempt to access data in buffer.
# Using the ffi/lib objects:
# http://cffi.readthedocs.org/en/latest/using.html#using-the-ffi-lib-objects
# One char at time, Looks inefficient.
#data = ''.join([buffer[i] for i in xrange(buffer_len)])
# Second attempt.
# FFI Interface:
# http://cffi.readthedocs.org/en/latest/using.html#ffi-interface
# Works but doc says "str() gives inconsistent results".
#data = str( ffi.buffer(buffer, buffer_len) )
# Convert C buffer to Python str.
# Doc says [:] is recommended instead of str().
data = ffi.buffer(buffer, buffer_len)[:]
# The goal is to return repr(data)
# but it has to be converted to a C buffer.
result = ffi.new('char []', repr(data))
# Save reference to data so it's not freed until released by C program.
noGCDict[ffi.addressof(result)] = result
return result
except:
print >>sys.stderr, traceback.format_exc()
return ffi.NULL
# Release object so that Python can reclaim the memory.
#ffi.callback("int (char*)")
def release_object(ptr):
try:
del noGCDict[ptr]
return 0
except:
print >>sys.stderr, traceback.format_exc()
return 1
def fill_api(ptr):
global api
api = ffi.cast("struct API*", ptr)
api.add_numbers = add_numbers
api.dump_buffer = dump_buffer
api.release_object = release_object
Compile:
gcc -o test_cffi test_cffi.c -I/home/jmudd/pgsql-native/Python-2.7.10.install/include/python2.7 -L/home/jmudd/pgsql-native/Python-2.7.10.install/lib -lpython2.7
Execute:
$ test_cffi
sum: 57.900000
buffer: 'T\x9e\x04\x08\xa8\x93\xff\xbf]\x86\x04\x08\x00\x00\x00\x00\x00\x00\x00\x00'
$
Few tips for binding it with Python 3
file() not supported, use open()
ffi.cdef(open('interface.h').read())
PyObject* PyStr_FromString(const char *u)
Create a PyStr from a UTF-8 encoded null-terminated character buffer.
Python 2: PyString_FromString
Python 3: PyUnicode_FromString
Change to: PYVERIFY( pName = PyUnicode_FromString("interface") )
Program name
wchar_t *name = Py_DecodeLocale(argv[0], NULL);
Py_SetProgramName(name);
for compiling
gcc cc.c -o cc -I/usr/include/python3.6m -I/usr/include/x86_64-linux-gnu/python3.6m -lpython3.6m
I butchered dump def .. maybe it will give some ideas
def get_prediction(buffer, buffer_len):
try:
data = ffi.buffer(buffer, buffer_len)[:]
result = ffi.new('char []', data)
print('\n I am doing something here here........',data )
resultA = ffi.new('char []', b"Failed") ### New message
##noGCDict[ffi.addressof(resultA)] = resultA
return resultA
except:
print >>sys.stderr, traceback.format_exc()
return ffi.NULL
}
Hopefully it will help and save you some time
apparently Python need to be able to compile to win32 dll, it will solve the problem
In such a way that converting c# code to win32 dlls will make it usable by any development tool