How does one go about to parse a group of required but mutually exclusive arguments using the python C-api?
E.g. given the function definition
static PyObject* my_func(PyObject *self, PyObject *args, PyObject *kwargs) {
double a; // first argument, required
double b=0, c=0; // second argument, required but mutually exclusive, b is default keyword if no keyword is set
char d[] = "..."; // third argument, optional
// parse arguments
...
}
My idea here was to parse the input arguments twice, i.e. replacing ... above with:
static const char *kwList1[] = {"a","b","c","d"};
static const char *kwList2[] = {"a","b","d"};
int ret;
if (!(ret = PyArg_ParseTupleAndKeywords(args,kwargs,"d|dds",(char **)kwList1,&a,&b,&c,&d))) {
ret = PyArg_ParseTupleAndKeywords(args,kwargs,"d|ds",(char **)kwList2,&a,&b,&d));
}
if (!ret) return NULL;
// verify that one of, but not both, variables b and c are non-zero
...
However, the second call to PyArg_ParseTupleAndKeywords() returns 0 for valid input so I assume here that the variables args and kwargs have some attributes set by the first call to PyArg_ParseTupleAndKeywords() that causes the second call to fail (output python error is: TypeError: a float is required).
I'm aware that the above could be solved using the argparse python module but would prefer a solution directly using the C-api. One idea here would be if it were possible to first copy of the input args and kwargs into two new PyObject variables and use these in the second call to PyArg_ParseTupleAndKeywords(), however I can't find any api-function to do so (guess I also would need to know howto release the memory allocated for this).
Seems like the issue were that the first call to PyArg_ParseTupleAndKeywords() set the error indicator which caused the second call to the function to fail. So the solution is to insert a call to PyErr_Clear() between the calls to PyArg_ParseTupleAndKeywords(). In summary, the following code performs the task
static PyObject* my_func(PyObject *self, PyObject *args, PyObject *kwargs) {
double a; // first argument, required
double b=0, c=0; // second argument, required but mutually exclusive, b is default keyword if no keyword is set
char d[] = "..."; // third argument, optional
// parse arguments
static const char *kwList1[] = {"a","b","c","d"};
static const char *kwList2[] = {"a","b","d"};
int ret;
if (!(ret = PyArg_ParseTupleAndKeywords(args,kwargs,"d|dds",(char **)kwList1,&a,&b,&c,&d))) {
PyErr_Clear();
ret = PyArg_ParseTupleAndKeywords(args,kwargs,"d|ds",(char **)kwList2,&a,&b,&d));
}
if (!ret) return NULL;
// verify that one of, but not both, variables b and c are non-zero
if (b==0 && c==0) {
PyErr_SetString(PyExc_TypeError,"Required mutually exclusive arguments 'b' or 'c' (pos 2) not found (or input with value 0)");
return NULL;
} else if (b!=0 && c!=0) {
PyErr_SetString(PyExc_TypeError,"Use of multiple mutually exclusive required arguments 'b' and 'c' (pos 2)");
return NULL;
}
...
}
Then again this does not guard against the calling the function with both the arguments b and c given that one of them is 0 and the other not. However this is a minor problem.
Related
I want to use Boost.Python to create a Python wrapper for a C++ constructor with optional arguments. I want the Python wrapper to act like this:
class Foo():
def __init__(self, filename, phase, stages=None, level=0):
"""
filename -- string
phase -- int
stages -- optional list of strings
level -- optional int
"""
if stages is None:
stages = []
# ...
How do I do this with Boost.Python? I don't see how to do it with make_constructor, and I don't know how to make a constructor with raw_function. Is there some better documentation than this out there?
My specific problem is trying to add two optional arguments (stages and level) to these two constructors:
https://github.com/BVLC/caffe/blob/rc3/python/caffe/_caffe.cpp#L76-L96
Thanks to Dan's comments, I found a solution that works. I'll copy most of it here since there are some interesting tidbits about how to extract objects from bp::object, etc.
// Net constructor
shared_ptr<Net<Dtype> > Net_Init(string param_file, int phase,
const int level, const bp::object& stages,
const bp::object& weights_file) {
CheckFile(param_file);
// Convert stages from list to vector
vector<string> stages_vector;
if (!stages.is_none()) {
for (int i = 0; i < len(stages); i++) {
stages_vector.push_back(bp::extract<string>(stages[i]));
}
}
// Initialize net
shared_ptr<Net<Dtype> > net(new Net<Dtype>(param_file,
static_cast<Phase>(phase), level, &stages_vector));
// Load weights
if (!weights_file.is_none()) {
std::string weights_file_str = bp::extract<std::string>(weights_file);
CheckFile(weights_file_str);
net->CopyTrainedLayersFrom(weights_file_str);
}
return net;
}
BOOST_PYTHON_MODULE(_caffe) {
bp::class_<Net<Dtype>, shared_ptr<Net<Dtype> >, boost::noncopyable >("Net",
bp::no_init)
.def("__init__", bp::make_constructor(&Net_Init,
bp::default_call_policies(), (bp::arg("network_file"), "phase",
bp::arg("level")=0, bp::arg("stages")=bp::object(),
bp::arg("weights_file")=bp::object())))
}
The generated signature is:
__init__(boost::python::api::object, std::string network_file, int phase,
int level=0, boost::python::api::object stages=None,
boost::python::api::object weights_file=None)
And I can use it like:
net = caffe.Net('network.prototxt', weights_file='weights.caffemodel',
phase=caffe.TEST, level=1, stages=['deploy'])
Full code available in pull request here: https://github.com/BVLC/caffe/pull/3863
Asked because of this: Default argument in c++
Say I have a function such as this: void f(int p1=1, int p2=2, int p3=3, int p4=4);
And I want to call it using only some of the arguments - the rest will be the defaults.
Something like this would work:
template<bool P1=true, bool P2=true, bool P3=true, bool P4=true>
void f(int p1=1, int p2=2, int p3=3, int p4=4);
// specialize:
template<>
void f<false, true, false, false>(int p1) {
f(1, p1);
}
template<>
void f<false, true, true, false>(int p1, int p2) {
f(1, p1, p2);
}
// ... and so on.
// Would need a specialization for each combination of arguments
// which is very tedious and error-prone
// Use:
f<false, true, false, false>(5); // passes 5 as p2 argument
But it requires too much code to be practical.
Is there a better way to do this?
Use the Named Parameters Idiom (→ FAQ link).
The Boost.Parameters library (→ link) can also solve this task, but paid for by code verbosity and greatly reduced clarity. It's also deficient in handling constructors. And it requires having the Boost library installed, of course.
Have a look at the Boost.Parameter library.
It implements named paramaters in C++. Example:
#include <boost/parameter/name.hpp>
#include <boost/parameter/preprocessor.hpp>
#include <iostream>
//Define
BOOST_PARAMETER_NAME(p1)
BOOST_PARAMETER_NAME(p2)
BOOST_PARAMETER_NAME(p3)
BOOST_PARAMETER_NAME(p4)
BOOST_PARAMETER_FUNCTION(
(void),
f,
tag,
(optional
(p1, *, 1)
(p2, *, 2)
(p3, *, 3)
(p4, *, 4)))
{
std::cout << "p1: " << p1
<< ", p2: " << p2
<< ", p3: " << p3
<< ", p4: " << p4 << "\n";
}
//Use
int main()
{
//Prints "p1: 1, p2: 5, p3: 3, p4: 4"
f(_p2=5);
}
Although Boost.Parameters is amusing, it suffers (unfortunately) for a number of issues, among which placeholder collision (and having to debug quirky preprocessors/template errors):
BOOST_PARAMETER_NAME(p1)
Will create the _p1 placeholder that you then use later on. If you have two different headers declaring the same placeholder, you get a conflict. Not fun.
There is a much simpler (both conceptually and practically) answer, based on the Builder Pattern somewhat is the Named Parameters Idiom.
Instead of specifying such a function:
void f(int a, int b, int c = 10, int d = 20);
You specify a structure, on which you will override the operator():
the constructor is used to ask for mandatory arguments (not strictly in the Named Parameters Idiom, but nobody said you had to follow it blindly), and default values are set for the optional ones
each optional parameter is given a setter
Generally, it is combined with Chaining which consists in making the setters return a reference to the current object so that the calls can be chained on a single line.
class f {
public:
// Take mandatory arguments, set default values
f(int a, int b): _a(a), _b(b), _c(10), _d(20) {}
// Define setters for optional arguments
// Remember the Chaining idiom
f& c(int v) { _c = v; return *this; }
f& d(int v) { _d = v; return *this; }
// Finally define the invocation function
void operator()() const;
private:
int _a;
int _b;
int _c;
int _d;
}; // class f
The invocation is:
f(/*a=*/1, /*b=*/2).c(3)(); // the last () being to actually invoke the function
I've seen a variant putting the mandatory arguments as parameters to operator(), this avoids keeping the arguments as attributes but the syntax is a bit weirder:
f().c(3)(/*a=*/1, /*b=*/2);
Once the compiler has inlined all the constructor and setters call (which is why they are defined here, while operator() is not), it should result in similarly efficient code compared to the "regular" function invocation.
This isn't really an answer, but...
In C++ Template Metaprogramming by David Abrahams and Aleksey Gurtovoy (published in 2004!) the authors talk about this:
While writing this book, we reconsidered the interface used for named
function parameter support. With a little experimentation we
discovered that it’s possible to provide the ideal syntax by using
keyword objects with overloaded assignment operators:
f(slew = .799, name = "z");
They go on to say:
We’re not going to get into the implementation details of this named
parameter library here; it’s straightforward enough that we suggest
you try implementing it yourself as an exercise.
This was in the context of template metaprogramming and Boost::MPL. I'm not too sure how their "straighforward" implementation would jive with default parameters, but I assume it would be transparent.
Refering to http://mail.python.org/pipermail/python-dev/2009-June/090210.html
AND http://dan.iel.fm/posts/python-c-extensions/
and here is other places i searched regarding my question:
http://article.gmane.org/gmane.comp.python.general/424736
http://joyrex.spc.uchicago.edu/bookshelves/python/cookbook/pythoncook-CHP-16-SECT-3.html
http://docs.python.org/2/c-api/sequence.html#PySequence_Check
Python extension module with variable number of arguments
I am inexperienced in Python/C API.
I have the following code:
sm_int_list = (1,20,3)
c_int_array = (ctypes.c_int * len(sm_int_list))(*sm_int_list)
sm_str_tuple = ('some','text', 'here')
On the C extension side, i have done something like this:
static PyObject* stuff_here(PyObject *self, PyObject *args)
{
char* input;
int *i1, *i2;
char *s1, *s2;
// args = (('some','text', 'here'), [1,20,3], ('some','text', 'here'), [1,20,3])
**PyArg_ParseTuple(args, "(s#:):#(i:)#(s#:):#(i:)#", &s1, &i1, &s2, &i2)**;
/*stuff*/
}
such that:
stuff.here(('some','text', 'here'), [1,20,3], ('some','text', 'here'), [1,20,3])
returns data in the same form as args after some computation.
I would like to know the PyArg_ParseTuple expression, is it the proper way to parse
an array of varying string
an array of integers
UPDATE NEW
Is this the correct way?:
static PyObject* stuff_here(PyObject *self, PyObject *args)
unsigned int tint[], cint[];
ttotal=0, ctotal=0;
char *tstr, *cstr;
int *t_counts, *c_counts;
Py_ssize_t size;
PyObject *t_str1, *t_int1, *c_str2, *c_int2; //the C var that takes in the py variable value
PyObject *tseq, cseq;
int t_seqlen=0, c_seqlen=0;
if (!PyArg_ParseTuple(args, "OOiOOi", &t_str1, &t_int1, &ttotal, &c_str2, &c_int2, &ctotal))
{
return NULL;
}
if (!PySequence_Check(tag_str1) && !PySequence_Check(cat_str2)) return NULL;
else:
{
//All things t
tseq = PySequence_Fast(t_str1, "iterable");
t_seqlen = PySequence_Fast_GET_SIZE(tseq);
t_counts = PySequence_Fast(t_int1);
//All things c
cseq = PySequence_Fast(c_str2);
c_seqlen = PySequence_Fast_GET_SIZE(cseq);
c_counts = PySequence_Fast(c_int2);
//Make c arrays of all things tag and cat
for (i=0; i<t_seqlen; i++)
{
tstr[i] = PySequence_Fast_GET_ITEM(tseq, i);
tcounts[i] = PySequence_Fast_GET_ITEM(t_counts, i);
}
for (i=0; i<c_seqlen; i++)
{
cstr[i] = PySequence_Fast_GET_ITEM(cseq, i);
ccounts[i] = PySequence_Fast_GET_ITEM(c_counts, i);
}
}
OR
PyArg_ParseTuple(args, "(s:)(i:)(s:)(i:)", &s1, &i1, &s2, &i2)
And then again while returning,
Py_BuildValue("sisi", arr_str1,arr_int1,arr_str2,arr_int2) ??
Infact if someone could in detail clarify the various PyArg_ParseTuple function that would be of great benefit. the Python C API, as i find it in the documentation, is not exactly a tutorial on things to do.
You can use PyArg_ParseTuple to parse a real tuple, that has a fixed structure. Especially the number of items in the subtuples cannot change.
As the 2.7.5 documentation says, your format "(s#:):#(i:)#(s#:):#(i:)#" is wrong since : cannot occur in nested parenthesis. The format "(sss)(iii)(sss)(iii)", along with total of 12 pointer arguments should match your arguments. Likewise for Py_BuildValue you can use the same format string (which creates 4 tuples within 1 tuple), or "(sss)[iii](sss)[iii]" if the type matters (this makes the integers to be in lists instead of tuples).
If I have the following function and the optional argument myobj is not passed, does myobj remain NULL or is it set to Py_None?
static PyObject * myfunc(PyObject * self, PyObject * args) {
PyObject * myobj = NULL;
if (!PyArg_ParseTuple(args, "|O", &myobj)) {
return NULL;
}
// ...
}
According Parsing arguments and building values,
| Indicates that the remaining arguments in the Python argument list are optional. The C variables corresponding to optional arguments should be initialized to their default value — when an optional argument is not specified, PyArg_ParseTuple() does not touch the contents of the corresponding C variable(s).
Does this apply to PyObject *s? It's obviously a pointer that exists in C so one could say it's a C variable, but it's a pointer to a python object so one could also say it does not count as a C variable.
It will remain NULL. And of course a pointer to a struct is a C object.
If I'm trying to overload an embedded Python function so that the second argument can be a long or an Object, is there a standard way to do it? Is this it?
What I'm trying now (names changed to protect the innocent):
bool UseLongVar2 = true;
if (!PyArg_ParseTuple(args, "ll:foo", &LongVar1, &LongVar2))
{
PyErr_Clear();
if (!PyArg_ParseTuple(args, "lO&:foo", &LongVar1, convertObject, &Object))
{
UseLongVar2 = false;
return NULL;
}
}
What I normally do is have two C functions that take the different arguments. The "python-facing" function's job is to parse out the arguments, call the appropriate C function, and build the return value if any.
This is pretty common when, for example, you want to allow both byte and Unicode strings.
Here is an example of what I mean.
// Silly example: get the length of a string, supporting Unicode and byte strings
static PyObject* getlen_py(PyObject *self, PyObject *args)
{
// Unpack our argument (error handling omitted...)
PyObject *arg = NULL;
PyArg_UnpackTuple(args, "getlen", 1, 1, arg) ;
if ( PyUnicode_Check(arg) )
{
// It's a Unicode string
return PyInt_FromLong(getlen_w(PyUnicode_AS_UNICODE(arg))) ;
}
else
{
// It's a byte string
return PyInt_FromLong(getlen_a(PyString_AS_STRING(arg))) ;
}
}