Related
I'm struggling how to create a python Enum object inside the Python C API. The enum class has assigned tp_base to PyEnum_Type, so it inherits Enum. But, I can't figure out a way to tell the Enum base class what items are in the enum. I want to allow iteration and lookup from Python using the __members__ attribute that every Python Enum provides.
Thank you,
Jelle
It is not straightforward at all. The Enum is a Python class using a Python metaclass. It is possible to create it in C but it will be just emulating the constructing Python code in C - the end result is the same and while it speeds up things slightly, you'll most probably run the code only once within each program run.
In any case it is possible, but it is not easy at all. I'll show how to do it in Python:
from enum import Enum
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3
print(Color)
print(Color.RED)
is the same as:
from enum import Enum
name = 'Color'
bases = (Enum,)
enum_meta = type(Enum)
namespace = enum_meta.__prepare__(name, bases)
namespace['RED'] = 1
namespace['GREEN'] = 2
namespace['BLUE'] = 3
Color = enum_meta(name, bases, namespace)
print(Color)
print(Color.RED)
The latter is the code that you need to translate into C.
Edited note: An answer on a very similar question details how enum.Enum has a functional interface that can be used instead. That is almost certainly the correct approach. I think my answer here is a useful alternative approach to be aware of, although it probably isn't the best solution to this problem.
I'm aware that this answer is slightly cheating, but this is exactly the kind of code that's better written in Python, and in the C API we still have access to the full Python interpreter. My reasoning for this is that the main reason to keep things entirely in C is performance, and it seems unlikely that creating enum objects will be performance critical.
I'll give three versions, essentially depending on the level of complexity.
First, the simplest case: the enum is entirely known and defined and compile-time. Here we simply set up an empty global dict, run the Python code, then extract the enum from the global dict:
PyObject* get_enum(void) {
const char str[] = "from enum import Enum\n"
"class Colour(Enum):\n"
" RED = 1\n"
" GREEN = 2\n"
" BLUE = 3\n"
"";
PyObject *global_dict=NULL, *should_be_none=NULL, *output=NULL;
global_dict = PyDict_New();
if (!global_dict) goto cleanup;
should_be_none = PyRun_String(str, Py_file_input, global_dict, global_dict);
if (!should_be_none) goto cleanup;
// extract Color from global_dict
output = PyDict_GetItemString(global_dict, "Colour");
if (!output) {
// PyDict_GetItemString does not set exceptions
PyErr_SetString(PyExc_KeyError, "could not get 'Colour'");
} else {
Py_INCREF(output); // PyDict_GetItemString returns a borrow reference
}
cleanup:
Py_XDECREF(global_dict);
Py_XDECREF(should_be_none);
return output;
}
Second, we might want to change what we define in C at runtime. For example, maybe the input parameters pick the enum values. Here, I'm going to use string formatting to insert the appropriate values into our string. There's a number of options here: sprintf, PyBytes_Format, the C++ standard library, using Python strings (perhaps with another call into Python code?). Pick whichever you're most comfortable with.
PyObject* get_enum_fmt(int red, int green, int blue) {
const char str[] = "from enum import Enum\n"
"class Colour(Enum):\n"
" RED = %d\n"
" GREEN = %d\n"
" BLUE = %d\n"
"";
PyObject *formatted_str=NULL, *global_dict=NULL, *should_be_none=NULL, *output=NULL;
formatted_str = PyBytes_FromFormat(str, red, green, blue);
if (!formatted_str) goto cleanup;
global_dict = PyDict_New();
if (!global_dict) goto cleanup;
should_be_none = PyRun_String(PyBytes_AsString(formatted_str), Py_file_input, global_dict, global_dict);
if (!should_be_none) goto cleanup;
// extract Color from global_dict
output = PyDict_GetItemString(global_dict, "Colour");
if (!output) {
// PyDict_GetItemString does not set exceptions
PyErr_SetString(PyExc_KeyError, "could not get 'Colour'");
} else {
Py_INCREF(output); // PyDict_GetItemString returns a borrow reference
}
cleanup:
Py_XDECREF(formatted_str);
Py_XDECREF(global_dict);
Py_XDECREF(should_be_none);
return output;
}
Obviously you can do as much or as little as you like with string formatting - I've just picked a simple example to show the point. The main differences from the previous version are the call to PyBytes_FromFormat to set up the string, and the call to PyBytes_AsString that gets the underlying char* out of the prepared bytes object.
Finally, we could prepare the enum attributes in C Python dict and pass it in. This necessitates a bit of a change. Essentially I use #AnttiHaapala's lower-level Python code, but insert namespace.update(contents) after the call to __prepare__.
PyObject* get_enum_dict(const char* key1, int value1, const char* key2, int value2) {
const char str[] = "from enum import Enum\n"
"name = 'Colour'\n"
"bases = (Enum,)\n"
"enum_meta = type(Enum)\n"
"namespace = enum_meta.__prepare__(name, bases)\n"
"namespace.update(contents)\n"
"Colour = enum_meta(name, bases, namespace)\n";
PyObject *global_dict=NULL, *contents_dict=NULL, *value_as_object=NULL, *should_be_none=NULL, *output=NULL;
global_dict = PyDict_New();
if (!global_dict) goto cleanup;
// create and fill the contents dictionary
contents_dict = PyDict_New();
if (!contents_dict) goto cleanup;
value_as_object = PyLong_FromLong(value1);
if (!value_as_object) goto cleanup;
int set_item_result = PyDict_SetItemString(contents_dict, key1, value_as_object);
Py_CLEAR(value_as_object);
if (set_item_result!=0) goto cleanup;
value_as_object = PyLong_FromLong(value2);
if (!value_as_object) goto cleanup;
set_item_result = PyDict_SetItemString(contents_dict, key2, value_as_object);
Py_CLEAR(value_as_object);
if (set_item_result!=0) goto cleanup;
set_item_result = PyDict_SetItemString(global_dict, "contents", contents_dict);
if (set_item_result!=0) goto cleanup;
should_be_none = PyRun_String(str, Py_file_input, global_dict, global_dict);
if (!should_be_none) goto cleanup;
// extract Color from global_dict
output = PyDict_GetItemString(global_dict, "Colour");
if (!output) {
// PyDict_GetItemString does not set exceptions
PyErr_SetString(PyExc_KeyError, "could not get 'Colour'");
} else {
Py_INCREF(output); // PyDict_GetItemString returns a borrow reference
}
cleanup:
Py_XDECREF(contents_dict);
Py_XDECREF(global_dict);
Py_XDECREF(should_be_none);
return output;
}
Again, this presents a reasonably flexible way to get values from C into a generated enum.
For the sake of testing I used the follow simple Cython wrapper - this is just presented for completeness to help people try these functions.
cdef extern from "cenum.c":
object get_enum()
object get_enum_fmt(int, int, int)
object get_enum_dict(char*, int, char*, int)
def py_get_enum():
return get_enum()
def py_get_enum_fmt(red, green, blue):
return get_enum_fmt(red, green, blue)
def py_get_enum_dict(key1, value1, key2, value2):
return get_enum_dict(key1, value1, key2, value2)
To reiterate: this answer is only partly in the C API, but the approach of calling Python from C is one that I've found productive at times for "run-once" code that would be tricky to write entirely in C.
I am working concurrently in C# and in Python.
Is there a difference, in terms of what is being created in memory, between passing a reference type in C#, and passing (by assignment) in Python? It seems in either case, if the variable is changed* in the function, it is changed in the outside scope as well.
(*) of course in Python it must be mutable for this to occur. An immutable object cannot be changed - but that is another topic.
Are we basically just talking different terminology for the same process, or is there a conceptual difference to be learned here, in terms of the underlying mechanism in memory?
First, all arguments are passed by value by default in C#. This has nothing to do with the type being a reference type or a value type, both behave exactly the same way.
Now, the question is, what is a variable? A variable is a placeholder for a value, nothing more. When a variabe is passed by copy, a copy of the value is made.
And what is the value stored in a variable? Well, if the type of the variable is a reference type, the value is basically the memory address of the object its referencing. If its a value type, then the value is the objet itself.
So when you say:
It seems in either case, if the variable is changed* in the function, it is changed in the outside scope as well.
That is deeply wrong because you seem to me be mixing up the type of the argument with how it is passed along:
First example:
var a = new object();
Foo(a);
var isNull = ReferenceEquals(a, null); //false!
void Foo(object o) { o = null; }
Here, a refence typed variable a is passed by value, a copy is made and then inside Foo its reassigned to null. a doesn't care a copy is reasigned inside Foo, it will still point to the same object.
Things of course change if you pass the argument by reference:
var a = new object();
Foo(ref a);
var isNull = ReferenceEquals(a, null); //true!
void Foo(ref object o) { o = null; }
Now you are not making a copy of a named o, you are passing a itself with an alias named o.
Things behave exactly the same with value types:
var a = 1;
Foo(a);
var isNull = 1 == 0; //false!
void Foo(int i) { i = 0; }
And
var a = 1;
Foo(ref a);
var isNull = 1 == 0; //true!
void Foo(ref int i) { i = 0; }
The difference between value types and reference types when you pass it a long by value is due to what the value of the variable is. Like we said before, reference typed variables store the address, so even if you pass along a copy, the copy points to the same object, so any changes in the object are visible from both variables:
var ii = new List<int>();
Foo(ii);
var b = ii.Count == 1; //true!
void Foo(List<int> list) { list.Add(1); }
But with value types, the value is the object itself, so you are passing along a copy of the object, and you are therefore modifying a copy:
struct MutableStruct
{
public int I { get; set; }
}
var m = new mutableStruct();
Foo(m);
var b = m.I == 1; //false!
void Foo(MutableStruct mutableStruct) { mutableStruct.I = 1; }
Does this make things clearer?
Here is my C code:
//int MyFunc(char* res); -> This is the definition of C function
char data[4096];
MyFunc(data);
printf("Data is : %s\n", data);
The data variable is updated by the C function. I used bytearray in Python to pass the variable as argument but the updated array is not reflecting. Any working code sample is much appreciated.
EDIT: I am using Python 3.7.
My Python code:
data = bytearray(b'1234567890')
str_buffer = create_string_buffer(bytes(data), len(data))
print(MyFunc(str_buffer))
print(str_buffer.value) #Output: b''
str_buffer does not contain the values updated by MyFunc().
Calling MyFunc() from C# using the below signature works for me. I am looking for a Python 3.7 equivalent of it.
[DllImport("mydll.dll", CharSet = CharSet.Ansi, CallingConvention = CallingConvention.Cdecl)]
public static extern int MyFunc(StringBuilder data);
A bytearray isn't the right way to pass a char * to a C function. Use create_string_buffer instead. Also, len(data) is an off-by-one error that results in a null terminator not being present, so either stick a + 1 on that or remove it, as the default length is right. Here's a minimal working example. First, a C function that turns every letter uppercase, and returns the number of letters that were already uppercase:
#include <ctype.h>
int MyFunc(char* res) {
int i = 0;
while(*res) {
if(isupper(*res)) {
++i;
} else {
*res = toupper(*res);
}
++res;
}
return i;
}
I compiled it with gcc -fPIC -shared upstring.c -o upstring.so. Since you're on Windows, you'll have to adapt this.
Now, some Python that calls it:
from ctypes import *
upstring = CDLL("./upstring.so") # Since you're on Windows, you'll have to adapt this too.
data = bytearray(b'abc12DEFGHI')
str_buffer = create_string_buffer(bytes(data)) # Note: using len(data) would be an off-by-one error that would lose the null terminator, so either omit it or use len(data)+1
print(upstring.MyFunc(str_buffer)) # prints 6
print(str_buffer.value) # prints b'ABC12DEFGHI'
I'm relatively new to C, coming from python mainly and I want to be able to create a new variable as a string to use the function strcat() based of an integer. For example, if each time I loop through it and increment an integer, 1, a new variable that is a string/char is now "1", and 2, "2" and so on, so that when using strcat(dest, value), it makes a string for example called: (1st loop) "TEST1.txt", (2nd loop) "TEST2.txt" and so on. If this doesn't really make sense, the best way I can describe this is in python, to achieve the exact same thing, say:
a = 1
while True:
file = open("Test" + str(a) + ".txt", "w")
file.close()
a += 1
if a == 10:
break
I know how to do it with strings, for example if a isn't an integer, from the python code to do the "Test" + str(a) in c, i can do:
char* a = "test";
char* b = "1";
strcat(a, b);
if i print that i would get "test1", but I need it to be an integer first then a string so that i can increment it, thanks
There is no great sense to use strcat in this case. Just write
char file_name[] = "test0.txt";
do
{
++file_name[4];
FILE *fp = fopen( file_name , "w")
//...
fclose( fp )
} while ( file_name[4] != '9' );
Or you can usethe function sprintf as for example
#include <stdio.h>
int main(void)
{
char name[10];
int i = 0;
do
{
sprintf( name, "%s%1d%s", "Test", ++i, ".txt" );
printf( "%s\n", name );
} while ( i != 9 );
return 0;
}
The program output is
Test1.txt
Test2.txt
Test3.txt
Test4.txt
Test5.txt
Test6.txt
Test7.txt
Test8.txt
Test9.txt
In the standard library stdlib.h you can use the function itoa to return a string given a integer. (For reference, you can do the reverse using atoi)
char * itoa ( int value, char * str, int base );
I don't have a C reference to hand on the function, but some links to get you started are either here or here.
I have a self-made C library that I want to access using python. The problem is that the code consists essentially of two parts, an initialization to read in data from a number of files and a few calculations that need to be done only once. The other part is called in a loop and uses the data generated before repeatedly. To this function I want to pass parameters from python.
My idea was to write two C wrapper functions, "init" and "loop" - "init" reads the data and returns a void pointer to a structure that "loop" can use together with additional parameters that I can pass on from python. Something like
void *init() {
struct *mystruct ret = (mystruct *)malloc(sizeof(mystruct));
/* Fill ret with data */
return ret;
}
float loop(void *data, float par1, float par2) {
/* do stuff with data, par1, par2, return result */
}
I tried calling "init" from python as a c_void_p, but since "loop" changes some of the contents of "data" and ctypes' void pointers are immutable, this did not work.
Other solutions to similar problems I saw seem to require knowledge of how much memory "init" would use, and I do not know that.
Is there a way to pass data from one C function to another through python without telling python exactly what or how much it is? Or is there another way to solve my problem?
I tried (and failed) to write a minimum crashing example, and after some debugging it turned out there was a bug in my C code. Thanks to everyone who replied!
Hoping that this might help other people, here is a sort-of-minimal working version (still without separate 'free' - sorry):
pybug.c:
#include <stdio.h>
#include <stdlib.h>
typedef struct inner_struct_s {
int length;
float *array;
} inner_struct_t;
typedef struct mystruct_S {
int id;
float start;
float end;
inner_struct_t *inner;
} mystruct_t;
void init(void **data) {
int i;
mystruct_t *mystruct = (mystruct_t *)malloc(sizeof(mystruct_t));
inner_struct_t *inner = (inner_struct_t *)malloc(sizeof(inner_struct_t));
inner->length = 10;
inner->array = calloc(inner->length, sizeof(float));
for (i=0; i<inner->length; i++)
inner->array[i] = 2*i;
mystruct->id = 0;
mystruct->start = 0;
mystruct->end = inner->length;
mystruct->inner = inner;
*data = mystruct;
}
float loop(void *data, float par1, float par2, int newsize) {
mystruct_t *str = data;
inner_struct_t *inner = str->inner;
int i;
inner->length = newsize;
inner->array = realloc(inner->array, newsize * sizeof(float));
for (i=0; i<inner->length; i++)
inner->array[i] = par1 + i * par2;
return inner->array[inner->length-1];
}
compile as
cc -c -fPIC pybug.c
cc -shared -o libbug.so pybug.o
Run in python:
from ctypes import *
sl = CDLL('libbug.so')
# What arguments do functions take / return?
sl.init.argtype = c_void_p
sl.loop.restype = c_float
sl.loop.argtypes = [c_void_p, c_float, c_float, c_int]
# Init takes a pointer to a pointer
px = c_void_p()
sl.init(byref(px))
# Call the loop a couple of times
for i in range(10):
print sl.loop(px, i, 5, 10*i+5)
You should have a corresponding function to free the data buffer when the caller is done. Otherwise I don't see the issue. Just pass the pointer to loop that you get from init.
init.restype = c_void_p
loop.argtypes = [c_void_p, c_float, c_float]
loop.restype = c_float
I'm not sure what you mean by "ctypes' void pointers are immutable", unless you're talking about c_char_p and c_wchar_p. The issue there is if you pass a Python string as an argument it uses Python's private pointer to the string buffer. If a function can change the string, you should first copy it to a c_char or c_wchar array.
Here's a simple example showing the problem of passing a Python string (2.x byte string) as an argument to a function that modifies it. In this case it changes index 0 to '\x00':
>>> import os
>>> from ctypes import *
>>> open('tmp.c', 'w').write("void f(char *s) {s[0] = 0;}")
>>> os.system('gcc -shared -fPIC -o tmp.so tmp.c')
0
>>> tmp = CDLL('./tmp.so')
>>> tmp.f.argtypes = [c_void_p]
>>> tmp.f.restype = None
>>> tmp.f('a')
>>> 'a'
'\x00'
>>> s = 'abc'
>>> tmp.f(s)
>>> s
'\x00bc'
This is specific to passing Python strings as arguments. It isn't a problem to pass pointers to data structures that are intended to be mutable, either ctypes data objects such as a Structure, or pointers returned by libraries.
Is your C code in a DLL? If so can might consider creating a global pointer in there. init() will do any initialization required and set the pointer equal to newly allocated memory and loop() will operate on that memory. Also don't forget to free it up with a close() function