I need some advice. Two questions, does something already exist for this, what modules should I use to develop this.
I have some structures that come from an XML file. I want to represent them in Python Classes (maybe using a factory to create a class per structure). But I want these classes to have a function that will emit the structure as a C Struct.
From my research ctypes seems like the recommended thing to use to represent the structures in Python classes, but I don't see any methods for anything that will emit C Stucts for the creation of a header file.
From OP's comment I think the minimal solution a set of helper functions instead of classes. the xmltodict library makes it easy to turn the XML data into nested dictionaries, more or less like JSON. A set of helpers that parse the contents and generate appropriate C-struct strings is all that's really needed. If you can work with dictionaries :
{
"name": "my_struct",
"members": {
[
"name": "intmember",
"ctype": "int"
},
{
"name": "floatmember",
"ctype": "float"
}
]
}
You can do something like:
from string import Template
struct_template_string = '''
typedef $structname struct {
$defs
} $structname;
'''
struct_template = Template(struct_template_string)
member_template = Template(" $ctype $name;")
def spec_to_struct(spec_dict):
structname = spec_dict['name']
member_data = spec_dict['members']
members = [member_template.substitute(d) for d in member_data]
return struct_template.substitute(structname = structname, defs = "\n".join(members))
Which will produce something like:
typedef my_struct struct {
int intmember;
float floatmember;
} my_struct;
I'd try to get it working with basic functions first before trying to build up a class scaffold. It would be pretty easy to hide the details in a class using property descriptors:
class data_property(object):
def __init__(self, path, wrapper = None):
self.path = path
self.wrapper = wrapper
def __get__(self, instance, owner):
result = instance[self.path]
if self.wrapper:
if hasattr(result, '__iter__'):
return [self.wrapper(**i) for i in result]
return self.wrapper(**result)
return result
class MemberWrapper(dict):
name = data_property('name')
type = data_property('ctype')
class StructWrapper(dict):
name = data_property('name')
members = data_property('members', MemberWrapper )
test = StructWrapper(**example)
print test.name
print test.members
for member in test.members:
print member.type, member.name
# my_struct
# [{'name': 'intmember', 'ctype': 'int'}, {'name': 'floatmember', 'ctype': 'float'}]
# int intmember
# float floatmember
Related
I wonder if anyone might have a suggestion for a better way to build up a Pydantic model for this case?
The data set I am working with (JSON) is mostly the same structure throughout, but with some differences only down at the lowest levels of the tree. ie:
// data.json
{
"FirstItem": {
"Name": "first item",
"Data": {
"attr_1": "a",
"attr_2": "b"
}
},
"SecondItem": {
"Name": "second item",
"Data": {
"attr_3": "d",
"attr_4": "e"
}
},
...
}
So I am wondering, is there a suggested method for building a Pydantic model that uses a standard 'Item' (in this case, it would have 'Name' and 'Data'), but then change the 'Data' on a case-by-case basis?
I have a working example, but it feels quite verbose?
working example:
from pydantic import BaseModel
class FirstItemData(BaseModel):
attr_1: str
attr_2: str
class FirstItem(BaseModel):
Name: str
Data: FirstItemData # <--- The unique part
class SecondItemData(BaseModel):
attr_3: str
attr_4: str
class SecondItem(BaseModel):
Name: str
Data: SecondItemData
class Example(BaseModel):
FirstItem: FirstItem
SecondItem: SecondItem
o = Example.parse_file("data.json")
The above does work, but it feels like building the Item 'holder' each time (the part with 'Name' and 'Data') is redundant? Is there way to specify a generic 'container' structure, and then swap out the 'Data'"? Something like:
class GenericContainer(BaseModel):
Name: str
Data: ????
class Example(BaseModel):
FirstItem: GenericContainer(Data = FirstItemData)
SecondItem: GenericContainer(Data = SecondItemData)
or something of that sort? In this case I have several dozen of these unique 'Items' (only unique in their 'Data' part) and it doesn't seem correct to create 2 classes for each one? Does it?
I do realize that using the type Dict in place of the detailed 'Data' does work to load in the data, but it comes in as a dict instead of an object, which is not ideal in this case.
any thoughts or suggestions are much appreciated. Thanks!
Based on the comment from Hernán Alarcón, i wanted to try and
i believe this should work. Perhaps it will usefull to someone.
from pydantic.generics import BaseModel, GenericModel
from typing import Generic, TypeVar, Optional
class FirstItemData(BaseModel):
attr_1: str
attr_2: str
class SecondItemData(BaseModel):
attr_3: str
attr_4: str
TypeX = TypeVar('TypeX')
class GenericContainer(GenericModel, Generic[TypeX]):
Name: str
Data: TypeX
class ItemBag(BaseModel):
FirstItem: Optional[GenericContainer[FirstItemData]]
SecondItem: Optional[GenericContainer[SecondItemData]]
# some tests
one_bag = ItemBag(FirstItem = {"Name":"My first item", "Data":{"attr_1":"test1", "attr_2":"test2"}})
another_bag = ItemBag(FirstItem = {"Name":"My first item", "Data":{"attr_1":"test1", "attr_2":"test2"}}, SecondItem = {"Name":"My first item", "Data":{"attr_3":"test3", "attr_4":"test4"}})
# failing tests to slightly check validation
one_failing_bag = ItemBag(FirstItem = {"Name":"My first item", "Data":{"attr_3":"test1", "attr_42":"test2"}})
another_failing_bag = ItemBag(SecondItem = {"Name":"My second item", "Data":{"attr_3":"test3", "attr_42":"test2"}})
# the parsing way
parsed_bag = ItemBag.parse_obj({"FirstItem":{"Name":"My first item", "Data":{"attr_1":"test1", "attr_2":"test2"}}, "SecondItem": {"Name":"My first item", "Data":{"attr_3":"test3", "attr_4":"test4"}}})
So it works,
but i am not sure i'd choose genericity versus readability.
I am getting Data via a REST-Interface and I want to store those data in a class-object.
my class could looks like this:
class Foo:
firstname = ''
lastname = ''
street = ''
number = ''
and the json may look like this:
[
{
"fname": "Carl",
"lname": "any name",
"address": ['carls street', 12]
}
]
What's the easiest way to map between the json and my class?
My problem is: I want to have a class with a different structure than the json.
I want the names of the attributes to be more meaningful.
Of course I know that I could simply write a to_json method and a from_json method which does what I want.
The thing is: I have a lot of those classes and I am looking for more declarative way to write the code.
e.g. in Java I probably would use mapstruct.
Thanks for your help!
Use a dict for the json input. Use **kwargs in an __init__ method in your class and map the variables accordingly.
I had a similar problem, and I solved it by using #classmethod
import json
class Robot():
def __init__(self, x, y):
self.type = "new-robot"
self.x = x
self.y = y
#classmethod
def create_robot(cls, sdict):
if sdict["type"] == "new-robot":
position = sdict["position"]
return cls(position['x'], position['y'])
else:
raise Exception ("Unable to create a new robot!!!")
if __name__=='__main__':
input_string = '{"type": "new-robot", "position": {"x": 3, "y": 3}}'
cmd = json.loads(input_string)
bot = Robot.create_robot(cmd)
print(bot.type)
Perhaps you could you two classes, one directly aligned with the Json (your source class) and the other having the actual structure you need. Then you could map them using the ObjectMapper class[https://pypi.org/project/object-mapper/]. This is very close to the MapStruct Library for Java.
ObjectMapper is a class for automatic object mapping. It helps you to create objects between project layers (data layer, service layer, view) in a simple, transparent way.
I am trying to create a blockchain and need a function to put my Blocks into json format and be able to create objects again. This seems quite difficult, since my constructor does not take all the attributes as parameters. My constructor looks like this:
class Block:
id = 0
def __init__(self, transaction, prevhash):
self.transactions = []
self.transactions.append(transaction)
self.prevhash = prevhash
self.timestamp = time.time()
self.id = Block.id
Block.id = Block.id + 1
I encode them the following (which seems to work perfectly fine):
def to_json(self):
return json.dumps(self, indent=4, cls=BlockEncoder)
class BlockEncoder(JSONEncoder):
def default(self, o):
return o.__dict__
It creates that output for me (just an example):
{
"transactions": [
{
"ipfshash": 1,
"title": null,
"length": null,
"id": 0
},
{
"ipfshash": 3,
"title": null,
"length": null,
"id": 2
}
],
"prevhash": 10,
"timestamp": 1591350715.1794589,
"id": 0
}
What im trying now to get it into an object again is (i need a generator like object to call the function, but it works. Still trying to figure out how i can make that one static :D)
def from_json(self, jsondict):
return json.loads(jsondict, object_hook=custom_block_decoder)
def custom_block_decoder(jsondict):
return namedtuple('Block', jsondict.keys())(*jsondict.values())
So, if I do that with some element and print it, it wont use my defined __str__ function, also I cant call any other function of my Block class. It seems that namedtuple('Block', jsondict.keys())(*jsondict.values()) just names my object type "Block" but does not really apply anything to it to make it an object. I can call element.attribute, but cant call for element.to_json() again, since the error message is AttributeError: 'Block' object has no attribute 'to_json'. I thought about decoding it per hand, but since I cant use multiple constructors that does not seem a good idea. Would be great if somebody could help me
Why are you using namedtuple?
It seems that namedtuple('Block', jsondict.keys())(*jsondict.values())
just names my object type "Block" but does not really apply anything to it to make it an object.
Everything is an object. namedtuple is a factory function that creates a subclass of tuple that supports named attribute access, but it is still a tuple. I'm not sure why would you expect it to be an instance of the custom Block class that you defined. It doesn't make much sense to use it here.
In any case, you have to write the deserialization routine yourself. In general, this would look something like
class Foo:
def __init__(self, bar):
self.bar = bar
self.baz = 42
def frobnicate(self):
baz = self.baz
self.baz = self.baz ** 2 // 4
#classmethod
def from_dict(cls, dict_foo):
foo = cls.__new__(cls) # just object.__new__
# here is where you have to implement the logic yourself to rebuild your object.
# hopefully it doesn't get too complex
foo.__dict__.update(dict_foo)
return foo
Note, using a classmethod is customary, then you can use
Foo.from_dict({'bar': 8, 'baz': 42})
And it plays a bit better with inheritance. These sorts of classmethods are usually referred to as alternative constructors.
In QuickFix, how can I get the name of the tag from the tag number using Python? For example, for OrdStatus, how do I convert tag number 5 to the String "OrdStatus_CANCELED"?
.NET:
If you are using QuickFIX/N, you can achieve this using a DataDictionary instance with whatever data source you want (e.g., FIX42.xml). Note that you can get the DataDictionary instance associated with a given Session or the application itself with Session's properties SessionDataDictionary and ApplicationDataDictionary, respectively.
Consider this trivial C# program:
namespace QuickFixTests
{
using System;
using QuickFix;
using QuickFix.DataDictionary;
using QuickFix.Fields;
class Program
{
static void Main(string[] args)
{
var qfm = new Message();
qfm.SetField(new OrdStatus('4'));
var ordStatus = qfm.GetField(Tags.OrdStatus);
var dd = new DataDictionary("FIX42.xml");
Console.WriteLine(dd.FieldsByTag[39].EnumDict[ordStatus]); // Prints CANCELED
}
}
}
C++/Python:
The C++ DataDictionary class has a method getValueName:
bool getValueName( int field, const std::string& value, std::string& name ) const
{
ValueToName::const_iterator i = m_valueNames.find( std::make_pair(field, value) );
if(i == m_valueNames.end()) return false;
name = i->second;
return true;
}
The following snippets (with comments added) from one of the Python DataDictionary unit tests show how to use getValueName given a DataDictionary instance.
# Create a DataDictionary
def setUp(self):
self.object = fix.DataDictionary()
# Add a dummy value
self.object.addValueName( 23, "BOO", "VALUE_23_BOO" )
# Test that the dummy value's name in the dictionary matches what was set
self.assertEquals( "VALUE_23_BOO", self.object.getValueName(23, "BOO", "")
I'm new to Python so forgive me if I'm not even using the right terminology... I'm using Python 3.2 and I'm trying to figure out whether I can decorate a class property with some declarative-style information.
In my mind it would look like this:
class MyTestClass:
def __init__(self, foo):
self.foo = foo
#property
#somedeclarativeInfo("ABC",123)
def radius(self):
return self.__foo
#radius.setter
def radius(self, foo):
self.__foo = foo
There are then two different things I'd want to do with the class:
A - Be able to interact with the foo property just like any other property (simple gets and sets)
B - Be able to dynamically find properties on a particular class that are decorated with this descriptor and be able to pull out the "ABC" and 123 values, etc.
I think maybe I should be creating a descriptor to accomplish what I want, but I'm not sure if I'm on the right track, or if this can be done.
Since my background is .Net I whipped up the following example to show what I want to do, in case that helps anyone understand my goal:
using System;
using System.Reflection;
namespace SampleWithProperties
{
public class MyCustomAttribute : Attribute
{
public string Val1;
public string Val2;
public MyCustomAttribute(string val1,string val2)
{
Val2 = val2;
Val1 = val1;
}
}
public class Foo
{
[MyCustomAttribute("abc","def")]
public string PropertyA { get; set; }
[MyCustomAttribute("xyz","X")]
public int PropertyB { get; set; }
public string PropertyC { get; set; }
}
class Program
{
static void Main(string[] args)
{
// Show that we can figure out which properties have the custom attribute,
// and that we can get the values for Val1 and Val2
foreach(PropertyInfo propertyInfo in typeof(Foo).GetProperties())
{
Console.WriteLine("Found a property named "+propertyInfo.Name);
foreach(Attribute attribute in propertyInfo.GetCustomAttributes(
attributeType:typeof(MyCustomAttribute),inherit:true))
{
Console.WriteLine("Found a MyCustomAttribute on the property.");
MyCustomAttribute myCustomAttribute = attribute as MyCustomAttribute;
Console.WriteLine("Val1 = " + myCustomAttribute.Val1);
Console.WriteLine("Val2 = " + myCustomAttribute.Val2);
}
Console.WriteLine();
}
// Show that the properties can be used like normal
Foo foo = new Foo {PropertyA = "X", PropertyB = 2, PropertyC = "Z"};
Console.WriteLine("Created an instance of Foo just for fun. Its property values are "+
foo.PropertyA+","+foo.PropertyB+","+foo.PropertyC);
}
}
}
Can this be done?
There is no simple way to do what you want with properties. You can't simply set attributes on or get attributes from items protected by a property.
def declarativeInfo(*args, **kwargs):
def wrapper(obj):
for arg in args:
setattr(obj, arg, arg)
for k, v in kwargs:
setattr(obj, k, v)
return obj
return wrapper
class MyTestClass:
def __init__(self, foo):
print MyTestClass.__dict__
self.radius = self.Radius('foo')
#declarativeInfo(bar="ABC",baz=123)
class Radius(object):
def __init__(self, foo):
self.value = foo
a = MyTestClass('foo')
print a.radius.value
print a.radius.a
is the easiest way to do this. You can always, of course, make value a property.
If you really want radius to be a normal property, you can store the information elsewhere in a dict and retrieve it from self.propdict or something.
OK, I wrote this question when I was first getting started with Python. I now know how to do in Python exactly what the .Net sample code I posted did. Granted, the biggest thing I didn't realize when I originally posted the question was that descriptors alter the behavior of your attributes/properties(whatever you call them). Nonetheless, we can still allow these attributes to act like properties (and not change their behavior) yet put some metadata on them with the decorator. I'm currently implementing some protocol serialization/deserialization stuff where this is going to come in handy.
class MyCustomDescriptor:
def __init__(self,val1,val2):d
self._val1 = val1
self._val2 = val2
#property
def val1(self): return self._val1
#property
def val2(self): return self._val2
def __call__(self,decorated_method_reference):
self._decorated_method_reference = decorated_method_reference
return self
def __get__(self,instance,type=None):
if not instance:
return self
return self._decorated_method_reference(instance)
class Foo:
def __init__(self,attribute_a_value,attribute_b_value,attribute_c_value):
self._attribute_a_value = attribute_a_value
self._attribute_b_value = attribute_b_value
self._attribute_c_value = attribute_c_value
#MyCustomDescriptor(val1="abc",val2="def")
def attribute_a(self): return self._attribute_a_value
#MyCustomDescriptor(val1="xyz",val2="X")
def attribute_b(self): return self._attribute_b_value
#property
def attribute_c(self): return self._attribute_c_value
# Show that by inspecting class Foo we can figure out which attribute are marked with MyCustomDescriptor and that
# we can get the values for val1 and val2. We don't even need an instance of Foo to do this. The class itself is sufficient.
print("Inspecting class Foo. Looking for attributes marked with MyCustomDescriptor...")
for attribute_name in dir(Foo):
attribute_as_object = getattr(Foo,attribute_name)
if type(attribute_as_object) == MyCustomDescriptor:
print("attribute "+attribute_name+" is decorated with MyCustomDescriptor. val1="+attribute_as_object.val1+" val2="+attribute_as_object.val2)
# Show that the properties on Foo work like normal properties. Note that I skipped implementing setters but could have done so.
foo_instance = Foo(attribute_a_value="X",attribute_b_value=2,attribute_c_value="Z")
print("Created an instance of Foo just for fun. It's property values are "+str(foo_instance.attribute_a)+", "+str(foo_instance.attribute_b)+", "+str(foo_instance.attribute_c))
The output is:
Inspecting class Foo.
Looking for attributes marked with MyCustomDescriptor...
attribute attribute_a is decorated with MyCustomDescriptor.
val1=abc val2=def
attribute attribute_b is decorated with MyCustomDescriptor.
val1=xyz val2=X
Created an instance of Foo just for fun.
It's property values are X, 2, Z