Develop Reference

Python 3 Overriding class method with attribute

I have a class with a bunch of properties. I want to override an arbitrary number of them with a dict parsed from a yaml file. I've tried a few approaches inculding __getattributes__ and setting the instance __dict__ with the new variable.
The yaml would look like
property_a: 1
property_b: 2
The first approach I tried with __getattribute__ results in a recursion error because I'm trying to access self.yamlsettings over and over again
import yaml
class Properties(object):
def __init__(self):
with open("config/staging/kjh.yaml") as f:
yamlsettings = yaml.load(f)
self.yamlsettings = yamlsettings
def __getattribute__(self, attr):
try:
return self.yamlsettings[attr]
except KeyError:
return object.__getattribute__(self, attr)
#property
def property_a(self):
return "a"
#property
def property_b(self):
return "b"
#property
def property_c(self):
return "c"
The second approach I tried was setting the instance's dict to the key value pair in the yaml file.
The problem is why I'm trying to access the attribute it calls the property rather than the attribute.
import yaml
class Properties(object):
def __init__(self):
with open("config/staging/kjh.yaml") as f:
yamlsettings = yaml.load(f)
for k, v in yamlsettings.items():
self.__dict__[k] = v
#property
def property_a(self):
return "a"
#property
def property_b(self):
return "b"
#property
def property_c(self):
return "c"
prop = Properties()
prop.__dict__
>> {'property_a': 1, 'property_b': 2}
prop.property_a
>> 'a'
Can anyone point me in the right direction? I think I might be able to achieve this through a getter but it seems extremely verbose because I have so many properties.
Thanks!

To avoid the recursion error, use the superclass (object) method to access self.yamlsettings:
...
def __getatttibute__(self, attr):
try:
return object.__getattribute__(
self, 'yamlsettings'
)[attr]
except KeyError:
return object.__getattribute__(self, attr)

Ruby like DSL in Python

I'm currently writing my first bigger project in Python, and I'm now wondering how to define a class method so that you can execute it in the class body of a subclass of the class.
First to give some more context, a slacked down (I removed everything non essential for this question) example of how I'd do the thing I'm trying to do in Ruby:
If I define a class Item like this:
class Item
def initialize(data={})
#data = data
end
def self.define_field(name)
define_method("#{name}"){ instance_variable_get("#data")[name.to_s] }
define_method("#{name}=") do |value|
instance_variable_get("#data")[name.to_s] = value
end
end
end
I can use it like this:
class MyItem < Item
define_field("name")
end
item = MyItem.new
item.name = "World"
puts "Hello #{item.name}!"
Now so far I tried achieving something similar in Python, but I'm not happy with the result I've got so far:
class ItemField(object):
def __init__(self, name):
self.name = name
def __get__(self, item, owner=None):
return item.values[self.name]
def __set__(self, item, value):
item.values[self.name] = value
def __delete__(self, item):
del item.values[self.name]
class Item(object):
def __init__(self, data=None):
if data == None: data = {}
self.values = data
for field in type(self).fields:
self.values[field.name] = None
setattr(self, field.name, field)
#classmethod
def define_field(cls, name):
if not hasattr(cls, "fields"): cls.fields = []
cls.fields.append(ItemField(name, default))
Now I don't know how I can call define_field from withing a subclass's body. This is what I wished that it was possible:
class MyItem(Item):
define_field("name")
item = MyItem({"name": "World"})
puts "Hello {}!".format(item.name)
item.name = "reader"
puts "Hello {}!".format(item.name)
There's this similar question but none of the answers are really satisfying, somebody recommends caling the function with __func__() but I guess I can't do that, because I can't get a reference to the class from within its anonymous body (please correct me if I'm wrong about this.)
Somebody else pointed out that it's better to use a module level function for doing this which I also think would be the easiest way, however the main intention of me doing this is to make the implementation of subclasses clean and having to load that module function wouldn't be to nice either. (Also I'd have to do the function call outside the class body and I don't know but I think this is messy.)
So basically I think my approach is wrong, because Python wasn't designed to allow this kind of thing to be done. What would be the best way to achieve something as in the Ruby example with Python?
(If there's no better way I've already thought about just having a method in the subclass which returns an array of the parameters for the define_field method.)

Perhaps calling a class method isn't the right route here. I'm not quite up to speed on exactly how and when Python creates classes, but my guess is that the class object doesn't yet exist when you'd call the class method to create an attribute.
It looks like you want to create something like a record. First, note that Python allows you to add attributes to your user-created classes after creation:
class Foo(object):
pass
>>> foo = Foo()
>>> foo.x = 42
>>> foo.x
42
Maybe you want to constrain which attributes the user can set. Here's one way.
class Item(object):
def __init__(self):
if type(self) is Item:
raise NotImplementedError("Item must be subclassed.")
def __setattr__(self, name, value):
if name not in self.fields:
raise AttributeError("Invalid attribute name.")
else:
self.__dict__[name] = value
class MyItem(Item):
fields = ("foo", "bar", "baz")
So that:
>>> m = MyItem()
>>> m.foo = 42 # works
>>> m.bar = "hello" # works
>>> m.test = 12 # raises AttributeError
Lastly, the above allows you the user subclass Item without defining fields, like such:
class MyItem(Item):
pass
This will result in a cryptic attribute error saying that the attribute fields could not be found. You can require that the fields attribute be defined at the time of class creation by using metaclasses. Furthermore, you can abstract away the need for the user to specify the metaclass by inheriting from a superclass that you've written to use the metaclass:
class ItemMetaclass(type):
def __new__(cls, clsname, bases, dct):
if "fields" not in dct:
raise TypeError("Subclass must define 'fields'.")
return type.__new__(cls, clsname, bases, dct)
class Item(object):
__metaclass__ = ItemMetaclass
fields = None
def __init__(self):
if type(self) == Item:
raise NotImplementedError("Must subclass Type.")
def __setattr__(self, name, value):
if name in self.fields:
self.__dict__[name] = value
else:
raise AttributeError("The item has no such attribute.")
class MyItem(Item):
fields = ("one", "two", "three")

You're almost there! If I understand you correctly:
class Item(object):
def __init__(self, data=None):
fields = data or {}
for field, value in data.items():
if hasattr(self, field):
setattr(self, field, value)
#classmethod
def define_field(cls, name):
setattr(cls, name, None)
EDIT: As far as I know, it's not possible to access the class being defined while defining it. You can however call the method on the __init__ method:
class Something(Item):
def __init__(self):
type(self).define_field("name")
But then you're just reinventing the wheel.

When defining a class, you cannot reference the class itself inside its own definition block. So you have to call define_field(...) on MyItem after its definition. E.g.,
class MyItem(Item):
pass
MyItem.define_field("name")
item = MyItem({"name": "World"})
print("Hello {}!".format(item.name))
item.name = "reader"
print("Hello {}!".format(item.name))

Python "callable" attribute (pseudo-property)

In python, I can alter the state of an instance by directly assigning to attributes, or by making method calls which alter the state of the attributes:
foo.thing = 'baz'
or:
foo.thing('baz')
Is there a nice way to create a class which would accept both of the above forms which scales to large numbers of attributes that behave this way? (Shortly, I'll show an example of an implementation that I don't particularly like.) If you're thinking that this is a stupid API, let me know, but perhaps a more concrete example is in order. Say I have a Document class. Document could have an attribute title. However, title may want to have some state as well (font,fontsize,justification,...), but the average user might be happy enough just setting the title to a string and being done with it ...
One way to accomplish this would be to:
class Title(object):
def __init__(self,text,font='times',size=12):
self.text = text
self.font = font
self.size = size
def __call__(self,*text,**kwargs):
if(text):
self.text = text[0]
for k,v in kwargs.items():
setattr(self,k,v)
def __str__(self):
return '<title font={font}, size={size}>{text}</title>'.format(text=self.text,size=self.size,font=self.font)
class Document(object):
_special_attr = set(['title'])
def __setattr__(self,k,v):
if k in self._special_attr and hasattr(self,k):
getattr(self,k)(v)
else:
object.__setattr__(self,k,v)
def __init__(self,text="",title=""):
self.title = Title(title)
self.text = text
def __str__(self):
return str(self.title)+'<body>'+self.text+'</body>'
Now I can use this as follows:
doc = Document()
doc.title = "Hello World"
print (str(doc))
doc.title("Goodbye World",font="Helvetica")
print (str(doc))
This implementation seems a little messy though (with __special_attr). Maybe that's because this is a messed up API. I'm not sure. Is there a better way to do this? Or did I leave the beaten path a little too far on this one?
I realize I could use #property for this as well, but that wouldn't scale well at all if I had more than just one attribute which is to behave this way -- I'd need to write a getter and setter for each, yuck.

It is a bit harder than the previous answers assume.
Any value stored in the descriptor will be shared between all instances, so it is not the right place to store per-instance data.
Also, obj.attrib(...) is performed in two steps:
tmp = obj.attrib
tmp(...)
Python doesn't know in advance that the second step will follow, so you always have to return something that is callable and has a reference to its parent object.
In the following example that reference is implied in the set argument:
class CallableString(str):
def __new__(class_, set, value):
inst = str.__new__(class_, value)
inst._set = set
return inst
def __call__(self, value):
self._set(value)
class A(object):
def __init__(self):
self._attrib = "foo"
def get_attrib(self):
return CallableString(self.set_attrib, self._attrib)
def set_attrib(self, value):
try:
value = value._value
except AttributeError:
pass
self._attrib = value
attrib = property(get_attrib, set_attrib)
a = A()
print a.attrib
a.attrib = "bar"
print a.attrib
a.attrib("baz")
print a.attrib
In short: what you want cannot be done transparently. You'll write better Python code if you don't insist hacking around this limitation

You can avoid having to use #property on potentially hundreds of attributes by simply creating a descriptor class that follows the appropriate rules:
# Warning: Untested code ahead
class DocAttribute(object):
tag_str = "<{tag}{attrs}>{text}</{tag}>"
def __init__(self, tag_name, default_attrs=None):
self._tag_name = tag_name
self._attrs = default_attrs if default_attrs is not None else {}
def __call__(self, *text, **attrs):
self._text = "".join(text)
self._attrs.update(attrs)
return self
def __get__(self, instance, cls):
return self
def __set__(self, instance, value):
self._text = value
def __str__(self):
# Attrs left as an exercise for the reader
return self.tag_str.format(tag=self._tag_name, text=self._text)
Then you can use Document's __setattr__ method to add a descriptor based on this class if it is in a white list of approved names (or not in a black list of forbidden ones, depending on your domain):
class Document(object):
# prelude
def __setattr__(self, name, value):
if self.is_allowed(name): # Again, left as an exercise for the reader
object.__setattr__(self, name, DocAttribute(name)(value))

Use class variables as instance vars?

What I would like to do there is declaring class variables, but actually use them as vars of the instance. I have a class Field and a class Thing, like this:
class Field(object):
def __set__(self, instance, value):
for key, v in vars(instance.__class__).items():
if v is self:
instance.__dict__.update({key: value})
def __get__(self, instance, owner):
for key, v in vars(instance.__class__).items():
if v is self:
try:
return instance.__dict__[key]
except:
return None
class Thing(object):
foo = Field()
So when I instantiate a thing and set attribute foo, it will be added to the instance, not the class, the class variable is never actually re-set.
new = Thing()
new.foo = 'bar'
# (foo : 'bar') is stored in new.__dict__
This works so far, but the above code for Field is rather awkward. It has too look for the Field object instance in the classes props, otherwise there seems no way of knowing the name of the property (foo) in __set__ and __get__. Is there another, more straight forward way to accomplish this?

Every instance of Field (effectively) has a name. Its name is the attribute name (or key) which references it in Thing. Instead of having to look up the key dynamically, you could instantiate Fields with the name at the time the class attribute is set in Thing:
class Field(object):
def __init__(self, name):
self.name = name
def __set__(self, instance, value):
instance.__dict__.update({self.name: value})
def __get__(self, instance, owner):
if instance is None:
return self
try:
return instance.__dict__[self.name]
except KeyError:
return None
def make_field(*args):
def wrapper(cls):
for arg in args:
setattr(cls, arg, Field(arg))
return cls
return wrapper
#make_field('foo')
class Thing(object):
pass
And it can be used like this:
new = Thing()
Before new.foo is set, new.foo returns None:
print(new.foo)
# None
After new.foo is set, 'foo' is an instance attribute of new:
new.foo = 'bar'
print(new.__dict__)
# {'foo': 'bar'}
You can access the descriptor (the Field instance itself) with Thing.foo:
print(Thing.foo)
# <__main__.Field object at 0xb76cedec>
PS. I'm assuming you have a good reason why
class Thing(object):
foo = None
does not suffice.

Reread your question and realized I had it wrong:
You don't need to override the default python behavior to do this. For example, you could do the following:
class Thing(object):
foo = 5
>>> r = Thing()
>>> r.foo = 10
>>> s = Thing()
>>> print Thing.foo
5
>>> print r.foo
10
>>> print s.foo
5
If you want the default to be 'None' for a particular variable, you could just set the class-wide value to be None. That said, you would have to declare it specifically for each variable.

The easiest way would be to call the attribute something else than the name of the descriptor variable - preferably starting with _ to signal its an implementation detail. That way, you end up with:
def __set__(self, instance, value):
instance._foo = value
def __get__(self, instance, owner):
return getattr(instance, '_foo', None)
The only drawback of this is that you can't determine the name of the key from the one used for the descriptor. If that increased coupling isn't a problem compared to the loop, you could just use a property:
class Thing:
#property
def foo(self):
return getattr(self, '_foo', None)
#foo.setter
def foo(self, value):
self._foo = value
otherwise, you could pass the name of the variable into the descriptor's __init__, so that you have:
class Thing:
foo = Field('_foo')
Of course, all this assumes that the simplest and most Pythonic way - use a real variable Thing().foo that you set to None in Thing.__init__ - isn't an option for some reason. If that way will work for you, you should prefer it.

Create per-instance property descriptor?

Usually Python descriptor are defined as class attributes. But in my case, I want every object instance to have different set descriptors that depends on the input. For example:
class MyClass(object):
def __init__(self, **kwargs):
for attr, val in kwargs.items():
self.__dict__[attr] = MyDescriptor(val)
Each object are have different set of attributes that are decided at instantiation time. Since these are one-off objects, it is not convenient to first subclass them.
tv = MyClass(type="tv", size="30")
smartphone = MyClass(type="phone", os="android")
tv.size # do something smart with the descriptor
Assign Descriptor to the object does not seem to work. If I try to access the attribute, I got something like
<property at 0x4067cf0>
Do you know why is this not working? Is there any work around?

This is not working because you have to assign the descriptor to the class of the object.
class Descriptor:
def __get__(...):
# this is called when the value is got
def __set__(...
def __del__(...
if you write
obj.attr
=> type(obj).__getattribute__(obj, 'attr') is called
=> obj.__dict__['attr'] is returned if there else:
=> type(obj).__dict__['attr'] is looked up
if this contains a descriptor object then this is used.
so it does not work because the type dictionairy is looked up for descriptors and not the object dictionairy.
there are possible work arounds:
put the descriptor into the class and make it use e.g. obj.xxxattr to store the value.
If there is only one descriptor behaviour this works.
overwrite setattr and getattr and delattr to respond to discriptors.
put a discriptor into the class that responds to descriptors stored in the object dictionairy.

You are using descriptors in the wrong way.
Descriptors don't make sense on an instance level. After all the __get__/__set__
methods give you access to the instance of the class.
Without knowing what exactly you want to do, I'd suggest you put the per-instance
logic inside the __set__ method, by checking who is the "caller/instance" and act accordingly.
Otherwise tell us what you are trying to achieve, so that we can propose alternative solutions.

I dynamically create instances by execing a made-up class. This may suit your use case.
def make_myclass(**kwargs):
class MyDescriptor(object):
def __init__(self, val):
self.val = val
def __get__(self, obj, cls):
return self.val
def __set__(self, obj, val):
self.val = val
cls = 'class MyClass(object):\n{}'.format('\n'.join(' {0} = MyDescriptor({0})'.format(k) for k in kwargs))
#check if names in kwargs collide with local names
for key in kwargs:
if key in locals():
raise Exception('name "{}" collides with local name'.format(key))
kwargs.update(locals())
exec(cls, kwargs, locals())
return MyClass()
Test;
In [577]: tv = make_myclass(type="tv", size="30")
In [578]: tv.type
Out[578]: 'tv'
In [579]: tv.size
Out[579]: '30'
In [580]: tv.__dict__
Out[580]: {}
But the instances are of different class.
In [581]: phone = make_myclass(type='phone')
In [582]: phone.type
Out[582]: 'phone'
In [583]: tv.type
Out[583]: 'tv'
In [584]: isinstance(tv,type(phone))
Out[584]: False
In [585]: isinstance(phone,type(tv))
Out[585]: False
In [586]: type(tv)
Out[586]: MyClass
In [587]: type(phone)
Out[587]: MyClass
In [588]: type(phone) is type(tv)
Out[588]: False

This looks like a use-case for named tuples

The reason it is not working is because Python only checks for descriptors when looking up attributes on the class, not on the instance; the methods in question are:
__getattribute__
__setattr__
__delattr__
It is possible to override those methods on your class in order to implement the descriptor protocol on instances as well as classes:
# do not use in production, example code only, needs more checks
class ClassAllowingInstanceDescriptors(object):
def __delattr__(self, name):
res = self.__dict__.get(name)
for method in ('__get__', '__set__', '__delete__'):
if hasattr(res, method):
# we have a descriptor, use it
res = res.__delete__(name)
break
else:
res = object.__delattr__(self, name)
return res
def __getattribute__(self, *args):
res = object.__getattribute__(self, *args)
for method in ('__get__', '__set__', '__delete__'):
if hasattr(res, method):
# we have a descriptor, call it
res = res.__get__(self, self.__class__)
return res
def __setattr__(self, name, val):
# check if object already exists
res = self.__dict__.get(name)
for method in ('__get__', '__set__', '__delete__'):
if hasattr(res, method):
# we have a descriptor, use it
res = res.__set__(self, val)
break
else:
res = object.__setattr__(self, name, val)
return res
#property
def world(self):
return 'hello!'
When the above class is used as below:
huh = ClassAllowingInstanceDescriptors()
print(huh.world)
huh.uni = 'BIG'
print(huh.uni)
huh.huh = property(lambda *a: 'really?')
print(huh.huh)
print('*' * 50)
try:
del huh.world
except Exception, e:
print(e)
print(huh.world)
print('*' * 50)
try:
del huh.huh
except Exception, e:
print(e)
print(huh.huh)
The results are:
hello!
BIG
really?
can't delete attribute
hello!
can't delete attribute
really?