Related
I've been going through all the Stackoverflow answers on dynamic property setting, but for whatever reason I can't seem to get this to work.
I have a class, Evolution_Base, that in its init creates an instance of Value_Differences. Value_Differences should be dynamically creating properties, based on the list I pass, that returns the function value from _get_df_change:
from pandas import DataFrame
from dataclasses import dataclass
import pandas as pd
class Evolution_Base():
def __init__(self, res_date_0 : DataFrame , res_date_1 : DataFrame):
#dataclass
class Results_Data():
res_date_0_df : DataFrame
res_date_1_df : DataFrame
self.res = Results_Data(res_date_0_df= res_date_0,
res_date_1_df= res_date_1)
property_list = ['abc', 'xyz']
self.difference = Value_Differences(parent = self, property_list=property_list)
# Shared Functions
def _get_df_change(self, df_name, operator = '-'):
df_0 = getattr(self.res.res_date_0_df, df_name.lower())
df_1 = getattr(self.res.res_date_1_df, df_name.lower())
return self._df_change(df_1, df_0, operator=operator)
def _df_change(self, df_1 : pd.DataFrame, df_0 : pd.DataFrame, operator = '-') -> pd.DataFrame:
"""
Returns df_1 <operator | default = -> df_0
"""
# is_numeric mask
m_1 = df_1.select_dtypes('number')
m_0 = df_0.select_dtypes('number')
def label_me(x):
x.columns = ['t_1', 't_0']
return x
if operator == '-':
return label_me(df_1[m_1] - df_0[m_0])
elif operator == '+':
return label_me(df_1[m_1] + df_0[m_0])
class Value_Differences():
def __init__(self, parent : Evolution_Base, property_list = []):
self._parent = parent
for name in property_list:
def func(self, prop_name):
return self._parent._get_df_change(name)
# I've tried the following...
setattr(self, name, property(fget = lambda cls_self: func(cls_self, name)))
setattr(self, name, property(func(self, name)))
setattr(self, name, property(func))
Its driving me nuts... Any help appreciated!
My desired outcome is for:
evolution = Evolution_Base(df_1, df_2)
evolution.difference.abc == evolution._df_change('abc')
evolution.difference.xyz == evolution._df_change('xyz')
EDIT: The simple question is really, how do I setattr for a property function?
As asked
how do I setattr for a property function?
To be usable as a property, the accessor function needs to be wrapped as a property and then assigned as an attribute of the class, not the instance.
That function, meanwhile, needs to have a single unbound parameter - which will be an instance of the class, but is not necessarily the current self. Its logic needs to use the current value of name, but late binding will be an issue because of the desire to create lambdas in a loop.
A clear and simple way to work around this is to define a helper function accepting the Value_Differences instance and the name to use, and then bind the name value eagerly.
Naively:
from functools import partial
def _get_from_parent(name, instance):
return instance._parent._get_df_change(name)
class Value_Differences:
def __init__(self, parent: Evolution_Base, property_list = []):
self._parent = parent
for name in property_list:
setattr(Value_Differences, name, property(
fget = partial(_get_from_parent, name)
))
However, this of course has the issue that every instance of Value_Differences will set properties on the class, thus modifying what properties are available for each other instance. Further, in the case where there are many instances that should have the same properties, the setup work will be repeated at each instance creation.
The apparent goal
It seems that what is really sought, is the ability to create classes dynamically, such that a list of property names is provided and a corresponding class pops into existence, with code filled in for the properties implementing a certain logic.
There are multiple approaches to this.
Factory A: Adding properties to an instantiated template
Just like how functions can be nested within each other and the inner function will be an object that can be modified and returned (as is common when creating a decorator), a class body can appear within a function and a new class object (with the same name) is created every time the function runs. (The code in the OP already does this, for the Results_Data dataclass.)
def example():
class Template:
pass
return Template
>>> TemplateA, TemplateB = example(), example()
>>> TemplateA is TemplateB
False
>>> isinstance(TemplateA(), TemplateB)
False
>>> isinstance(TemplateB(), TemplateA)
False
So, a "factory" for value-difference classes could look like
from functools import partial
def _make_value_comparer(property_names, access_func):
class ValueDifferences:
def __init__(self, parent):
self._parent = parent
for name in property_names:
setattr(Value_Differences, name, property(
fget = partial(access_func, name)
))
return ValueDifferences
Notice that instead of hard-coding a helper, this factory expects to be provided with a function that implements the access logic. That function takes two parameters: a property name, and the ValueDifferences instance. (They're in that order because it's more convenient for functools.partial usage.)
Factory B: Using the type constructor directly
The built-in type in Python has two entirely separate functions.
With one argument, it discloses the type of an object.
With three arguments, it creates a new type. The class syntax is in fact syntactic sugar for a call to this builtin. The arguments are:
a string name (will be set as the __name__ attribute)
a list of classes to use as superclasses (will be set as __bases__)
a dict mapping attribute names to their values (including methods and properties - will become the __dict__, roughly)
In this style, the same factory could look something like:
from functools import partial
def _make_value_comparer(property_names, access_func):
methods = {
name: property(fget = partial(access_func, name)
for name in property_names
}
methods['__init__'] = lambda self, parent: setattr(self, '_parent', parent)
return type('ValueDifferences', [], methods)
Using the factory
In either of the above cases, EvolutionBase would be modified in the same way.
Presumably, every EvolutionBase should use the same ValueDifferences class (i.e., the one that specifically defines abc and xyz properties), so the EvolutionBase class can cache that class as a class attribute, and use it later:
class Evolution_Base():
def _get_from_parent(name, mvd):
# mvd._parent will be an instance of Evolution_Base.
return mvd._parent._get_df_change(name)
_MyValueDifferences = _make_value_comparer(['abc', 'xyz'], _get_from_parent)
def __init__(self, res_date_0 : DataFrame , res_date_1 : DataFrame):
#dataclass
class Results_Data():
res_date_0_df : DataFrame
res_date_1_df : DataFrame
self.res = Results_Data(res_date_0_df= res_date_0,
res_date_1_df= res_date_1)
self.difference = _MyValueDifferences(parent = self)
Notice that the cached _MyValueDifferences class no longer requires a list of property names to be constructed. That's because it was already provided when the class was created. The actual thing that varies per instance of _MyValueDifferences, is the parent, so that's all that gets passed.
Simpler approaches
It seems that the goal is to have a class whose instances are tightly associated with instances of Evolution_Base, providing properties specifically named abc and xyz that are computed using the Evolution_Base's data.
That could just be hard-coded as a nested class:
class Evolution_Base:
class EBValueDifferences:
def __init__(self, parent):
self._parent = parent
#property
def abc(self):
return self._parent._get_df_change('abc')
#property
def xyz(self):
return self._parent._get_df_change('xyz')
def __init__(self, res_date_0 : DataFrame , res_date_1 : DataFrame):
#dataclass
class Results_Data():
res_date_0_df : DataFrame
res_date_1_df : DataFrame
self.res = Results_Data(res_date_0_df = res_date_0,
res_date_1_df = res_date_1)
self.difference = EBValueDifferences(self)
# _get_df_change etc. as before
Even simpler, provide corresponding properties directly on Evolution_Base:
class Evolution_Base:
#property
def abc_difference(self):
return self._get_df_change('abc')
#property
def xyz_difference(self):
return self._get_df_change('xyz')
def __init__(self, res_date_0 : DataFrame , res_date_1 : DataFrame):
#dataclass
class Results_Data():
res_date_0_df : DataFrame
res_date_1_df : DataFrame
self.res = Results_Data(res_date_0_df = res_date_0,
res_date_1_df = res_date_1)
# _get_df_change etc. as before
# client code now calls my_evolution_base.abc_difference
# instead of my_evolution_base.difference.abc
If there are a lot of such properties, they could be attached using a much simpler dynamic approach (that would still be reusable for other classes that define a _get_df_change):
def add_df_change_property(name, cls):
setattr(
cls, f'{name}_difference',
property(fget = lambda instance: instance._get_df_change(name))
)
which can also be adapted for use as a decorator:
from functools import partial
def exposes_df_change(name):
return partial(add_df_change_property, name)
#exposes_df_change('abc')
#exposes_df_change('def')
class Evolution_Base:
# `self.difference` can be removed, no other changes needed
This is quite the rabbit hole. Impossible is a big call, but I will say this: they don't intend you to do this. The 'Pythonic' way of achieving your example use case is the __getattr__ method. You could also override the __dir__ method to insert your custom attributes for discoverability.
This is the code for that:
class Value_Differences():
def __init__(self, parent : Evolution_Base, property_list = []):
self._parent = parent
self._property_list = property_list
def __dir__(self):
return sorted(set(
dir(super(Value_Differences, self)) + \
list(self.__dict__.keys()) + self._property_list))
def __getattr__(self, __name: str):
if __name in self._property_list:
return self._parent._get_df_change(__name)
But that wasn't the question, and respect for a really, really interesting question. This is one of those things that you look at and say 'hmm, should be possible' and can get almost to a solution. I initially thought what you asked for was technically possible, just very hacky to achieve. But it turns out that it would be very, very weird hackery if it was possible.
Two small foundational things to start with:
Remind ourselves of the hierarchy of Python objects that the runtime is working with when defining and instantiating classes:
The metaclass (defaulting to type), which is used to build classes. I'm going to refer to this as the Metaclass Type Object (MTO).
The class definition, which is used to build objects. I'm going to refer to this as the Class Type Object (CTO).
And the class instance or object, which I'll refer to as the Class Instance Object (CIO).
MTOs are subclasses of type. CTOs are subclasses of object. CIOs are instances of CTOs, but instantiated by MTOs.
Python runs code inside class definitions as if it was running a function:
class Class1:
print("1")
def __init__(self, v1):
print("4")
print("2")
print("3")
c1 = Class1("x")
print("5")
gives 1, 2, 3, 4, 5
Put these two things together with:
class Class1:
def attr1_get(self):
return 'attr1 value'
attr1 = property(attr1_get)
we are defining a function attr1_get as part of the class definition. We are then running an inline piece of code that creates an object of type property. Note that this is just the name of the object's type - it isn't a property as you would describe it. Just an object with some attributes, being references to various functions. We then assign that object to an attribute in the class we are defining.
In the terms I used above, once that code is run we have a CTO instantiated as an object in memory that contains an attribute attr1 of type property (an object subclass, containing a bunch of attributes itself - one of which is a reference to the function attr1_get).
That can be used to instantiate an object, the CIO.
This is where the MTO comes in. You instantiate the property object while defining the CTO so that when the runtime applies the MTO to create the CIO from the CTO, an attribute on the CIO will be formed with a custom getter function for that attribute rather than the 'standard' getter function the runtime would use. The property object means something to the type object when it is building a new object.
So when we run:
c1 = Class1()
we don't get a CIO c1 with an attribute attr1 that is an object of type property. The metaclass of type type formed a set of references against the attribute's internal state to all the functions we stored in the property object. Note that this is happening inside the runtime, and you can't call this directly from your code - you just tell the type metaclass to do it by using the property wrapper object.
So if you directly assign a property() result to an attribute of a CIO, you have a Pythonic object assigned that references some functions, but the internal state for the runtime to use to reference the getter, setter, etc. is not set up. The getter of an attribute that contains a property object is the standard getter and so returns the object instance, and not the result of the functions it wraps,
This next bit of code demonstrates how this flows:
print("Let's begin")
class MetaClass1(type):
print("Starting to define MetaClass1")
def __new__(cls, name, bases, dct):
x = super().__new__(cls, name, bases, dct)
print("Metaclass1 __new__({})".format(str(cls)))
return x
print("__new__ of MetaClass1 is defined")
def __init__(cls, name, bases, dct):
print("Metaclass1 __init__({})".format(str(cls)))
print("__init__ of MetaClass1 is defined")
print("Metaclass is defined")
class Class1(object,metaclass=MetaClass1):
print("Starting to define Class1")
def __new__(cls, *args, **kwargs):
print("Class1 __new__({})".format(str(cls)))
return super(Class1, cls).__new__(cls, *args, **kwargs)
print("__new__ of Class1 is defined")
def __init__(self):
print("Class1 __init__({})".format(str(self)))
print("__init__ of Class1 is defined")
def g1(self):
return 'attr1 value'
print("g1 of Class1 is defined")
attr1 = property(g1)
print("Class1.attr1 = ", attr1)
print("attr1 of Class1 is defined")
def addProperty(self, name, getter):
setattr(self, name, property(getter))
print("self.", name, " = ", getattr(self, name))
print("addProperty of Class1 is defined")
print("Class is defined")
c1 = Class1()
print("Instance is created")
print(c1.attr1)
def g2(cls):
return 'attr2 value'
c1.addProperty('attr2', g2)
print(c1.attr2)
I have put all those print statements there to demonstrate the order in which things happen very clearly.
In the middle, you see:
g1 of Class1 is defined
Class1.attr1 = <property object at 0x105115c10>
attr1 of Class1 is defined
We have created an object of type property and assigned it to a class attribute.
Continuing:
addProperty of Class1 is defined
Metaclass1 __new__(<class '__main__.MetaClass1'>)
Metaclass1 __init__(<class '__main__.Class1'>)
Class is defined
The metaclass got instantiated, being passed first itself (__new__) and then the class it will work on (__init__). This happened right as we stepped out of the class definition. I have only included the metaclass to show what will happen with the type metaclass by default.
Then:
Class1 __new__(<class '__main__.Class1'>)
Class1 __init__(<__main__.Class1 object at 0x105124c10>)
Instance is created
attr1 value
self. attr2 = <property object at 0x105115cb0>
<property object at 0x105115cb0>
Class1 is instantiated, providing first its type to __new__ and then its instance to __init__.
We see that attr1 is instantiated properly, but attr2 is not. That is because setattr is being called once the class instance is already constructed and is just saying attr2 is an instance of the class property and not defining attr2 as the actual runtime construct of a property.
Which is made more clear if we run:
print(c1.attr2.fget(c1))
print(c1.attr1.fget(c1))
attr2 (a property object) isn't aware of the class or instance of the containing attribute's parent. The function it wraps still needs to be given the instance to work on.
attr1 doesn't know what to do with that, because as far as it is concerned it is a string object, and has no concept of how the runtime is mapping its getter.
The fundamental reason why what you tried doesn't work is that a property, a use case of a descriptor, by design must be stored as a class variable, not as an instance attribute.
Excerpt from the documentation of descriptor:
To use the descriptor, it must be stored as a class variable in
another class:
To create a class with dynamically named properties that has access to a parent class, one elegant approach is to create the class within a method of the main class, and use setattr to create class attributes with dynamic names and property objects. A class created in the closure of a method automatically has access to the self object of the parent instance, avoiding having to manage a clunky _parent attribute like you do in your attempt:
class Evolution_Base:
def __init__(self, property_list):
self.property_list = property_list
self._difference = None
#property
def difference(self):
if not self._difference:
class Value_Differences:
pass
for name in self.property_list:
# use default value to store the value of name in each iteration
def func(obj, prop_name=name):
return self._get_df_change(prop_name) # access self via closure
setattr(Value_Differences, name, property(func))
self._difference = Value_Differences()
return self._difference
def _get_df_change(self, df_name):
return f'df change of {df_name}' # simplified return value for demo purposes
so that:
evolution = Evolution_Base(['abc', 'xyz'])
print(evolution.difference.abc)
print(evolution.difference.xyz)
would output:
df change of abc
df change of xyz
Demo: https://replit.com/#blhsing/ExtralargeNaturalCoordinate
Responding directly to your question, you can create a class:
class FooBar:
def __init__(self, props):
def make_prop(name):
return property(lambda accessor_self: self._prop_impl(name))
self.accessor = type(
'Accessor',
tuple(),
{p: make_prop(p) for p in props}
)()
def _prop_impl(self, arg):
return arg
o = FooBar(['foo', 'bar'])
assert o.accessor.foo == o._prop_impl('foo')
assert o.accessor.bar == o._prop_impl('bar')
Further, it would be beneficiary to cache created class to make equivalent objects more similar and eliminate potential issues with equality comparison.
That said, I am not sure if this is desired. There's little benefit of replacing method call syntax (o.f('a')) with property access (o.a). I believe it can be detrimental on multiple accounts: dynamic properties are confusing, harder to document, etc., finally while none of this is strictly guaranteed in crazy world of dynamic python -- they kind of communicate wrong message: that the access is cheap and does not involve computation and that perhaps you can attempt to write to it.
I think that when you define the function func in the loop, it closes over the current value of the name variable, not the value of the name variable at the time the property is accessed. To fix this, you can use a lambda function to create a closure that captures the value of name at the time the property is defined.
class Value_Differences():
def __init__(self, parent : Evolution_Base, property_list = []):
self._parent = parent
for name in property_list:
setattr(self, name, property(fget = lambda self, name=name: self._parent._get_df_change(name)))
Does this help you ?
The simple question is really, how do I setattr for a property function?
In python we can set dynamic attributes like this:
class DynamicProperties():
def __init__(self, property_list):
self.property_list = property_list
def add_properties(self):
for name in self.property_list:
setattr(self.__class__, name, property(fget=lambda self: 1))
dync = DynamicProperties(['a', 'b'])
dync.add_properties()
print(dync.a) # prints 1
print(dync.b) # prints 1
Correct me if I am wrong but from reviewing your code, you want to create a dynamic attributes then set their value to a specific function call within the same class, where the passed in data is passed in attributes in the constructor " init " this is achievable, an example:
class DynamicProperties():
def __init__(self, property_list, data1, data2):
self.property_list = property_list
self.data1 = data1
self.data2 = data2
def add_properties(self):
for name in self.property_list:
setattr(self.__class__, name, property(fget=lambda self: self.change(self.data1, self.data2) ))
def change(self, data1, data2):
return data1 - data2
dync = DynamicProperties(['a', 'b'], 1, 2)
dync.add_properties()
print(dync.a == dync.change(1, 2)) # prints true
print(dync.b == dync.change(1,2)) # prints true
You just have to add more complexity to the member, __getattr__ / __setattr__ gives you the string, so it can be interpreted as needed. The biggest "problem" doing this is that the return might no be consistent and piping it back to a library that expect an object to have a specific behavior can cause soft errors.
This example is not the same as yours, but it has the same concept, manipulate columns with members. To get a copy with changes a set is not needed, with a copy, modify and return, the new instance can be created with whatever needed.
For example, the __getattr__ in this line will:
Check and interpret the string xyz_mull_0
Validate that the members and the operand exists
Make a copy of data_a
Modify the copy and return it
var = data_a.xyz_mull_0()
This looks more complex that it actually is, with the same instance members its clear what it is doing, but the _of modifier needs a callback, this is because the __getattr__ can only have one parameter, so it needs to save the attr and return a callback to be called with the other instance that then will call back to the __getattr__ and complete the rest of the function.
import re
class FlexibleFrame:
operand_mod = {
'sub': lambda a, b: a - b,
'add': lambda a, b: a + b,
'div': lambda a, b: a / b,
'mod': lambda a, b: a % b,
'mull': lambda a, b: a * b,
}
#staticmethod
def add_operand(name, func):
if name not in FlexibleFrame.operand_mod.keys():
FlexibleFrame.operand_mod[name] = func
# This makes this class subscriptable
def __getitem__(self, item):
return self.__dict__[item]
# Uses:
# -> object.value
# -> object.member()
# -> object.<name>_<operand>_<name|int>()
# -> object.<name>_<operand>_<name|int>_<flow>()
def __getattr__(self, attr):
if re.match(r'^[a-zA-Z]+_[a-zA-Z]+_[a-zA-Z0-9]+(_of)?$', attr):
seg = attr.split('_')
var_a, operand, var_b = seg[0:3]
# If there is a _of: the second operand is from the other
# instance, the _of is removed and a callback is returned
if len(seg) == 4:
self.__attr_ref = '_'.join(seg[0:3])
return self.__getattr_of
# Checks if this was a _of attribute and resets it
if self.__back_ref is not None:
other = self.__back_ref
self.__back_ref = None
self.__attr_ref = None
else:
other = self
if var_a not in self.__dict__:
raise AttributeError(
f'No match of {var_a} in (primary) {__class__.__name__}'
)
if operand not in FlexibleFrame.operand_mod.keys():
raise AttributeError(
f'No match of operand {operand}'
)
# The return is a copy of self, if not the instance
# is getting modified making x = a.b() useless
ret = FlexibleFrame(**self.__dict__)
# Checks if the second operand is a int
if re.match(r'^\d+$', var_b) :
ref_b_num = int(var_b)
for i in range(len(self[var_a])):
ret[var_a][i] = FlexibleFrame.operand_mod[operand](
self[var_a][i], ref_b_num
)
elif var_b in other.__dict__:
for i in range(len(self[var_a])):
# out_index = operand[type](in_a_index, in_b_index)
ret[var_a][i] = FlexibleFrame.operand_mod[operand](
self[var_a][i], other[var_b][i]
)
else:
raise AttributeError(
f'No match of {var_b} in (secondary) {__class__.__name__}'
)
# This swaps the .member to a .member()
# it also adds and extra () in __getattr_of
return lambda: ret
# return ret
if attr in self.__dict__:
return self[attr]
raise AttributeError(
f'No match of {attr} in {__class__.__name__}'
)
def __getattr_of(self, other):
self.__back_ref = other
return self.__getattr__(self.__attr_ref)()
def __init__(self, **kwargs):
self.__back_ref = None
self.__attr_ref = None
#TODO: Check if data columns match in size
# if not, implement column_<name>_filler=<default>
for i in kwargs:
self.__dict__[i] = kwargs[i]
if __name__ == '__main__':
data_a = FlexibleFrame(**{
'abc': [i for i in range(10)],
'nmv': [i for i in range(10)],
'xyz': [i for i in range(10)],
})
data_b = FlexibleFrame(**{
'fee': [i + 10 for i in range(10)],
'foo': [i + 10 for i in range(10)],
})
FlexibleFrame.add_operand('set', lambda a, b: b)
var = data_a.xyz_mull_0()
var = var.abc_set_xyz()
var = var.xyz_add_fee_of(data_b)
As a extra thing, lambdas in python have this thing, so it can make difficult using them when self changes.
It seems you're bending the language to do weird things. I'd take it as a smell that your code is probably getting convoluted but I'm not saying there would never be a use-case for it so here is a minimal example of how to do it:
class Obj:
def _df_change(self, arg):
print('change', arg)
class DynAttributes(Obj):
def __getattr__(self, name):
return self._df_change(name)
class Something:
difference = DynAttributes()
a = Something()
b = Obj()
assert a.difference.hello == b._df_change('hello')
When calling setattr , use self.__class__ instead of self
Code sample:
class A:
def __init__(self,names : List[str]):
for name in names:
setattr(self.__class__,name,property(fget=self.__create_getter(name)))
def __create_getter(self,name: str):
def inner(self):
print(f"invoking {name}")
return 10
return inner
a = A(['x','y'])
print(a.x + 1)
print(a.y + 2)
I am trying to understand what Python's descriptors are and what they are useful for. I understand how they work, but here are my doubts. Consider the following code:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
What is instance and owner here? (in __get__). What is the purpose of these parameters?
How would I call/use this example?
The descriptor is how Python's property type is implemented. A descriptor simply implements __get__, __set__, etc. and is then added to another class in its definition (as you did above with the Temperature class). For example:
temp=Temperature()
temp.celsius #calls celsius.__get__
Accessing the property you assigned the descriptor to (celsius in the above example) calls the appropriate descriptor method.
instance in __get__ is the instance of the class (so above, __get__ would receive temp, while owner is the class with the descriptor (so it would be Temperature).
You need to use a descriptor class to encapsulate the logic that powers it. That way, if the descriptor is used to cache some expensive operation (for example), it could store the value on itself and not its class.
An article about descriptors can be found here.
EDIT: As jchl pointed out in the comments, if you simply try Temperature.celsius, instance will be None.
Why do I need the descriptor class?
It gives you extra control over how attributes work. If you're used to getters and setters in Java, for example, then it's Python's way of doing that. One advantage is that it looks to users just like an attribute (there's no change in syntax). So you can start with an ordinary attribute and then, when you need to do something fancy, switch to a descriptor.
An attribute is just a mutable value. A descriptor lets you execute arbitrary code when reading or setting (or deleting) a value. So you could imagine using it to map an attribute to a field in a database, for example – a kind of ORM.
Another use might be refusing to accept a new value by throwing an exception in __set__ – effectively making the "attribute" read only.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
This is pretty subtle (and the reason I am writing a new answer here - I found this question while wondering the same thing and didn't find the existing answer that great).
A descriptor is defined on a class, but is typically called from an instance. When it's called from an instance both instance and owner are set (and you can work out owner from instance so it seems kinda pointless). But when called from a class, only owner is set – which is why it's there.
This is only needed for __get__ because it's the only one that can be called on a class. If you set the class value you set the descriptor itself. Similarly for deletion. Which is why the owner isn't needed there.
How would I call/use this example?
Well, here's a cool trick using similar classes:
class Celsius:
def __get__(self, instance, owner):
return 5 * (instance.fahrenheit - 32) / 9
def __set__(self, instance, value):
instance.fahrenheit = 32 + 9 * value / 5
class Temperature:
celsius = Celsius()
def __init__(self, initial_f):
self.fahrenheit = initial_f
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
(I'm using Python 3; for python 2 you need to make sure those divisions are / 5.0 and / 9.0). That gives:
100.0
32.0
Now there are other, arguably better ways to achieve the same effect in python (e.g. if celsius were a property, which is the same basic mechanism but places all the source inside the Temperature class), but that shows what can be done...
I am trying to understand what Python's descriptors are and what they can be useful for.
Descriptors are objects in a class namespace that manage instance attributes (like slots, properties, or methods). For example:
class HasDescriptors:
__slots__ = 'a_slot' # creates a descriptor
def a_method(self): # creates a descriptor
"a regular method"
#staticmethod # creates a descriptor
def a_static_method():
"a static method"
#classmethod # creates a descriptor
def a_class_method(cls):
"a class method"
#property # creates a descriptor
def a_property(self):
"a property"
# even a regular function:
def a_function(some_obj_or_self): # creates a descriptor
"create a function suitable for monkey patching"
HasDescriptors.a_function = a_function # (but we usually don't do this)
Pedantically, descriptors are objects with any of the following special methods, which may be known as "descriptor methods":
__get__: non-data descriptor method, for example on a method/function
__set__: data descriptor method, for example on a property instance or slot
__delete__: data descriptor method, again used by properties or slots
These descriptor objects are attributes in other object class namespaces. That is, they live in the __dict__ of the class object.
Descriptor objects programmatically manage the results of a dotted lookup (e.g. foo.descriptor) in a normal expression, an assignment, or a deletion.
Functions/methods, bound methods, property, classmethod, and staticmethod all use these special methods to control how they are accessed via the dotted lookup.
A data descriptor, like property, can allow for lazy evaluation of attributes based on a simpler state of the object, allowing instances to use less memory than if you precomputed each possible attribute.
Another data descriptor, a member_descriptor created by __slots__, allows memory savings (and faster lookups) by having the class store data in a mutable tuple-like datastructure instead of the more flexible but space-consuming __dict__.
Non-data descriptors, instance and class methods, get their implicit first arguments (usually named self and cls, respectively) from their non-data descriptor method, __get__ - and this is how static methods know not to have an implicit first argument.
Most users of Python need to learn only the high-level usage of descriptors, and have no need to learn or understand the implementation of descriptors further.
But understanding how descriptors work can give one greater confidence in one's mastery of Python.
In Depth: What Are Descriptors?
A descriptor is an object with any of the following methods (__get__, __set__, or __delete__), intended to be used via dotted-lookup as if it were a typical attribute of an instance. For an owner-object, obj_instance, with a descriptor object:
obj_instance.descriptor invokes
descriptor.__get__(self, obj_instance, owner_class) returning a value
This is how all methods and the get on a property work.
obj_instance.descriptor = value invokes
descriptor.__set__(self, obj_instance, value) returning None
This is how the setter on a property works.
del obj_instance.descriptor invokes
descriptor.__delete__(self, obj_instance) returning None
This is how the deleter on a property works.
obj_instance is the instance whose class contains the descriptor object's instance. self is the instance of the descriptor (probably just one for the class of the obj_instance)
To define this with code, an object is a descriptor if the set of its attributes intersects with any of the required attributes:
def has_descriptor_attrs(obj):
return set(['__get__', '__set__', '__delete__']).intersection(dir(obj))
def is_descriptor(obj):
"""obj can be instance of descriptor or the descriptor class"""
return bool(has_descriptor_attrs(obj))
A Data Descriptor has a __set__ and/or __delete__.
A Non-Data-Descriptor has neither __set__ nor __delete__.
def has_data_descriptor_attrs(obj):
return set(['__set__', '__delete__']) & set(dir(obj))
def is_data_descriptor(obj):
return bool(has_data_descriptor_attrs(obj))
Builtin Descriptor Object Examples:
classmethod
staticmethod
property
functions in general
Non-Data Descriptors
We can see that classmethod and staticmethod are Non-Data-Descriptors:
>>> is_descriptor(classmethod), is_data_descriptor(classmethod)
(True, False)
>>> is_descriptor(staticmethod), is_data_descriptor(staticmethod)
(True, False)
Both only have the __get__ method:
>>> has_descriptor_attrs(classmethod), has_descriptor_attrs(staticmethod)
(set(['__get__']), set(['__get__']))
Note that all functions are also Non-Data-Descriptors:
>>> def foo(): pass
...
>>> is_descriptor(foo), is_data_descriptor(foo)
(True, False)
Data Descriptor, property
However, property is a Data-Descriptor:
>>> is_data_descriptor(property)
True
>>> has_descriptor_attrs(property)
set(['__set__', '__get__', '__delete__'])
Dotted Lookup Order
These are important distinctions, as they affect the lookup order for a dotted lookup.
obj_instance.attribute
First the above looks to see if the attribute is a Data-Descriptor on the class of the instance,
If not, it looks to see if the attribute is in the obj_instance's __dict__, then
it finally falls back to a Non-Data-Descriptor.
The consequence of this lookup order is that Non-Data-Descriptors like functions/methods can be overridden by instances.
Recap and Next Steps
We have learned that descriptors are objects with any of __get__, __set__, or __delete__. These descriptor objects can be used as attributes on other object class definitions. Now we will look at how they are used, using your code as an example.
Analysis of Code from the Question
Here's your code, followed by your questions and answers to each:
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)
class Temperature(object):
celsius = Celsius()
Why do I need the descriptor class?
Your descriptor ensures you always have a float for this class attribute of Temperature, and that you can't use del to delete the attribute:
>>> t1 = Temperature()
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
Otherwise, your descriptors ignore the owner-class and instances of the owner, instead, storing state in the descriptor. You could just as easily share state across all instances with a simple class attribute (so long as you always set it as a float to the class and never delete it, or are comfortable with users of your code doing so):
class Temperature(object):
celsius = 0.0
This gets you exactly the same behavior as your example (see response to question 3 below), but uses a Pythons builtin (property), and would be considered more idiomatic:
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
What is instance and owner here? (in get). What is the purpose of these parameters?
instance is the instance of the owner that is calling the descriptor. The owner is the class in which the descriptor object is used to manage access to the data point. See the descriptions of the special methods that define descriptors next to the first paragraph of this answer for more descriptive variable names.
How would I call/use this example?
Here's a demonstration:
>>> t1 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1
>>>
>>> t1.celsius
1.0
>>> t2 = Temperature()
>>> t2.celsius
1.0
You can't delete the attribute:
>>> del t2.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __delete__
And you can't assign a variable that can't be converted to a float:
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __set__
ValueError: invalid literal for float(): 0x02
Otherwise, what you have here is a global state for all instances, that is managed by assigning to any instance.
The expected way that most experienced Python programmers would accomplish this outcome would be to use the property decorator, which makes use of the same descriptors under the hood, but brings the behavior into the implementation of the owner class (again, as defined above):
class Temperature(object):
_celsius = 0.0
#property
def celsius(self):
return type(self)._celsius
#celsius.setter
def celsius(self, value):
type(self)._celsius = float(value)
Which has the exact same expected behavior of the original piece of code:
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius
0.0
>>> t1.celsius = 1.0
>>> t2.celsius
1.0
>>> del t1.celsius
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: can't delete attribute
>>> t1.celsius = '0x02'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in celsius
ValueError: invalid literal for float(): 0x02
Conclusion
We've covered the attributes that define descriptors, the difference between data- and non-data-descriptors, builtin objects that use them, and specific questions about use.
So again, how would you use the question's example? I hope you wouldn't. I hope you would start with my first suggestion (a simple class attribute) and move on to the second suggestion (the property decorator) if you feel it is necessary.
Before going into the details of descriptors it may be important to know how attribute lookup in Python works. This assumes that the class has no metaclass and that it uses the default implementation of __getattribute__ (both can be used to "customize" the behavior).
The best illustration of attribute lookup (in Python 3.x or for new-style classes in Python 2.x) in this case is from Understanding Python metaclasses (ionel's codelog). The image uses : as substitute for "non-customizable attribute lookup".
This represents the lookup of an attribute foobar on an instance of Class:
Two conditions are important here:
If the class of instance has an entry for the attribute name and it has __get__ and __set__.
If the instance has no entry for the attribute name but the class has one and it has __get__.
That's where descriptors come into it:
Data descriptors which have both __get__ and __set__.
Non-data descriptors which only have __get__.
In both cases the returned value goes through __get__ called with the instance as first argument and the class as second argument.
The lookup is even more complicated for class attribute lookup (see for example Class attribute lookup (in the above mentioned blog)).
Let's move to your specific questions:
Why do I need the descriptor class?
In most cases you don't need to write descriptor classes! However you're probably a very regular end user. For example functions. Functions are descriptors, that's how functions can be used as methods with self implicitly passed as first argument.
def test_function(self):
return self
class TestClass(object):
def test_method(self):
...
If you look up test_method on an instance you'll get back a "bound method":
>>> instance = TestClass()
>>> instance.test_method
<bound method TestClass.test_method of <__main__.TestClass object at ...>>
Similarly you could also bind a function by invoking its __get__ method manually (not really recommended, just for illustrative purposes):
>>> test_function.__get__(instance, TestClass)
<bound method test_function of <__main__.TestClass object at ...>>
You can even call this "self-bound method":
>>> test_function.__get__(instance, TestClass)()
<__main__.TestClass at ...>
Note that I did not provide any arguments and the function did return the instance I had bound!
Functions are Non-data descriptors!
Some built-in examples of a data-descriptor would be property. Neglecting getter, setter, and deleter the property descriptor is (from Descriptor HowTo Guide "Properties"):
class Property(object):
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
Since it's a data descriptor it's invoked whenever you look up the "name" of the property and it simply delegates to the functions decorated with #property, #name.setter, and #name.deleter (if present).
There are several other descriptors in the standard library, for example staticmethod, classmethod.
The point of descriptors is easy (although you rarely need them): Abstract common code for attribute access. property is an abstraction for instance variable access, function provides an abstraction for methods, staticmethod provides an abstraction for methods that don't need instance access and classmethod provides an abstraction for methods that need class access rather than instance access (this is a bit simplified).
Another example would be a class property.
One fun example (using __set_name__ from Python 3.6) could also be a property that only allows a specific type:
class TypedProperty(object):
__slots__ = ('_name', '_type')
def __init__(self, typ):
self._type = typ
def __get__(self, instance, klass=None):
if instance is None:
return self
return instance.__dict__[self._name]
def __set__(self, instance, value):
if not isinstance(value, self._type):
raise TypeError(f"Expected class {self._type}, got {type(value)}")
instance.__dict__[self._name] = value
def __delete__(self, instance):
del instance.__dict__[self._name]
def __set_name__(self, klass, name):
self._name = name
Then you can use the descriptor in a class:
class Test(object):
int_prop = TypedProperty(int)
And playing a bit with it:
>>> t = Test()
>>> t.int_prop = 10
>>> t.int_prop
10
>>> t.int_prop = 20.0
TypeError: Expected class <class 'int'>, got <class 'float'>
Or a "lazy property":
class LazyProperty(object):
__slots__ = ('_fget', '_name')
def __init__(self, fget):
self._fget = fget
def __get__(self, instance, klass=None):
if instance is None:
return self
try:
return instance.__dict__[self._name]
except KeyError:
value = self._fget(instance)
instance.__dict__[self._name] = value
return value
def __set_name__(self, klass, name):
self._name = name
class Test(object):
#LazyProperty
def lazy(self):
print('calculating')
return 10
>>> t = Test()
>>> t.lazy
calculating
10
>>> t.lazy
10
These are cases where moving the logic into a common descriptor might make sense, however one could also solve them (but maybe with repeating some code) with other means.
What is instance and owner here? (in __get__). What is the purpose of these parameters?
It depends on how you look up the attribute. If you look up the attribute on an instance then:
the second argument is the instance on which you look up the attribute
the third argument is the class of the instance
In case you look up the attribute on the class (assuming the descriptor is defined on the class):
the second argument is None
the third argument is the class where you look up the attribute
So basically the third argument is necessary if you want to customize the behavior when you do class-level look-up (because the instance is None).
How would I call/use this example?
Your example is basically a property that only allows values that can be converted to float and that is shared between all instances of the class (and on the class - although one can only use "read" access on the class otherwise you would replace the descriptor instance):
>>> t1 = Temperature()
>>> t2 = Temperature()
>>> t1.celsius = 20 # setting it on one instance
>>> t2.celsius # looking it up on another instance
20.0
>>> Temperature.celsius # looking it up on the class
20.0
That's why descriptors generally use the second argument (instance) to store the value to avoid sharing it. However in some cases sharing a value between instances might be desired (although I cannot think of a scenario at this moment). However it makes practically no sense for a celsius property on a temperature class... except maybe as purely academic exercise.
Why do I need the descriptor class?
Inspired by Fluent Python by Buciano Ramalho
Imaging you have a class like this
class LineItem:
price = 10.9
weight = 2.1
def __init__(self, name, price, weight):
self.name = name
self.price = price
self.weight = weight
item = LineItem("apple", 2.9, 2.1)
item.price = -0.9 # it's price is negative, you need to refund to your customer even you delivered the apple :(
item.weight = -0.8 # negative weight, it doesn't make sense
We should validate the weight and price in avoid to assign them a negative number, we can write less code if we use descriptor as a proxy as this
class Quantity(object):
__index = 0
def __init__(self):
self.__index = self.__class__.__index
self._storage_name = "quantity#{}".format(self.__index)
self.__class__.__index += 1
def __set__(self, instance, value):
if value > 0:
setattr(instance, self._storage_name, value)
else:
raise ValueError('value should >0')
def __get__(self, instance, owner):
return getattr(instance, self._storage_name)
then define class LineItem like this:
class LineItem(object):
weight = Quantity()
price = Quantity()
def __init__(self, name, weight, price):
self.name = name
self.weight = weight
self.price = price
and we can extend the Quantity class to do more common validating
You'd see https://docs.python.org/3/howto/descriptor.html#properties
class Property(object):
"Emulate PyProperty_Type() in Objects/descrobject.c"
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError("can't delete attribute")
self.fdel(obj)
def getter(self, fget):
return type(self)(fget, self.fset, self.fdel, self.__doc__)
def setter(self, fset):
return type(self)(self.fget, fset, self.fdel, self.__doc__)
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)
Easy to digest (with example) Explanation for __get__ & __set__ & __call__ in classes, what is Owner, Instance?
Some points to mug up before diving in:
__get__ __set__ are called descriptors of the class to work/save their internal attributes namely: __name__ (name of class/owner class), variables - __dict__ etc. I will explain what is an owner later
Descriptors are used in design patterers more commonly, for example, with decorators (to abstract things out). You can consider it's more often used in software architecture design to make things less redundant and more readable (seems ironical). Thus abiding SOLID and DRY principles.
If you are not designing software that should abide by SOLID and DRY principles, you probably don't need them, but it's always wise to understand them.
1. Conside this code:
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints:TypeError: __call__() missing 1 required positional argument: 'arg2'
So, when instance.method("first", "second") is called, __call__ method is called from the Method class (call method makes a class object just callable like a function - whenever a class instance is called __call__ gets instiantiated), and following arguments are assigned: instance: "first", arg1: "second", and the last arg2 is left out, this prints out the error: TypeError: __call__() missing 1 required positional argument: 'arg2'
2. how to solve it?
Since __call__ takes instance as first argument (instance, arg1, arg2), but instance of what?
Instance is the instance of main class (MyClass) which is calling the descriptor class (Method). So, instance = MyClass() is the instance and so who is the owner? the class holding the discriptor class - MyClass, However, there is no method in our descriptor class (Method) to recognise it as an instance. So that is where we need __get__ method. Again consider the code below:
from types import MethodType
class Method:
def __init__(self, name):
self.name = name
def __call__(self, instance, arg1, arg2):
print(f"{self.name}: {instance} called with {arg1} and {arg2}")
def __set__(self, instance, value):
self.value = value
instance.__dict__["method"] = value
def __get__(self, instance, owner):
if instance is None:
return self
print (instance, owner)
return MethodType(self, instance)
class MyClass:
method = Method("Internal call")
instance = MyClass()
instance.method("first", "second")
# Prints: Internal call: <__main__.MyClass object at 0x7fb7dd989690> called with first and second
forget about set for now according to docs:
__get__ "Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access)."
if you do: instance.method.__get__(instance)
Prints:<__main__.MyClass object at 0x7fb7dd9eab90> <class '__main__.MyClass'>
this means instance: object of MyClass which is instance
and Owner is MyClass itself
3. __set__ Explaination:
__set__ is used to set some value in the class __dict__ object (let's say using a command line). command for setting the internal value for set is: instance.descriptor = 'value' # where descriptor is method in this case
(instance.__dict__["method"] = value in the code just update the __dict__ object of the descriptor)
So do: instance.method = 'value' now to check if the value = 'value' is set in the __set__ method we can access __dict__ object of the descriptor method.
Do:
instance.method.__dict__ prints: {'_name': 'Internal call', 'value': 'value'}
Or you can check the __dict__ value using vars(instance.method)
prints: {'name': 'Internal call', 'value': 'value'}
I hope things are clear now:)
I tried (with minor changes as suggested) the code from Andrew Cooke's answer. (I am running python 2.7).
The code:
#!/usr/bin/env python
class Celsius:
def __get__(self, instance, owner): return 9 * (instance.fahrenheit + 32) / 5.0
def __set__(self, instance, value): instance.fahrenheit = 32 + 5 * value / 9.0
class Temperature:
def __init__(self, initial_f): self.fahrenheit = initial_f
celsius = Celsius()
if __name__ == "__main__":
t = Temperature(212)
print(t.celsius)
t.celsius = 0
print(t.fahrenheit)
The result:
C:\Users\gkuhn\Desktop>python test2.py
<__main__.Celsius instance at 0x02E95A80>
212
With Python prior to 3, make sure you subclass from object which will make the descriptor work correctly as the get magic does not work for old style classes.
Suppose we have the following class hierarchy:
class ClassA:
#property
def foo(self): return "hello"
class ClassB(ClassA):
#property
def bar(self): return "world"
If I explore __dict__ on ClassB like so, I only see the bar attribute:
for name,_ in ClassB.__dict__.items():
if name.startswith("__"):
continue
print(name)
Output is bar
I can roll my own means to get attributes on not only the specified type but its ancestors. However, my question is whether there's already a way in python for me to do this without re-inventing a wheel.
def return_attributes_including_inherited(type):
results = []
return_attributes_including_inherited_helper(type,results)
return results
def return_attributes_including_inherited_helper(type,attributes):
for name,attribute_as_object in type.__dict__.items():
if name.startswith("__"):
continue
attributes.append(name)
for base_type in type.__bases__:
return_attributes_including_inherited_helper(base_type,attributes)
Running my code as follows...
for attribute_name in return_attributes_including_inherited(ClassB):
print(attribute_name)
... gives back both bar and foo.
Note that I'm simplifying some things: name collisions, using items() when for this example I could use dict, skipping over anything that starts with __, ignoring the possibility that two ancestors themselves have a common ancestor, etc.
EDIT1 - I tried to keep the example simple. But I really want both the attribute name and the attribute reference for each class and ancestor class. One of the answers below has me on a better track, I'll post some better code when I get it to work.
EDIT2 - This does what I want and is very succinct. It's based on Eli's answer below.
def get_attributes(type):
attributes = set(type.__dict__.items())
for type in type.__mro__:
attributes.update(type.__dict__.items())
return attributes
It gives back both the attribute names and their references.
EDIT3 - One of the answers below suggested using inspect.getmembers. This appears very useful because it's like dict only it operates on ancestor classes as well.
Since a large part of what I was trying to do was find attributes marked with a particular descriptor, and include ancestors classes, here is some code that would help do that in case it helps anyone:
class MyCustomDescriptor:
# This is greatly oversimplified
def __init__(self,foo,bar):
self._foo = foo
self._bar = bar
pass
def __call__(self,decorated_function):
return self
def __get__(self,instance,type):
if not instance:
return self
return 10
class ClassA:
#property
def foo(self): return "hello"
#MyCustomDescriptor(foo="a",bar="b")
def bar(self): pass
#MyCustomDescriptor(foo="c",bar="d")
def baz(self): pass
class ClassB(ClassA):
#property
def something_we_dont_care_about(self): return "world"
#MyCustomDescriptor(foo="e",bar="f")
def blah(self): pass
# This will get attributes on the specified type (class) that are of matching_attribute_type. It just returns the attributes themselves, not their names.
def get_attributes_of_matching_type(type,matching_attribute_type):
return_value = []
for member in inspect.getmembers(type):
member_name = member[0]
member_instance = member[1]
if isinstance(member_instance,matching_attribute_type):
return_value.append(member_instance)
return return_value
# This will return a dictionary of name & instance of attributes on type that are of matching_attribute_type (useful when you're looking for attributes marked with a particular descriptor)
def get_attribute_name_and_instance_of_matching_type(type,matching_attribute_type):
return_value = {}
for member in inspect.getmembers(ClassB):
member_name = member[0]
member_instance = member[1]
if isinstance(member_instance,matching_attribute_type):
return_value[member_name] = member_instance
return return_value
You should use python's inspect module for any such introspective capabilities.
.
.
>>> class ClassC(ClassB):
... def baz(self):
... return "hiya"
...
>>> import inspect
>>> for attr in inspect.getmembers(ClassC):
... print attr
...
('__doc__', None)
('__module__', '__main__')
('bar', <property object at 0x10046bf70>)
('baz', <unbound method ClassC.baz>)
('foo', <property object at 0x10046bf18>)
Read more about the inspect module here.
You want to use dir:
for attr in dir(ClassB):
print attr
Sadly there isn't a single composite object. Every attribute access for a (normal) python object first checks obj.__dict__, then the attributes of all it's base classes; while there are some internal caches and optimizations, there isn't a single object you can access.
That said, one thing that could improve your code is to use cls.__mro__ instead of cls.__bases__... instead of the class's immediate parents, cls.__mro__ contains ALL the ancestors of the class, in the exact order Python would search, with all common ancestors occuring only once. That would also allow your type-searching method to be non-recursive. Loosely...
def get_attrs(obj):
attrs = set(obj.__dict__)
for cls in obj.__class__.__mro__:
attrs.update(cls.__dict__)
return sorted(attrs)
... does a fair approximation of the default dir(obj) implementation.
Here is a function I wrote, back in the day. The best answer is using the inspect module, as using __dict__ gives us ALL functions (ours + inherited) and (ALL?) data members AND properties. Where inspect gives us enough information to weed out what we don't want.
def _inspect(a, skipFunctionsAlways=True, skipMagic = True):
"""inspects object attributes, removing all the standard methods from 'object',
and (optionally) __magic__ cruft.
By default this routine skips __magic__ functions, but if you want these on
pass False in as the skipMagic parameter.
By default this routine skips functions, but if you want to see all the functions,
pass in False to the skipFunctionsAlways function. This works together with the
skipMagic parameter: if the latter is True, you won't see __magic__ methods.
If skipFunctionsAlways = False and skipMagic = False, you'll see all the __magic__
methods declared for the object - including __magic__ functions declared by Object
NOT meant to be a comprehensive list of every object attribute - instead, a
list of every object attribute WE (not Python) defined. For a complete list
of everything call inspect.getmembers directly"""
objType = type(object)
def weWantIt(obj):
#return type(a) != objType
output= True
if (skipFunctionsAlways):
output = not ( inspect.isbuiltin(obj) ) #not a built in
asStr = ""
if isinstance(obj, types.MethodType):
if skipFunctionsAlways: #never mind, we don't want it, get out.
return False
else:
asStr = obj.__name__
#get just the name of the function, we don't want the whole name, because we don't want to take something like:
#bound method LotsOfThings.bob of <__main__.LotsOfThings object at 0x103dc70>
#to be a special method because it's module name is special
#WD-rpw 02-23-2008
#TODO: it would be great to be able to separate out superclass methods
#maybe by getting the class out of the method then seeing if that attribute is in that class?
else:
asStr = str(obj)
if (skipMagic):
output = (asStr.find("__") == -1 ) #not a __something__
return (output)
for value in inspect.getmembers( a, weWantIt ):
yield value
{k: getattr(ClassB, k) for k in dir(ClassB)}
Proper values (instead of <property object...>) will be presented when using ClassB instance.
And of course You can filter this by adding things like if not k.startswith('__') in the end.
For example I have a base class as follows:
class BaseClass(object):
def __init__(self, classtype):
self._type = classtype
From this class I derive several other classes, e.g.
class TestClass(BaseClass):
def __init__(self):
super(TestClass, self).__init__('Test')
class SpecialClass(BaseClass):
def __init__(self):
super(TestClass, self).__init__('Special')
Is there a nice, pythonic way to create those classes dynamically by a function call that puts the new class into my current scope, like:
foo(BaseClass, "My")
a = MyClass()
...
As there will be comments and questions why I need this: The derived classes all have the exact same internal structure with the difference, that the constructor takes a number of previously undefined arguments. So, for example, MyClass takes the keywords a while the constructor of class TestClass takes b and c.
inst1 = MyClass(a=4)
inst2 = MyClass(a=5)
inst3 = TestClass(b=False, c = "test")
But they should NEVER use the type of the class as input argument like
inst1 = BaseClass(classtype = "My", a=4)
I got this to work but would prefer the other way, i.e. dynamically created class objects.
This bit of code allows you to create new classes with dynamic
names and parameter names.
The parameter verification in __init__ just does not allow
unknown parameters, if you need other verifications, like
type, or that they are mandatory, just add the logic
there:
class BaseClass(object):
def __init__(self, classtype):
self._type = classtype
def ClassFactory(name, argnames, BaseClass=BaseClass):
def __init__(self, **kwargs):
for key, value in kwargs.items():
# here, the argnames variable is the one passed to the
# ClassFactory call
if key not in argnames:
raise TypeError("Argument %s not valid for %s"
% (key, self.__class__.__name__))
setattr(self, key, value)
BaseClass.__init__(self, name[:-len("Class")])
newclass = type(name, (BaseClass,),{"__init__": __init__})
return newclass
And this works like this, for example:
>>> SpecialClass = ClassFactory("SpecialClass", "a b c".split())
>>> s = SpecialClass(a=2)
>>> s.a
2
>>> s2 = SpecialClass(d=3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in __init__
TypeError: Argument d not valid for SpecialClass
I see you are asking for inserting the dynamic names in the naming scope --
now, that is not considered a good practice in Python - you either have
variable names, known at coding time, or data - and names learned in runtime
are more "data" than "variables" -
So, you could just add your classes to a dictionary and use them from there:
name = "SpecialClass"
classes = {}
classes[name] = ClassFactory(name, params)
instance = classes[name](...)
And if your design absolutely needs the names to come in scope,
just do the same, but use the dictionary returned by the globals()
call instead of an arbitrary dictionary:
name = "SpecialClass"
globals()[name] = ClassFactory(name, params)
instance = SpecialClass(...)
(It indeed would be possible for the class factory function to insert the name dynamically on the global scope of the caller - but that is even worse practice, and is not compatible across Python implementations. The way to do that would be to get the caller's execution frame, through sys._getframe(1) and setting the class name in the frame's global dictionary in its f_globals attribute).
update, tl;dr: This answer had become popular, still its very specific to the question body. The general answer on how to
"dynamically create derived classes from a base class"
in Python is a simple call to type passing the new class name, a tuple with the baseclass(es) and the __dict__ body for the new class -like this:
>>> new_class = type("NewClassName", (BaseClass,), {"new_method": lambda self: ...})
update
Anyone needing this should also check the dill project - it claims to be able to pickle and unpickle classes just like pickle does to ordinary objects, and had lived to it in some of my tests.
type() is the function that creates classes and in particular sub-classes, like in the question:
def set_x(self, value):
self.x = value
# type() takes as argument the new class name, its base
# classes, and its attributes:
SubClass = type('SubClass', (BaseClass,), {'set_x': set_x})
# (More methods can be put in SubClass, including __init__().)
obj = SubClass()
obj.set_x(42)
print obj.x # Prints 42
print isinstance(obj, BaseClass) # True
In my case :
inst3 = globals()["SpecialClass"](b=False, c = "test")
To create a class with a dynamic attribute value, checkout the code below.
NB. This are code snippets in python programming language
def create_class(attribute_data, **more_data): # define a function with required attributes
class ClassCreated(optional extensions): # define class with optional inheritance
attribute1 = adattribute_data # set class attributes with function parameter
attribute2 = more_data.get("attribute2")
return ClassCreated # return the created class
# use class
myclass1 = create_class("hello") # *generates a class*
I would like to replace an object instance by another instance inside a method like this:
class A:
def method1(self):
self = func(self)
The object is retrieved from a database.
It is unlikely that replacing the 'self' variable will accomplish whatever you're trying to do, that couldn't just be accomplished by storing the result of func(self) in a different variable. 'self' is effectively a local variable only defined for the duration of the method call, used to pass in the instance of the class which is being operated upon. Replacing self will not actually replace references to the original instance of the class held by other objects, nor will it create a lasting reference to the new instance which was assigned to it.
As far as I understand, If you are trying to replace the current object with another object of same type (assuming func won't change the object type) from an member function. I think this will achieve that:
class A:
def method1(self):
newObj = func(self)
self.__dict__.update(newObj.__dict__)
It is not a direct answer to the question, but in the posts below there's a solution for what amirouche tried to do:
Python object conversion
Can I dynamically convert an instance of one class to another?
And here's working code sample (Python 3.2.5).
class Men:
def __init__(self, name):
self.name = name
def who_are_you(self):
print("I'm a men! My name is " + self.name)
def cast_to(self, sex, name):
self.__class__ = sex
self.name = name
def method_unique_to_men(self):
print('I made The Matrix')
class Women:
def __init__(self, name):
self.name = name
def who_are_you(self):
print("I'm a women! My name is " + self.name)
def cast_to(self, sex, name):
self.__class__ = sex
self.name = name
def method_unique_to_women(self):
print('I made Cloud Atlas')
men = Men('Larry')
men.who_are_you()
#>>> I'm a men! My name is Larry
men.method_unique_to_men()
#>>> I made The Matrix
men.cast_to(Women, 'Lana')
men.who_are_you()
#>>> I'm a women! My name is Lana
men.method_unique_to_women()
#>>> I made Cloud Atlas
Note the self.__class__ and not self.__class__.__name__. I.e. this technique not only replaces class name, but actually converts an instance of a class (at least both of them have same id()). Also, 1) I don't know whether it is "safe to replace a self object by another object of the same type in [an object own] method"; 2) it works with different types of objects, not only with ones that are of the same type; 3) it works not exactly like amirouche wanted: you can't init class like Class(args), only Class() (I'm not a pro and can't answer why it's like this).
Yes, all that will happen is that you won't be able to reference the current instance of your class A (unless you set another variable to self before you change it.) I wouldn't recommend it though, it makes for less readable code.
Note that you're only changing a variable, just like any other. Doing self = 123 is the same as doing abc = 123. self is only a reference to the current instance within the method. You can't change your instance by setting self.
What func(self) should do is to change the variables of your instance:
def func(obj):
obj.var_a = 123
obj.var_b = 'abc'
Then do this:
class A:
def method1(self):
func(self) # No need to assign self here
In many cases, a good way to achieve what you want is to call __init__ again. For example:
class MyList(list):
def trim(self,n):
self.__init__(self[:-n])
x = MyList([1,2,3,4])
x.trim(2)
assert type(x) == MyList
assert x == [1,2]
Note that this comes with a few assumptions such as the all that you want to change about the object being set in __init__. Also beware that this could cause problems with inheriting classes that redefine __init__ in an incompatible manner.
Yes, there is nothing wrong with this. Haters gonna hate. (Looking at you Pycharm with your in most cases imaginable, there's no point in such reassignment and it indicates an error).
A situation where you could do this is:
some_method(self, ...):
...
if(some_condition):
self = self.some_other_method()
...
return ...
Sure, you could start the method body by reassigning self to some other variable, but if you wouldn't normally do that with other parametres, why do it with self?
One can use the self assignment in a method, to change the class of instance to a derived class.
Of course one could assign it to a new object, but then the use of the new object ripples through the rest of code in the method. Reassiging it to self, leaves the rest of the method untouched.
class aclass:
def methodA(self):
...
if condition:
self = replace_by_derived(self)
# self is now referencing to an instance of a derived class
# with probably the same values for its data attributes
# all code here remains untouched
...
self.methodB() # calls the methodB of derivedclass is condition is True
...
def methodB(self):
# methodB of class aclass
...
class derivedclass(aclass):
def methodB(self):
#methodB of class derivedclass
...
But apart from such a special use case, I don't see any advantages to replace self.
You can make the instance a singleton element of the class
and mark the methods with #classmethod.
from enum import IntEnum
from collections import namedtuple
class kind(IntEnum):
circle = 1
square = 2
def attr(y): return [getattr(y, x) for x in 'k l b u r'.split()]
class Shape(namedtuple('Shape', 'k,l,b,u,r')):
self = None
#classmethod
def __repr__(cls):
return "<Shape({},{},{},{},{}) object at {}>".format(
*(attr(cls.self)+[id(cls.self)]))
#classmethod
def transform(cls, func):
cls.self = cls.self._replace(**func(cls.self))
Shape.self = Shape(k=1, l=2, b=3, u=4, r=5)
s = Shape.self
def nextkind(self):
return {'k': self.k+1}
print(repr(s)) # <Shape(1,2,3,4,5) object at 139766656561792>
s.transform(nextkind)
print(repr(s)) # <Shape(2,2,3,4,5) object at 139766656561888>