Trying to build something and found this odd behavior. I'm trying to build a class that I can call dict on
class Test1:
def __iter__(self):
return iter([(a, a) for a in range(10)])
obj = Test1()
dict(obj) # returns {0: 0, 1: 1, 2: 2 ...}
now in my use case the object has a __getattr__ overload, which is where the problem comes in so
class Test2:
def __iter__(self):
return iter([(a, a) for a in range(10)])
def __getattr__(self, attr):
raise Exception(f"why are you calling me {attr}")
obj = Test2()
dict(obj) # Exception: why are you calling me keys
The dict function is calling somewhere self.keys but obviously Test1().keys throws an AttributeError so it's being handed there somehow. How do I get __iter__ and __getattr__ to play nicely together. Or is there a better way of doing this?
Edit:
I guess raising an AttributeError works
class Test2:
def __iter__(self):
return iter([(a, a) for a in range(10)])
def __getattr__(self, attr):
raise AttributeError(f"why are you calling me {attr}")
obj = Test2()
dict(obj). # No Exception
You don't actually have to PROVIDE .keys (clearly, or your first example would have failed). You just need to provide the right exception. This works:
class Test2:
def __iter__(self):
return iter([(a, a) for a in range(10)])
def __getattr__(self, attr):
if attr == 'keys':
raise AttributeError(f"{type(self).__name__!r} object has no attribute {attr!r}")
raise Exception(f"why are you calling me {attr}")
obj = Test2()
dict(obj) # Exception: why are you calling me keys
Related
I have a class, where __getattr__ is implemented simply like:
def __getattr__(self, item): return self.__dict__[item]
The problem I'm seeing is that many Python libraries (e.g. numpy and pandas are trying to sniff whether my object has something called __array__ using this statement
hasattr(obj, '__array__`)
But my object is throwing an error at them saying there is no such attribute.
My dilemma: How can I make my class behave nicely with hasattr (by returning False) instead of throwing an error, WHILE at the same time, throw an error if any one wanted an attribute that doesn't exist (i.e. I still want that error to be thrown in any other case).
EDIT: reproducible code as requested:
class A:
def __getattr__(self, item): return self.__dict__[item]
a = A()
hasattr(a, "lol")
traceback:
File "<ipython-input-31-b7d3ffac514f>", line 4, in <module>
hasattr(a, "lol")
File "<ipython-input-31-b7d3ffac514f>", line 2, in __getattr__
def __getattr__(self, item): return self.__dict__[item]
KeyError: 'lol'
From the docs:
__getattr__ ... should either return the attribute value or raise an AttributeError exception.
hasattr(object, name) ... is implemented by calling getattr(object, name) and seeing whether it raises an AttributeError or not.
So you just need to raise an AttributeError:
class A:
def __getattr__(self, item):
try:
return self.__dict__[item]
except KeyError:
classname = type(self).__name__
msg = f'{classname!r} object has no attribute {item!r}'
raise AttributeError(msg)
a = A()
print(hasattr(a, "lol")) # -> False
print(a.lol) # -> AttributeError: 'A' object has no attribute 'lol'
(This error message is based on the one from object().lol.)
I would like to add some small functionality to a method of another class' instance.
What I tried is listed below. With the same approach it's possible to completely change the method. But I would like to keep a slightly altered version of the original method. Is that possible?
BTW: The classes can not inherit from one another, because I don't necessarily know what class B is.
class A:
def alter_fn(self, obj):
def wrapper():
func = getattr(obj, obj.fn_name)
out = func()
print('out = ', out)
return out
setattr(obj, obj.fn_name, wrapper)
return
class B:
def __init__(self) -> None:
self.fn_name = 'fn'
def fn(self):
return 123
a=A()
b=B()
a.alter_fn(b)
b.fn()
For obvious reasons this raises a recursion depth error but I do not know how to avoid that. I also tried func = copy(getattr(obj, obj.fn_name))
After altering the method b.fn I would like it to return the value, the unaltered function would return (here 123) but I would also like it to do something else with this output, in this example print it.
Move the assignment of func above the definition of wrapper.
...
def alter_fn(self, obj):
func = getattr(obj, obj.fn_name)
def wrapper():
...
This way getattr is called only once and wrapper can access func as a closure variable.
You just need to take the func = getattr(obj, obj.fn_name) out of the wrapperfunction.
class A:
def alter_fn(self, obj):
func = getattr(obj, obj.fn_name)
def wrapper():
out = func()
print('out = ', out)
return out
setattr(obj, obj.fn_name, wrapper)
return
You need to get the wrapped function before re-assigning it:
def alter_fn(self, obj):
func = getattr(obj, obj.fn_name)
def wrapper():
out = func()
print('out = ', out)
return out
setattr(obj, obj.fn_name, wrapper)
return
This way func is assigned only once, when the original function is called. Thus, you can safely call it without calling wrapper() again by accident.
Also consider using functools.wraps() to keep the docstring and name information of the string the same.
I am trying to write a function that gets the class when we pass a given method as argument.
For example, if we have
class Hello:
NAME = "HELLO TOTO"
def method(self) -> int:
return 5
#classmethod
def cls_method(cls) -> str:
return "Hi"
class Bonjour(Hello):
NOM = "BONJOUR TOTO"
def new_method(self) -> int:
return 0
I would get:
Hello from the methods Hello().method or Hello().cls_method
Bonjour from the methods Bonjour().new_method or Bonjour().cls_method
I searched on SO but could not find any direct answer to my question.
How could I implement such function (in Python 3.6+ if that matters)?
Thanks
I believe there's no fool-proof way, but this would work for most cases:
def get_class_of_bound_self(f):
assert hasattr(f, '__self__')
return f.__self__ if isinstance(f.__self__, type) else type(f.__self__)
Note that this would break down if f is a method of a metaclass M; it would return M instead of type.
I came with the following solution:
import inspect
def get_class(func: Callable[..., Any]) -> Any:
"""Return class of a method.
Args:
func: callable
Returns:
Class of the method, if the argument is a method
Raises:
AttributeError: if the argument is not callable or not a method
"""
if not callable(func):
raise AttributeError(f"{func} shall be callable")
if not inspect.ismethod(func):
raise AttributeError(f"Callable {func} shall be a method")
first_arg = func.__self__ # type: ignore # method have "self" attribute
return first_arg if inspect.isclass(first_arg) else first_arg.__class__
The last line return first_arg if inspect.isclass(first_arg) else first_arg.__class__ is to handle the cases of class methods (in which case func.__self__ corresponds to cls and is the class itself).
Another alternative without inspect module is with catching exceptions (a big thanks to #Elazar for the idea of using isistance(..., type)):
def get_class(func: Callable[..., Any]) -> Any:
"""Return class of a method.
Args:
func: callable
Returns:
Class of the method, if the argument is a method
Raises:
AttributeError: if the argument is not callable or not a method
"""
if not callable(func):
raise AttributeError(f"{func} shall be callable")
try:
first_arg = func.__self__ # type: ignore # method have "self" attribute
except AttributeError:
raise AttributeError(f"Callable {func} shall be a method")
cls_or_type = first_arg.__class__
return first_arg if isinstance(cls_or_type, type) else cls_or_type
And this is the code I have used to check if you might be interested:
def my_func() -> int:
"""It feels like a zero"""
return 0
for method in [
Hello().method,
Bonjour().method,
Hello().cls_method,
Bonjour().cls_method,
Bonjour().new_method,
]:
# MyClass = get_class(func)
MyClass = get_class_2(method)
for attr in ["NAME", "NOM"]:
print(f"... {method} - {attr} ...")
try:
print(getattr(MyClass, attr))
except AttributeError as exp:
print(f"Error when getting attribute: {exp}")
# class_ = get_class(my_func)
for not_method in [my_func, int, Hello]:
try:
MyClass = get_class(not_method)
print(f"{not_method} => NOK (no exception raised)")
except AttributeError:
print(f"{not_method} => OK")
I have a situation, where getattribute fallbacks to getattr and then again getattribute gets called.
How current getattribute gets called again? I am confused.
class Count(object):
def __init__(self,mymin,mymax):
self.mymin=mymin
self.mymax=mymax
self.current=None
def __getattr__(self, item):
print("akhjhd")
self.__dict__[item]=0
return 0
def __getattribute__(self, item):
print("this is called first")
if item.startswith('cur'):
print("this raised an error")
raise AttributeError
print("This will execute as well")
return object.__getattribute__(self,item)
obj1 = Count(1,10)
print(obj1.mymin)
print(obj1.mymax)
print(obj1.current)
Console Output:
this is called first
This will execute as well
1
this is called first
This will execute as well
10
this is called first
this raised an error
akhjhd
this is called first
This will execute as well
0
getattr is called because getattribute raises AttributeError
self.__dict__ invokes the "second" call to getattribute
Clean the code and add print(item) to make this clearer:
class Count(object):
def __init__(self):
self.current = None
def __getattr__(self, item):
print("in getattr")
self.__dict__[item] = 0
return 0
def __getattribute__(self, item):
print(item)
print("in __getattribute__ 1")
if item.startswith('cur'):
print("starts with 'cur'")
raise AttributeError
print("in __getattribute__ 2")
return object.__getattribute__(self, item)
obj1 = Count()
print(obj1.current)
Outputs
current
in __getattribute__ 1
starts with 'cur'
in getattr
__dict__
in __getattribute__ 1
in __getattribute__ 2
0
You need to consult with python Data model
Excerpts for __getattribute__:
Called unconditionally to implement attribute accesses for instances of the class. If the class also defines __getattr__(), the latter will not be called unless __getattribute__() either calls it explicitly or raises an AttributeError.
I see in your code:
if item.startswith('cur'):
print("this raised an error")
raise AttributeError
So I think you did it intentionally
I extended dict in a simple way to directly access it's values with the d.key notation instead of d['key']:
class ddict(dict):
def __getattr__(self, item):
return self[item]
def __setattr__(self, key, value):
self[key] = value
Now when I try to pickle it, it will call __getattr__ to find __getstate__, which is neither present nor necessary. The same will happen upon unpickling with __setstate__:
>>> import pickle
>>> class ddict(dict):
... def __getattr__(self, item):
... return self[item]
... def __setattr__(self, key, value):
... self[key] = value
...
>>> pickle.dumps(ddict())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getattr__
KeyError: '__getstate__'
How do I have to modify the class ddict in order to be properly pickable?
The problem is not pickle but that your __getattr__ method breaks the expected contract by raising KeyError exceptions. You need to fix your __getattr__ method to raise AttributeError exceptions instead:
def __getattr__(self, item):
try:
return self[item]
except KeyError:
raise AttributeError(item)
Now pickle is given the expected signal for a missing __getstate__ customisation hook.
From the object.__getattr__ documentation:
This method should return the (computed) attribute value or raise an AttributeError exception.
(bold emphasis mine).
If you insist on keeping the KeyError, then at the very least you need to skip names that start and end with double underscores and raise an AttributeError just for those:
def __getattr__(self, item):
if isinstance(item, str) and item[:2] == item[-2:] == '__':
# skip non-existing dunder method lookups
raise AttributeError(item)
return self[item]
Note that you probably want to give your ddict() subclass an empty __slots__ tuple; you don't need the extra __dict__ attribute mapping on your instances, since you are diverting attributes to key-value pairs instead. That saves you a nice chunk of memory per instance.
Demo:
>>> import pickle
>>> class ddict(dict):
... __slots__ = ()
... def __getattr__(self, item):
... try:
... return self[item]
... except KeyError:
... raise AttributeError(item)
... def __setattr__(self, key, value):
... self[key] = value
...
>>> pickle.dumps(ddict())
b'\x80\x03c__main__\nddict\nq\x00)\x81q\x01.'
>>> type(pickle.loads(pickle.dumps(ddict())))
<class '__main__.ddict'>
>>> d = ddict()
>>> d.foo = 'bar'
>>> d.foo
'bar'
>>> pickle.loads(pickle.dumps(d))
{'foo': 'bar'}
That pickle tests for the __getstate__ method on the instance rather than on the class as is the norm for special methods, is a discussion for another day.
First of all, I think you may need to distinguish between instance attribute and class attribute.
In Python official document Chapter 11.1.4 about pickling, it says:
instances of such classes whose dict or the result of calling getstate() is picklable (see section The pickle protocol for details).
Therefore, the error message you're getting is when you try to pickle an instance of the class, but not the class itself - in fact, your class definition will just pickle fine.
Now for pickling an object of your class, the problem is that you need to call the parent class's serialization implementation first to properly set things up. The correct code is:
In [1]: import pickle
In [2]: class ddict(dict):
...:
...: def __getattr__(self, item):
...: super.__getattr__(self, item)
...: return self[item]
...:
...: def __setattr__(self, key, value):
...: super.__setattr__(self, key, value)
...: self[key] = value
...:
In [3]: d = ddict()
In [4]: d.name = "Sam"
In [5]: d
Out[5]: {'name': 'Sam'}
In [6]: pickle.dumps(d)
Out[6]: b'\x80\x03c__main__\nddict\nq\x00)\x81q\x01X\x04\x00\x00\x00nameq\x02X\x03\x00\x00\x00Samq\x03s}q\x04h\x02h\x03sb.'