I'm trying to write a class. Objects of that class can take a label/value pair and store it in such a way that label can be accessed as an attribute, returning value: obj.label -> value
The main goal here is to get autocompletion in jupyter notebooks, so obj.<tab> should produce the list of labels as autocompletion suggestion. The below class accomplishes this:
class Autocompleter:
def __init__(self, ):
self._funcs = {}
def add(self, label):
self._funcs[label] = 1.
def __dir__(self):
return list(self._funcs.keys())
def __getattr__(self, name):
if name in dir(self):
return self._funcs[name]
The problem: When accessing an invalid attribute, the __getattr__ simply returns None. I'd rather have it throw an exception. I can think of 2 ways to achieve this, but unfortunately both break the autocompletion:
Changing __getattr__ to:
def __getattr__(self, name):
return self._funcs[name]
or
def __getattr__(self, name):
if name in dir(self):
return self._funcs[name]
else:
raise Exception('invalid name')
produces the desired exception, but breaks the autocompletion:
a = Autocompleter()
a.add('foo')
Now a.<tab> does not suggest foo for autocompletion, it simply does nothing. As far as I can tell, jedi is used by default for autompletion in jupyterlab.
Question: Is there a way to get both the exception on invalid names and the autocomplete feature working?
I figured it out myself. The correct way to handle an invalid attribute name is to raise an AttributeError. This can then be understood by jedi.
def __getattr__(self, name):
if name in dir(self):
return self._funcs[name]
raise AttributeError(name)
Note that the __dir__ method is still needed.
Related
I am working with htcondor python bindings (https://htcondor.readthedocs.io/en/latest/apis/python-bindings/index.html)
You don't need to know htcondor, but for reference I am working with the
htcondor.JobEvent class to get the data that I want.
From the description of that object it follows that it behaves like a dictionary,
but it has no __dict__ property.
Basically you can't tell what this object is, because it's translated from C++ into python, hence I want to wrap it with all it's functionalities to add more functionalities.
The way I am solving this atm is:
class HTCJobEventWrapper:
"""
Wrapper for HTCondor JobEvent.
Extracts event number and time_stamp of an event.
The wrapped event can be printed to the terminal for dev purpose.
:param job_event: HTCJobEvent
"""
def __init__(self, job_event: HTCJobEvent):
self.wrapped_class = job_event
self.event_number = job_event.get('EventTypeNumber')
self.time_stamp = date_time.strptime(
job_event.get('EventTime'),
STRP_FORMAT
)
def __getattr__(self, attr):
return getattr(self.wrapped_class, attr)
def get(self, *args, **kwargs):
"""Wraps wrapped_class get function."""
return self.wrapped_class.get(*args, **kwargs)
def items(self):
"""Wraps wrapped_class items method."""
return self.wrapped_class.items()
def keys(self):
"""Wraps wrapped_class keys method."""
return self.wrapped_class.keys()
def values(self):
"""Wraps wrapped_class values method."""
return self.wrapped_class.values()
def to_dict(self):
"""Turns wrapped_class items into a dictionary."""
return dict(self.items())
def __repr__(self):
return json.dumps(
self.to_dict(),
indent=2
)
With this it's possible to get any attribute and to use the methods described in the documentation.
However as you can see HTCJobEventWrapper is not of type htcondor.JobEvent
and is not inheriting from it.
If you try to instatiate a htcondor.JobEvent class it results in the following error: RuntimeError: This class cannot be instantiated from Python.
What I want:
I would like it to be a child class which copies a given htcondor.JobEvent object completely
and adds the functionalities I want and returns a HTCJobEventWrapper object
This kind of relates to this question: completely wrap an object in python
Is there a pythonic way to dynamically call every attribute, function or method on self.wrapped_class ? Just like with getattr ?
But in this case I've tried getattr but it works only for attributes.
I am observing following behavior since python passes object by reference?
class Person(object):
pass
person = Person()
person.name = 'UI'
def test(person):
person.name = 'Test'
test(person)
print(person.name)
>>> Test
I found copy.deepcopy() to deepcopy object to prevent modifying the passed object. Are there any other recommendations ?
import copy
class Person(object):
pass
person = Person()
person.name = 'UI'
def test(person):
person_copy = copy.deepcopy(person)
person_copy.name = 'Test'
test(person)
print(person.name)
>>> UI
I am observing following behavior since python passes object by reference?
Not really. it's a subtle question. you can look at python - How do I pass a variable by reference? - Stack Overflow
Personally, I don't fully agree with the accepted answer and recommend you google call by sharing. Then, you can make your own decision on this subtle question.
I found copy.deepcopy() to deepcopy object to prevent modifying the passed object. Are there any other recommendations ?
As far as I know, there no other better way, if you don't use third package.
You can use the __setattr__ magic method to implement a base class that allows you to "freeze" an object after you're done with it.
This is not bullet-proof; you can still access __dict__ to mutate the object, and you can also unfreeze the object by unsetting _frozen, and if the attribute's value itself is mutable, this doesn't help much (x.things.append('x') would work for a list of things).
class Freezable:
def freeze(self):
self._frozen = True
def __setattr__(self, key, value):
if getattr(self, "_frozen", False):
raise RuntimeError("%r is frozen" % self)
super().__setattr__(key, value)
class Person(Freezable):
def __init__(self, name):
self.name = name
p = Person("x")
print(p.name)
p.name = "y"
print(p.name)
p.freeze()
p.name = "q"
outputs
x
y
Traceback (most recent call last):
File "freezable.py", line 21, in <module>
p.name = 'q'
RuntimeError: <__main__.Person object at 0x10f82f3c8> is frozen
There are no really 100% watertight way, but you can make it difficult to inadvertently mutate an object that you want to keep frozen; the recommended way for most people is probably to use a frozen DataClass, or a frozen attrs class
In his talk on DataClasses (2018), #RaymonHettinger mentions three approaches: one way, is with a metaclass, another, like in the fractions module is to give attributes a read only property; the DataClass module extends __setattr__ and __delattr__, and overrides __hash__:
-> use a metaclass.
Good resources include #DavidBeasley books and talks at python.
-> give attributes a read only property
class SimpleFrozenObject:
def __init__(self, x=0):
self._x = x
#property
def x(self):
return self._x
f = SimpleFrozenObject()
f.x = 2 # raises AttributeError: can't set attribute
-> extend __setattr__ and __delattr__, and override `hash
class FrozenObject:
...
def __setattr__(self, name, value):
if type(self) is cls or name in (tuple of attributes to freeze,):
raise FrozenInstanceError(f'cannot assign to field {name}')
super(cls, self).__setattr__(name, value)
def __delattr__(self, name):
if type(self) is cls or name in (tuple of attributes to freeze,):
raise FrozenInstanceError(f'cannot delete field {name}')
super(cls, self).__delattr__(name, value)
def __hash__(self):
return hash((tuple of attributes to freeze,))
...
The library attrs also offers options to create immutable objects.
I'm trying to use properties and I tried to change python documentation's code. I'd expect the following would print anything, but it doesn't. Why does it not print anything?
class User:
def getter(self, name):
def get_prop(self):
print 'Getting {}'.format(name)
return getattr(self, name)
return get_prop
def setter(self, name):
def set_prop(self, value):
print 'Setting {} to {}'.format(name, value)
return setattr(self, name, value)
return set_prop
user_id = property(getter, setter)
u = User()
u.user_id = 10
u.user_id
There are two reasons your property doesn't work:
You need to use a new style class (by basing your class on object); you cannot use a property with a setter otherwise (only a getter is supported for old-style classes).
you are generating accessors as nested functions; you need to call those outer methods to generate those accessors, the property() function will no do this for you. As such, you can move those functions out of the class and use them as plain functions instead.
The following code works:
def getter(name):
def get_prop(self):
print 'Getting {}'.format(name)
return getattr(self, name)
return get_prop
def setter(name):
def set_prop(self, value):
print 'Setting {} to {}'.format(name, value)
return setattr(self, name, value)
return set_prop
class User(object):
user_id = property(getter('_user_id'), setter('_user_id'))
Note that I used _user_id for the property 'name' here, otherwise the getattr(self, name) call will trigger an infinite recursion; u.user_id would trigger a getattr(u, 'user_id') which triggers the property again.
Probably because properties only work with new-style objects. Change your class statement to:
class User(object):
Looking further, your getter and setter functions are returning functions, that you then do not call.
You seem to want dynamic attribute names, yet you have just one attribute name returned from property, in your case this is user_id. What is it you are trying to achieve?
This naive class attempts to mimic the attribute access of basic python objects. dict and cls explicitly stores the attributes and the class. The effect is that accessing .x of an instance will return dict[x], or if that fails, cls.x. Just like normal objects.
class Instance(object):
__slots__ = ["dict", "cls"]
def __getattribute__(self, key):
try:
return self.dict[key]
except KeyError:
return getattr(self.cls, key)
def __setattr__(self, key, value):
if key == "__class__":
self.cls = value
else:
self.dict[key] = value
But it's nowhere near as simple as that. One obvious issue is the complete disregard for descriptors. Just imagine that cls has properties. Doing Instance.some_property = 10 should access the property as defined in cls, but will instead happily set some_property as an attribute in dict.
Then there is the issue of binding methods of cls to instances of Instance, and possibly more that I don't even know.
There seem to be a lot of details to get the above class to function as close to python objects as possible, and the docs for descriptors I've read so far hasn't made it clear how to get, simply put, everything right.
What I am asking for is a reference for implementing a complete replacement for python's attribute access. That is, the above class, but correct.
Well, I needed this answer so I had to do the research. The below code covers the following:
data-descriptors are given precedence both when setting and getting attributes.
non-data descriptors are properly called in __getattribute__
There may be typos in the code below as I had to translate it from an internal project. And I am not sure if it is 100% like python objects, so if anyone could spot errors that would be great.
_sentinel = object()
def find_classattr(cls, key):
for base in cls.__mro__: # Using __mro__ for speed.
try: return base.__dict__[key]
except KeyError: pass
return _sentinel
class Instance(object):
__slots__ = ["dict", "cls"]
def __init__(self, d, cls):
object.__setattr__(self, "dict", d)
object.__setattr__(self, "cls", cls)
def __getattribute__(self, key):
d = object.__getattribute__(self, "dict")
cls = object.__getattribute__(self, "cls")
if key == "__class__":
return cls
# Data descriptors in the class, defined by presence of '__set__',
# overrides any other kind of attribute access.
cls_attr = find_classattr(cls, key)
if hasattr(cls_attr, '__set__'):
return cls_attr.__get__(self, cls)
# Next in order of precedence are instance attributes.
try:
return d[key]
except KeyError:
# Finally class attributes, that may or may not be non-data descriptors.
if hasattr(cls_attr, "__get__"):
return cls_attr.__get__(self, cls)
if cls_attr is not _sentinel:
return cls_attr
raise AttributeError("'{}' object has no attribute '{}'".format(
getattr(cls, '__name__', "?"), key))
def __setattr__(self, key, value):
d = object.__getattribute__(self, "dict")
cls = object.__getattribute__(self, "cls")
if key == "__class__":
object.__setattr__(self, "cls", value)
return
# Again, data descriptors override instance attributes.
cls_attr = find_classattr(cls, key)
if hasattr(cls_attr, '__set__'):
cls_attr.__set__(self, value)
else:
d[key] = value
Funny thing is I realized I had written exactly the same stuff before a couple of years ago, but the descriptor protocol is so arcane I had forgotten it since.
EDIT: Fixed bug where using getattr to find an attribute on the class would call it's descriptors on the class level (i.e. without the instance). Replaced it with a method that looks directly in the __dict__ of the bases.
If I have a class such as below (only with many more properties), is there are clean way to note which fields are required before calling a particular method?
class Example():
def __init__(self):
pass
#property
"""Have to use property methods to have docstrings..."""
def prop1(self):
return self._prop1
#prop1.setter
def task(self, value):
# validation logic..
self._prop1 = value
def method(self):
# check all required properties have been added
I could write an array by hand of all required propeties and loop through them in a method, but I was wondering if there is a cleaner way for example by implementing a #requiredProperty descriptor.
The class is used to generate a POST request for a web API. The request has 25+ parameters, some of which are required and some optional.
Rather than on the method calling the request having to loop through an array such as:
required_props = ['prop1','prop2',....]
I was hoping there was a way in Python of adding a required decorator to properties so I wouldn't have to keep track by hand. E.g.
#property, #required
def prop1(self):
return self._prop1
Would it not be best to make sure that all the attributes are supplied when an object is initialised? Then all your properties will be defined when you try to acces them.
For example,
class Example(object):
def __init__(self, prop1, prop2):
self.prop1 = prop1
self.prop2 = prop2
Also, note from PEP8:
For simple public data attributes, it
is best to expose just the attribute
name, without complicated
accessor/mutator methods.
So why use properties?
This should work the same way as in any OO language: A required property must be set during construction time. Calling the objects methods must never leave the object in a "bad" state, so that method can be called on any constructed object.
If the above doesn't hold true, you should think about refactoring your code.
Of course it is always possible to alter a python object to not be valid anymore by poking around in its guts. You don't do that unless you have a good reason. Don't bother checking for this, as your program should just blow up in your face whenever you do something stupid so you learn and stop.
It's hard to tell from your example what problem you are actually trying to solve, but I'm not convinced properties are the answer.
If you just want to check that an instance variable exists, you could use the special attribute __dict__, thus:
% cat ./test.py
#!/usr/bin/env python
class Example():
def __init__(self):
self.foo = None
def method(self):
assert 'foo' in self.__dict__
assert 'bar' in self.__dict__
Example().method()
% ./test.py
Traceback (most recent call last):
File "./test.py", line 12, in <module>
Example().method()
File "./test.py", line 10, in method
assert 'bar' in self.__dict__
AssertionError
But remember... EAFP: Easier to ask for forgiveness than permission.
As others have suggested, I suspect you are over-engineering. However, you could use a decorator to define 'required' attributes. Something along the lines of:
import functools
class MissingAttributeError(Exception):
pass
def requires(*required_attrs):
def wrapper(method):
#functools.wraps(method)
def inner_wrapper(self, *args, **kargs):
if not all(hasattr(self, attr) for attr in required_attrs):
raise MissingAttributeError()
return method(self, *args, **kargs)
return inner_wrapper
return wrapper
class Test(object):
def __init__(self, spam, eggs):
self.spam = spam
self.eggs = eggs
#requires('spam', 'eggs', 'ham')
def something(self):
return 'Done'
t = Test('fu', 'bar')
t.something() ## fails
t.ham = 'nicer than spam'
t.something() ## succeeds
Although defining attribute dependencies this way has a certain neatness to it, I'm not sure I recommend it.