Develop Reference

python jedi, autocompletion for dynamic attribute

I'm trying to write a class. Objects of that class can take a label/value pair and store it in such a way that label can be accessed as an attribute, returning value: obj.label -> value
The main goal here is to get autocompletion in jupyter notebooks, so obj.<tab> should produce the list of labels as autocompletion suggestion. The below class accomplishes this:
class Autocompleter:
def __init__(self, ):
self._funcs = {}
def add(self, label):
self._funcs[label] = 1.
def __dir__(self):
return list(self._funcs.keys())
def __getattr__(self, name):
if name in dir(self):
return self._funcs[name]
The problem: When accessing an invalid attribute, the __getattr__ simply returns None. I'd rather have it throw an exception. I can think of 2 ways to achieve this, but unfortunately both break the autocompletion:
Changing __getattr__ to:
def __getattr__(self, name):
return self._funcs[name]
or
def __getattr__(self, name):
if name in dir(self):
return self._funcs[name]
else:
raise Exception('invalid name')
produces the desired exception, but breaks the autocompletion:
a = Autocompleter()
a.add('foo')
Now a.<tab> does not suggest foo for autocompletion, it simply does nothing. As far as I can tell, jedi is used by default for autompletion in jupyterlab.
Question: Is there a way to get both the exception on invalid names and the autocomplete feature working?

I figured it out myself. The correct way to handle an invalid attribute name is to raise an AttributeError. This can then be understood by jedi.
def __getattr__(self, name):
if name in dir(self):
return self._funcs[name]
raise AttributeError(name)
Note that the __dir__ method is still needed.

The proper way of completely overriding attribute access in Python?

This naive class attempts to mimic the attribute access of basic python objects. dict and cls explicitly stores the attributes and the class. The effect is that accessing .x of an instance will return dict[x], or if that fails, cls.x. Just like normal objects.
class Instance(object):
__slots__ = ["dict", "cls"]
def __getattribute__(self, key):
try:
return self.dict[key]
except KeyError:
return getattr(self.cls, key)
def __setattr__(self, key, value):
if key == "__class__":
self.cls = value
else:
self.dict[key] = value
But it's nowhere near as simple as that. One obvious issue is the complete disregard for descriptors. Just imagine that cls has properties. Doing Instance.some_property = 10 should access the property as defined in cls, but will instead happily set some_property as an attribute in dict.
Then there is the issue of binding methods of cls to instances of Instance, and possibly more that I don't even know.
There seem to be a lot of details to get the above class to function as close to python objects as possible, and the docs for descriptors I've read so far hasn't made it clear how to get, simply put, everything right.
What I am asking for is a reference for implementing a complete replacement for python's attribute access. That is, the above class, but correct.

Well, I needed this answer so I had to do the research. The below code covers the following:
data-descriptors are given precedence both when setting and getting attributes.
non-data descriptors are properly called in __getattribute__
There may be typos in the code below as I had to translate it from an internal project. And I am not sure if it is 100% like python objects, so if anyone could spot errors that would be great.
_sentinel = object()
def find_classattr(cls, key):
for base in cls.__mro__: # Using __mro__ for speed.
try: return base.__dict__[key]
except KeyError: pass
return _sentinel
class Instance(object):
__slots__ = ["dict", "cls"]
def __init__(self, d, cls):
object.__setattr__(self, "dict", d)
object.__setattr__(self, "cls", cls)
def __getattribute__(self, key):
d = object.__getattribute__(self, "dict")
cls = object.__getattribute__(self, "cls")
if key == "__class__":
return cls
# Data descriptors in the class, defined by presence of '__set__',
# overrides any other kind of attribute access.
cls_attr = find_classattr(cls, key)
if hasattr(cls_attr, '__set__'):
return cls_attr.__get__(self, cls)
# Next in order of precedence are instance attributes.
try:
return d[key]
except KeyError:
# Finally class attributes, that may or may not be non-data descriptors.
if hasattr(cls_attr, "__get__"):
return cls_attr.__get__(self, cls)
if cls_attr is not _sentinel:
return cls_attr
raise AttributeError("'{}' object has no attribute '{}'".format(
getattr(cls, '__name__', "?"), key))
def __setattr__(self, key, value):
d = object.__getattribute__(self, "dict")
cls = object.__getattribute__(self, "cls")
if key == "__class__":
object.__setattr__(self, "cls", value)
return
# Again, data descriptors override instance attributes.
cls_attr = find_classattr(cls, key)
if hasattr(cls_attr, '__set__'):
cls_attr.__set__(self, value)
else:
d[key] = value
Funny thing is I realized I had written exactly the same stuff before a couple of years ago, but the descriptor protocol is so arcane I had forgotten it since.
EDIT: Fixed bug where using getattr to find an attribute on the class would call it's descriptors on the class level (i.e. without the instance). Replaced it with a method that looks directly in the __dict__ of the bases.

Python metaclasses: Why isn't __setattr__ called for attributes set during class definition?

I have the following python code:
class FooMeta(type):
def __setattr__(self, name, value):
print name, value
return super(FooMeta, self).__setattr__(name, value)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
I would have expected __setattr__ of the meta class being called for both FOO and a. However, it is not called at all. When I assign something to Foo.whatever after the class has been defined the method is called.
What's the reason for this behaviour and is there a way to intercept the assignments that happen during the creation of the class? Using attrs in __new__ won't work since I'd like to check if a method is being redefined.

A class block is roughly syntactic sugar for building a dictionary, and then invoking a metaclass to build the class object.
This:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Comes out pretty much as if you'd written:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Only without the namespace pollution (and in reality there's also a search through all the bases to determine the metaclass, or whether there's a metaclass conflict, but I'm ignoring that here).
The metaclass' __setattr__ can control what happens when you try to set an attribute on one of its instances (the class object), but inside the class block you're not doing that, you're inserting into a dictionary object, so the dict class controls what's going on, not your metaclass. So you're out of luck.
Unless you're using Python 3.x! In Python 3.x you can define a __prepare__ classmethod (or staticmethod) on a metaclass, which controls what object is used to accumulate attributes set within a class block before they're passed to the metaclass constructor. The default __prepare__ simply returns a normal dictionary, but you could build a custom dict-like class that doesn't allow keys to be redefined, and use that to accumulate your attributes:
from collections import MutableMapping
class SingleAssignDict(MutableMapping):
def __init__(self, *args, **kwargs):
self._d = dict(*args, **kwargs)
def __getitem__(self, key):
return self._d[key]
def __setitem__(self, key, value):
if key in self._d:
raise ValueError(
'Key {!r} already exists in SingleAssignDict'.format(key)
)
else:
self._d[key] = value
def __delitem__(self, key):
del self._d[key]
def __iter__(self):
return iter(self._d)
def __len__(self):
return len(self._d)
def __contains__(self, key):
return key in self._d
def __repr__(self):
return '{}({!r})'.format(type(self).__name__, self._d)
class RedefBlocker(type):
#classmethod
def __prepare__(metacls, name, bases, **kwargs):
return SingleAssignDict()
def __new__(metacls, name, bases, sad):
return super().__new__(metacls, name, bases, dict(sad))
class Okay(metaclass=RedefBlocker):
a = 1
b = 2
class Boom(metaclass=RedefBlocker):
a = 1
b = 2
a = 3
Running this gives me:
Traceback (most recent call last):
File "/tmp/redef.py", line 50, in <module>
class Boom(metaclass=RedefBlocker):
File "/tmp/redef.py", line 53, in Boom
a = 3
File "/tmp/redef.py", line 15, in __setitem__
'Key {!r} already exists in SingleAssignDict'.format(key)
ValueError: Key 'a' already exists in SingleAssignDict
Some notes:
__prepare__ has to be a classmethod or staticmethod, because it's being called before the metaclass' instance (your class) exists.
type still needs its third parameter to be a real dict, so you have to have a __new__ method that converts the SingleAssignDict to a normal one
I could have subclassed dict, which would probably have avoided (2), but I really dislike doing that because of how the non-basic methods like update don't respect your overrides of the basic methods like __setitem__. So I prefer to subclass collections.MutableMapping and wrap a dictionary.
The actual Okay.__dict__ object is a normal dictionary, because it was set by type and type is finicky about the kind of dictionary it wants. This means that overwriting class attributes after class creation does not raise an exception. You can overwrite the __dict__ attribute after the superclass call in __new__ if you want to maintain the no-overwriting forced by the class object's dictionary.
Sadly this technique is unavailable in Python 2.x (I checked). The __prepare__ method isn't invoked, which makes sense as in Python 2.x the metaclass is determined by the __metaclass__ magic attribute rather than a special keyword in the classblock; which means the dict object used to accumulate attributes for the class block already exists by the time the metaclass is known.
Compare Python 2:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Where the metaclass to invoke is determined from the dictionary, versus Python 3:
class Foo(metaclass=FooMeta):
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = FooMeta.__prepare__('Foo', ())
d['Foo'] = 123
def a(self):
pass
d['a'] = a
Foo = FooMeta('Foo', (), d)
Where the dictionary to use is determined from the metaclass.

There are no assignments happening during the creation of the class. Or: they are happening, but not in the context you think they are. All class attributes are collected from class body scope and passed to metaclass' __new__, as the last argument:
class FooMeta(type):
def __new__(self, name, bases, attrs):
print attrs
return type.__new__(self, name, bases, attrs)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
Reason: when the code in the class body executes, there's no class yet. Which means there's no opportunity for metaclass to intercept anything yet.

Class attributes are passed to the metaclass as a single dictionary and my hypothesis is that this is used to update the __dict__ attribute of the class all at once, e.g. something like cls.__dict__.update(dct) rather than doing setattr() on each item. More to the point, it's all handled in C-land and simply wasn't written to call a custom __setattr__().
It's easy enough to do whatever you want to the attributes of the class in your metaclass's __init__() method, since you're passed the class namespace as a dict, so just do that.

During the class creation, your namespace is evaluated to a dict and passed as an argument to the metaclass, together with the class name and base classes. Because of that, assigning a class attribute inside the class definition wouldn't work the way you expect. It doesn't create an empty class and assign everything. You also can't have duplicated keys in a dict, so during class creation attributes are already deduplicated. Only by setting an attribute after the class definition you can trigger your custom __setattr__.
Because the namespace is a dict, there's no way for you to check duplicated methods, as suggested by your other question. The only practical way to do that is parsing the source code.

Python name of class in class body

Is it possible to get the class name within the body of a class definition?
For example,
class Foo():
x = magic() # x should now be 'Foo'
I know that I can do this statically outside of the class body using a class method:
class Bar():
#classmethod
def magic(cls):
print cls.__name__
Bar.magic()
However this isn't what I want, I want the class name in the class body

Ok - got one more solution - this one is actually not that complex!
import traceback
def magic():
return traceback.extract_stack()[-2][2]
class Something(object):
print magic()
It will print out "Something". I'm not sure if extracted stack format is standardised in any way, but it works for python 2.6 (and 2.7 and 3.1)

AFAIK, the class object is not available until the class definition has been "executed", so it's not possible to get it during class definition.
If you need the class name for later use but don't use it during class definition (e.g. to compute other field names, or some such thing), then you can still automate the process using a class decorator.
def classname ( field ):
def decorator ( klass ):
setattr(klass, field, klass.__name__)
return klass
return decorator
(Caveat: not tested.)
With this definition, you can get something like:
#classname(field='x')
class Foo:
pass
and you would get field x with the class name in it, as in:
print Foo.x

Here you have a working solution for your specific case, but beware (I wrote it mainly to demonstrate that it IS indeed possible to do something like this):
You shouldn't use it
It is very specific
It has many limitations
I was just having fun with this
It is black magic
It may not work for your use case
It is not threadsafe
Do I have already said that you shouldn't use it?
Anyway, here you have the code:
import inspect
def NameAwareClassType():
frameInfo = inspect.getouterframes(inspect.currentframe())[1]
codeContext = frameInfo[4][0]
className = codeContext.split(' ', 1)[1].split('(', 1)[0]
class ClassNameGlobalRemoverType(type):
def __new__(mcs, name, bases, dict):
if name == className:
del globals()['__clsname__']
return type.__new__(mcs, name, bases, dict)
class NameAwareClass(object):
__metaclass__ = ClassNameGlobalRemoverType
globals()['__clsname__'] = className
return NameAwareClass
class A(NameAwareClassType()):
print __clsname__
def __init__(self):
pass
print __clsname__
Edit: https://gist.github.com/1085475 — there you have a version which allows to use __clsname__ during method execution; makes not much sense, as self.__class__.__name__ is a better approach and the __clsname__ variable does not hold a string anymore (I'm having fun experimenting with this)

I don't know of an elegant way to do this in Python 2.x -- but it's an interpreted language which means that something relatively simple along the following lines will do what you want and would be safe if you're sure of the code being executed:
classdef = """\
class %(classname)s(object):
x = '%(classname)s'
print x
"""
exec classdef % {'classname': 'Foo'}
foo = Foo()
print foo

class Bar():
#classmethod
def magic(cls):
return cls.__name__
#property
def x(self):
return self.magic()
def y(self):
return self.x
>>> a = Bar()
>>> a.x
'Bar'
>>> a.y()
'Bar'
This way you can use x as an attribute, at least within any instance and static methods. In class methods, you can just get the class name from the cls attribute anyway.

Accessing the class that owns a decorated method from the decorator

I'm writing a decorator for methods that must inspect the parent methods (the methods of the same name in the parents of the class in which I'm decorating).
Example (from the fourth example of PEP 318):
def returns(rtype):
def check_returns(f):
def new_f(*args, **kwds):
result = f(*args, **kwds)
assert isinstance(result, rtype), \
"return value %r does not match %s" % (result,rtype)
return result
new_f.func_name = f.func_name
# here I want to reach the class owning the decorated method f,
# it should give me the class A
return new_f
return check_returns
class A(object):
#returns(int)
def compute(self, value):
return value * 3
So I'm looking for the code to type in place of # here I want...
Thanks.

As bobince said it, you can't access the surrounding class, because at the time the decorator is invoked, the class does not exist yet. If you need access to the full dictionary of the class and the bases, you should consider a metaclass:
__metaclass__
This variable can be any callable accepting arguments for name, bases, and dict. Upon class creation, the callable is used instead of the built-in type().
Basically, we convert the returns decorator into something that just tells the metaclass to do some magic on class construction:
class CheckedReturnType(object):
def __init__(self, meth, rtype):
self.meth = meth
self.rtype = rtype
def returns(rtype):
def _inner(f):
return CheckedReturnType(f, rtype)
return _inner
class BaseInspector(type):
def __new__(mcs, name, bases, dct):
for obj_name, obj in dct.iteritems():
if isinstance(obj, CheckedReturnType):
# do your wrapping & checking here, base classes are in bases
# reassign to dct
return type.__new__(mcs, name, bases, dct)
class A(object):
__metaclass__ = BaseInspector
#returns(int)
def compute(self, value):
return value * 3
Mind that I have not tested this code, please leave comments if I should update this.
There are some articles on metaclasses by the highly recommendable David Mertz, which you might find interesting in this context.

here I want to reach the class owning the decorated method f
You can't because at the point of decoration, no class owns the method f.
class A(object):
#returns(int)
def compute(self, value):
return value * 3
Is the same as saying:
class A(object):
pass
#returns(int)
def compute(self, value):
return value*3
A.compute= compute
Clearly, the returns() decorator is built before the function is assigned to an owner class.
Now when you write a function to a class (either inline, or explicitly like this) it becomes an unbound method object. Now it has a reference to its owner class, which you can get by saying:
>>> A.compute.im_class
<class '__main__.A'>
So you can read f.im_class inside ‘new_f’, which is executed after the assignment, but not in the decorator itself.
(And even then it's a bit ugly relying on a CPython implementation detail if you don't need to. I'm not quite sure what you're trying to do, but things involving “get the owner class” are often doable using metaclasses.)