Related
Is there a magic method that can overload the assignment operator, like __assign__(self, new_value)?
I'd like to forbid a re-bind for an instance:
class Protect():
def __assign__(self, value):
raise Exception("This is an ex-parrot")
var = Protect() # once assigned...
var = 1 # this should raise Exception()
Is it possible? Is it insane? Should I be on medicine?
The way you describe it is absolutely not possible. Assignment to a name is a fundamental feature of Python and no hooks have been provided to change its behavior.
However, assignment to a member in a class instance can be controlled as you want, by overriding .__setattr__().
class MyClass(object):
def __init__(self, x):
self.x = x
self._locked = True
def __setattr__(self, name, value):
if self.__dict__.get("_locked", False) and name == "x":
raise AttributeError("MyClass does not allow assignment to .x member")
self.__dict__[name] = value
>>> m = MyClass(3)
>>> m.x
3
>>> m.x = 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __setattr__
AttributeError: MyClass does not allow assignment to .x member
Note that there is a member variable, _locked, that controls whether the assignment is permitted. You can unlock it to update the value.
No, as assignment is a language intrinsic which doesn't have a modification hook.
I don't think it's possible. The way I see it, assignment to a variable doesn't do anything to the object it previously referred to: it's just that the variable "points" to a different object now.
In [3]: class My():
...: def __init__(self, id):
...: self.id=id
...:
In [4]: a = My(1)
In [5]: b = a
In [6]: a = 1
In [7]: b
Out[7]: <__main__.My instance at 0xb689d14c>
In [8]: b.id
Out[8]: 1 # the object is unchanged!
However, you can mimic the desired behavior by creating a wrapper object with __setitem__() or __setattr__() methods that raise an exception, and keep the "unchangeable" stuff inside.
Inside a module, this is absolutely possible, via a bit of dark magic.
import sys
tst = sys.modules['tst']
class Protect():
def __assign__(self, value):
raise Exception("This is an ex-parrot")
var = Protect() # once assigned...
Module = type(tst)
class ProtectedModule(Module):
def __setattr__(self, attr, val):
exists = getattr(self, attr, None)
if exists is not None and hasattr(exists, '__assign__'):
exists.__assign__(val)
super().__setattr__(attr, val)
tst.__class__ = ProtectedModule
The above example assumes the code resides in a module named tst. You can do this in the repl by changing tst to __main__.
If you want to protect access through the local module, make all writes to it through tst.var = newval.
Using the top-level namespace, this is impossible. When you run
var = 1
It stores the key var and the value 1 in the global dictionary. It is roughly equivalent to calling globals().__setitem__('var', 1). The problem is that you cannot replace the global dictionary in a running script (you probably can by messing with the stack, but that is not a good idea). However you can execute code in a secondary namespace, and provide a custom dictionary for its globals.
class myglobals(dict):
def __setitem__(self, key, value):
if key=='val':
raise TypeError()
dict.__setitem__(self, key, value)
myg = myglobals()
dict.__setitem__(myg, 'val', 'protected')
import code
code.InteractiveConsole(locals=myg).interact()
That will fire up a REPL which almost operates normally, but refuses any attempts to set the variable val. You could also use execfile(filename, myg). Note this doesn't protect against malicious code.
I will burn in Python hell, but what's life without a little fun.
Important disclaimers:
I only provide this example for fun
I'm 100% sure I don't understand this well
It might not even be safe to do this, in any sense
I don't think this is practical
I don't think this is a good idea
I don't even want to seriously try to implement this
This doesn't work for jupyter (probably ipython too)*
Maybe you can't overload assignment, but you can (at least with Python ~3.9) achieve what you want even at the top-level namespace. It will be hard doing it "properly" for all cases, but here's a small example by hacking audithooks:
import sys
import ast
import inspect
import dis
import types
def hook(name, tup):
if name == "exec" and tup:
if tup and isinstance(tup[0], types.CodeType):
# Probably only works for my example
code = tup[0]
# We want to parse that code and find if it "stores" a variable.
# The ops for the example code would look something like this:
# ['LOAD_CONST', '<0>', 'STORE_NAME', '<0>',
# 'LOAD_CONST', 'POP_TOP', 'RETURN_VALUE', '<0>']
store_instruction_arg = None
instructions = [dis.opname[op] for op in code.co_code]
# Track the index so we can find the '<NUM>' index into the names
for i, instruction in enumerate(instructions):
# You might need to implement more logic here
# or catch more cases
if instruction == "STORE_NAME":
# store_instruction_arg in our case is 0.
# This might be the wrong way to parse get this value,
# but oh well.
store_instruction_arg = code.co_code[i + 1]
break
if store_instruction_arg is not None:
# code.co_names here is: ('a',)
var_name = code.co_names[store_instruction_arg]
# Check if the variable name has been previously defined.
# Will this work inside a function? a class? another
# module? Well... :D
if var_name in globals():
raise Exception("Cannot re-assign variable")
# Magic
sys.addaudithook(hook)
And here's the example:
>>> a = "123"
>>> a = 123
Traceback (most recent call last):
File "<stdin>", line 21, in hook
Exception: Cannot re-assign variable
>>> a
'123'
*For Jupyter I found another way that looked a tiny bit cleaner because I parsed the AST instead of the code object:
import sys
import ast
def hook(name, tup):
if name == "compile" and tup:
ast_mod = tup[0]
if isinstance(ast_mod, ast.Module):
assign_token = None
for token in ast_mod.body:
if isinstance(token, ast.Assign):
target, value = token.targets[0], token.value
var_name = target.id
if var_name in globals():
raise Exception("Can't re-assign variable")
sys.addaudithook(hook)
No there isn't
Think about it, in your example you are rebinding the name var to a new value.
You aren't actually touching the instance of Protect.
If the name you wish to rebind is in fact a property of some other entity i.e
myobj.var then you can prevent assigning a value to the property/attribute of the entity.
But I assume thats not what you want from your example.
Yes, It's possible, you can handle __assign__ via modify ast.
pip install assign
Test with:
class T():
def __assign__(self, v):
print('called with %s' % v)
b = T()
c = b
You will get
>>> import magic
>>> import test
called with c
The project is at https://github.com/RyanKung/assign
And the simpler gist: https://gist.github.com/RyanKung/4830d6c8474e6bcefa4edd13f122b4df
Generally, the best approach I found is overriding __ilshift__ as a setter and __rlshift__ as a getter, being duplicated by the property decorator.
It is almost the last operator being resolved just (| & ^) and logical are lower.
It is rarely used (__lrshift__ is less, but it can be taken to account).
Within using of PyPi assign package only forward assignment can be controlled, so actual 'strength' of the operator is lower.
PyPi assign package example:
class Test:
def __init__(self, val, name):
self._val = val
self._name = name
self.named = False
def __assign__(self, other):
if hasattr(other, 'val'):
other = other.val
self.set(other)
return self
def __rassign__(self, other):
return self.get()
def set(self, val):
self._val = val
def get(self):
if self.named:
return self._name
return self._val
#property
def val(self):
return self._val
x = Test(1, 'x')
y = Test(2, 'y')
print('x.val =', x.val)
print('y.val =', y.val)
x = y
print('x.val =', x.val)
z: int = None
z = x
print('z =', z)
x = 3
y = x
print('y.val =', y.val)
y.val = 4
output:
x.val = 1
y.val = 2
x.val = 2
z = <__main__.Test object at 0x0000029209DFD978>
Traceback (most recent call last):
File "E:\packages\pyksp\pyksp\compiler2\simple_test2.py", line 44, in <module>
print('y.val =', y.val)
AttributeError: 'int' object has no attribute 'val'
The same with shift:
class Test:
def __init__(self, val, name):
self._val = val
self._name = name
self.named = False
def __ilshift__(self, other):
if hasattr(other, 'val'):
other = other.val
self.set(other)
return self
def __rlshift__(self, other):
return self.get()
def set(self, val):
self._val = val
def get(self):
if self.named:
return self._name
return self._val
#property
def val(self):
return self._val
x = Test(1, 'x')
y = Test(2, 'y')
print('x.val =', x.val)
print('y.val =', y.val)
x <<= y
print('x.val =', x.val)
z: int = None
z <<= x
print('z =', z)
x <<= 3
y <<= x
print('y.val =', y.val)
y.val = 4
output:
x.val = 1
y.val = 2
x.val = 2
z = 2
y.val = 3
Traceback (most recent call last):
File "E:\packages\pyksp\pyksp\compiler2\simple_test.py", line 45, in <module>
y.val = 4
AttributeError: can't set attribute
So <<= operator within getting value at a property is the much more visually clean solution and it is not attempting user to make some reflective mistakes like:
var1.val = 1
var2.val = 2
# if we have to check type of input
var1.val = var2
# but it could be accendently typed worse,
# skipping the type-check:
var1.val = var2.val
# or much more worse:
somevar = var1 + var2
var1 += var2
# sic!
var1 = var2
In the global namespace this is not possible, but you could take advantage of more advanced Python metaprogramming to prevent multiple instances of a the Protect object from being created. The Singleton pattern is good example of this.
In the case of a Singleton you would ensure that once instantiated, even if the original variable referencing the instance is reassigned, that the object would persist. Any subsequent instances would just return a reference to the same object.
Despite this pattern, you would never be able to prevent a global variable name itself from being reassigned.
As mentioned by other people, there is no way to do it directly. It can be overridden for class members though, which is good for many cases.
As Ryan Kung mentioned, the AST of a package can be instrumented so that all assignments can have a side effect if the class assigned implements specific method(s). Building on his work to handle object creation and attribute assignment cases, the modified code and a full description is available here:
https://github.com/patgolez10/assignhooks
The package can be installed as: pip3 install assignhooks
Example <testmod.py>:
class SampleClass():
name = None
def __assignpre__(self, lhs_name, rhs_name, rhs):
print('PRE: assigning %s = %s' % (lhs_name, rhs_name))
# modify rhs if needed before assignment
if rhs.name is None:
rhs.name = lhs_name
return rhs
def __assignpost__(self, lhs_name, rhs_name):
print('POST: lhs', self)
print('POST: assigning %s = %s' % (lhs_name, rhs_name))
def myfunc():
b = SampleClass()
c = b
print('b.name', b.name)
to instrument it, e.g. <test.py>
import assignhooks
assignhooks.instrument.start() # instrument from now on
import testmod
assignhooks.instrument.stop() # stop instrumenting
# ... other imports and code bellow ...
testmod.myfunc()
Will produce:
$ python3 ./test.py
POST: lhs <testmod.SampleClass object at 0x1041dcc70>
POST: assigning b = SampleClass
PRE: assigning c = b
POST: lhs <testmod.SampleClass object at 0x1041dcc70>
POST: assigning c = b
b.name b
Beginning Python 3.8, it is possible to hint that a value is read-only using typing.Final. What this means is that nothing changes at runtime, allowing anyone to change the value, but if you're using any linter that can read type-hints then it's going to warn the user if they attempt to assign it.
from typing import Final
x: Final[int] = 3
x = 5 # Cannot assign to final name "x" (mypy)
This makes for way cleaner code, but it puts full trust in the user to respect it at runtime, making no attempt to stop users from changing values.
Another common pattern is to expose functions instead of module constants, like sys.getrecursionlimit and sys.setrecursionlimit.
def get_x() -> int:
return 3
Although users can do module.get_x = my_get_x, there's an obvious attempt on the user's part to break it, which can't be fixed. In this way we can prevent people from "accidentally" changing values in our module with minimal complexity.
A ugly solution is to reassign on destructor. But it's no real overload assignment.
import copy
global a
class MyClass():
def __init__(self):
a = 1000
# ...
def __del__(self):
a = copy.copy(self)
a = MyClass()
a = 1
Is there a magic method that can overload the assignment operator, like __assign__(self, new_value)?
I'd like to forbid a re-bind for an instance:
class Protect():
def __assign__(self, value):
raise Exception("This is an ex-parrot")
var = Protect() # once assigned...
var = 1 # this should raise Exception()
Is it possible? Is it insane? Should I be on medicine?
The way you describe it is absolutely not possible. Assignment to a name is a fundamental feature of Python and no hooks have been provided to change its behavior.
However, assignment to a member in a class instance can be controlled as you want, by overriding .__setattr__().
class MyClass(object):
def __init__(self, x):
self.x = x
self._locked = True
def __setattr__(self, name, value):
if self.__dict__.get("_locked", False) and name == "x":
raise AttributeError("MyClass does not allow assignment to .x member")
self.__dict__[name] = value
>>> m = MyClass(3)
>>> m.x
3
>>> m.x = 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __setattr__
AttributeError: MyClass does not allow assignment to .x member
Note that there is a member variable, _locked, that controls whether the assignment is permitted. You can unlock it to update the value.
No, as assignment is a language intrinsic which doesn't have a modification hook.
I don't think it's possible. The way I see it, assignment to a variable doesn't do anything to the object it previously referred to: it's just that the variable "points" to a different object now.
In [3]: class My():
...: def __init__(self, id):
...: self.id=id
...:
In [4]: a = My(1)
In [5]: b = a
In [6]: a = 1
In [7]: b
Out[7]: <__main__.My instance at 0xb689d14c>
In [8]: b.id
Out[8]: 1 # the object is unchanged!
However, you can mimic the desired behavior by creating a wrapper object with __setitem__() or __setattr__() methods that raise an exception, and keep the "unchangeable" stuff inside.
Inside a module, this is absolutely possible, via a bit of dark magic.
import sys
tst = sys.modules['tst']
class Protect():
def __assign__(self, value):
raise Exception("This is an ex-parrot")
var = Protect() # once assigned...
Module = type(tst)
class ProtectedModule(Module):
def __setattr__(self, attr, val):
exists = getattr(self, attr, None)
if exists is not None and hasattr(exists, '__assign__'):
exists.__assign__(val)
super().__setattr__(attr, val)
tst.__class__ = ProtectedModule
The above example assumes the code resides in a module named tst. You can do this in the repl by changing tst to __main__.
If you want to protect access through the local module, make all writes to it through tst.var = newval.
Using the top-level namespace, this is impossible. When you run
var = 1
It stores the key var and the value 1 in the global dictionary. It is roughly equivalent to calling globals().__setitem__('var', 1). The problem is that you cannot replace the global dictionary in a running script (you probably can by messing with the stack, but that is not a good idea). However you can execute code in a secondary namespace, and provide a custom dictionary for its globals.
class myglobals(dict):
def __setitem__(self, key, value):
if key=='val':
raise TypeError()
dict.__setitem__(self, key, value)
myg = myglobals()
dict.__setitem__(myg, 'val', 'protected')
import code
code.InteractiveConsole(locals=myg).interact()
That will fire up a REPL which almost operates normally, but refuses any attempts to set the variable val. You could also use execfile(filename, myg). Note this doesn't protect against malicious code.
I will burn in Python hell, but what's life without a little fun.
Important disclaimers:
I only provide this example for fun
I'm 100% sure I don't understand this well
It might not even be safe to do this, in any sense
I don't think this is practical
I don't think this is a good idea
I don't even want to seriously try to implement this
This doesn't work for jupyter (probably ipython too)*
Maybe you can't overload assignment, but you can (at least with Python ~3.9) achieve what you want even at the top-level namespace. It will be hard doing it "properly" for all cases, but here's a small example by hacking audithooks:
import sys
import ast
import inspect
import dis
import types
def hook(name, tup):
if name == "exec" and tup:
if tup and isinstance(tup[0], types.CodeType):
# Probably only works for my example
code = tup[0]
# We want to parse that code and find if it "stores" a variable.
# The ops for the example code would look something like this:
# ['LOAD_CONST', '<0>', 'STORE_NAME', '<0>',
# 'LOAD_CONST', 'POP_TOP', 'RETURN_VALUE', '<0>']
store_instruction_arg = None
instructions = [dis.opname[op] for op in code.co_code]
# Track the index so we can find the '<NUM>' index into the names
for i, instruction in enumerate(instructions):
# You might need to implement more logic here
# or catch more cases
if instruction == "STORE_NAME":
# store_instruction_arg in our case is 0.
# This might be the wrong way to parse get this value,
# but oh well.
store_instruction_arg = code.co_code[i + 1]
break
if store_instruction_arg is not None:
# code.co_names here is: ('a',)
var_name = code.co_names[store_instruction_arg]
# Check if the variable name has been previously defined.
# Will this work inside a function? a class? another
# module? Well... :D
if var_name in globals():
raise Exception("Cannot re-assign variable")
# Magic
sys.addaudithook(hook)
And here's the example:
>>> a = "123"
>>> a = 123
Traceback (most recent call last):
File "<stdin>", line 21, in hook
Exception: Cannot re-assign variable
>>> a
'123'
*For Jupyter I found another way that looked a tiny bit cleaner because I parsed the AST instead of the code object:
import sys
import ast
def hook(name, tup):
if name == "compile" and tup:
ast_mod = tup[0]
if isinstance(ast_mod, ast.Module):
assign_token = None
for token in ast_mod.body:
if isinstance(token, ast.Assign):
target, value = token.targets[0], token.value
var_name = target.id
if var_name in globals():
raise Exception("Can't re-assign variable")
sys.addaudithook(hook)
No there isn't
Think about it, in your example you are rebinding the name var to a new value.
You aren't actually touching the instance of Protect.
If the name you wish to rebind is in fact a property of some other entity i.e
myobj.var then you can prevent assigning a value to the property/attribute of the entity.
But I assume thats not what you want from your example.
Yes, It's possible, you can handle __assign__ via modify ast.
pip install assign
Test with:
class T():
def __assign__(self, v):
print('called with %s' % v)
b = T()
c = b
You will get
>>> import magic
>>> import test
called with c
The project is at https://github.com/RyanKung/assign
And the simpler gist: https://gist.github.com/RyanKung/4830d6c8474e6bcefa4edd13f122b4df
Generally, the best approach I found is overriding __ilshift__ as a setter and __rlshift__ as a getter, being duplicated by the property decorator.
It is almost the last operator being resolved just (| & ^) and logical are lower.
It is rarely used (__lrshift__ is less, but it can be taken to account).
Within using of PyPi assign package only forward assignment can be controlled, so actual 'strength' of the operator is lower.
PyPi assign package example:
class Test:
def __init__(self, val, name):
self._val = val
self._name = name
self.named = False
def __assign__(self, other):
if hasattr(other, 'val'):
other = other.val
self.set(other)
return self
def __rassign__(self, other):
return self.get()
def set(self, val):
self._val = val
def get(self):
if self.named:
return self._name
return self._val
#property
def val(self):
return self._val
x = Test(1, 'x')
y = Test(2, 'y')
print('x.val =', x.val)
print('y.val =', y.val)
x = y
print('x.val =', x.val)
z: int = None
z = x
print('z =', z)
x = 3
y = x
print('y.val =', y.val)
y.val = 4
output:
x.val = 1
y.val = 2
x.val = 2
z = <__main__.Test object at 0x0000029209DFD978>
Traceback (most recent call last):
File "E:\packages\pyksp\pyksp\compiler2\simple_test2.py", line 44, in <module>
print('y.val =', y.val)
AttributeError: 'int' object has no attribute 'val'
The same with shift:
class Test:
def __init__(self, val, name):
self._val = val
self._name = name
self.named = False
def __ilshift__(self, other):
if hasattr(other, 'val'):
other = other.val
self.set(other)
return self
def __rlshift__(self, other):
return self.get()
def set(self, val):
self._val = val
def get(self):
if self.named:
return self._name
return self._val
#property
def val(self):
return self._val
x = Test(1, 'x')
y = Test(2, 'y')
print('x.val =', x.val)
print('y.val =', y.val)
x <<= y
print('x.val =', x.val)
z: int = None
z <<= x
print('z =', z)
x <<= 3
y <<= x
print('y.val =', y.val)
y.val = 4
output:
x.val = 1
y.val = 2
x.val = 2
z = 2
y.val = 3
Traceback (most recent call last):
File "E:\packages\pyksp\pyksp\compiler2\simple_test.py", line 45, in <module>
y.val = 4
AttributeError: can't set attribute
So <<= operator within getting value at a property is the much more visually clean solution and it is not attempting user to make some reflective mistakes like:
var1.val = 1
var2.val = 2
# if we have to check type of input
var1.val = var2
# but it could be accendently typed worse,
# skipping the type-check:
var1.val = var2.val
# or much more worse:
somevar = var1 + var2
var1 += var2
# sic!
var1 = var2
In the global namespace this is not possible, but you could take advantage of more advanced Python metaprogramming to prevent multiple instances of a the Protect object from being created. The Singleton pattern is good example of this.
In the case of a Singleton you would ensure that once instantiated, even if the original variable referencing the instance is reassigned, that the object would persist. Any subsequent instances would just return a reference to the same object.
Despite this pattern, you would never be able to prevent a global variable name itself from being reassigned.
As mentioned by other people, there is no way to do it directly. It can be overridden for class members though, which is good for many cases.
As Ryan Kung mentioned, the AST of a package can be instrumented so that all assignments can have a side effect if the class assigned implements specific method(s). Building on his work to handle object creation and attribute assignment cases, the modified code and a full description is available here:
https://github.com/patgolez10/assignhooks
The package can be installed as: pip3 install assignhooks
Example <testmod.py>:
class SampleClass():
name = None
def __assignpre__(self, lhs_name, rhs_name, rhs):
print('PRE: assigning %s = %s' % (lhs_name, rhs_name))
# modify rhs if needed before assignment
if rhs.name is None:
rhs.name = lhs_name
return rhs
def __assignpost__(self, lhs_name, rhs_name):
print('POST: lhs', self)
print('POST: assigning %s = %s' % (lhs_name, rhs_name))
def myfunc():
b = SampleClass()
c = b
print('b.name', b.name)
to instrument it, e.g. <test.py>
import assignhooks
assignhooks.instrument.start() # instrument from now on
import testmod
assignhooks.instrument.stop() # stop instrumenting
# ... other imports and code bellow ...
testmod.myfunc()
Will produce:
$ python3 ./test.py
POST: lhs <testmod.SampleClass object at 0x1041dcc70>
POST: assigning b = SampleClass
PRE: assigning c = b
POST: lhs <testmod.SampleClass object at 0x1041dcc70>
POST: assigning c = b
b.name b
Beginning Python 3.8, it is possible to hint that a value is read-only using typing.Final. What this means is that nothing changes at runtime, allowing anyone to change the value, but if you're using any linter that can read type-hints then it's going to warn the user if they attempt to assign it.
from typing import Final
x: Final[int] = 3
x = 5 # Cannot assign to final name "x" (mypy)
This makes for way cleaner code, but it puts full trust in the user to respect it at runtime, making no attempt to stop users from changing values.
Another common pattern is to expose functions instead of module constants, like sys.getrecursionlimit and sys.setrecursionlimit.
def get_x() -> int:
return 3
Although users can do module.get_x = my_get_x, there's an obvious attempt on the user's part to break it, which can't be fixed. In this way we can prevent people from "accidentally" changing values in our module with minimal complexity.
A ugly solution is to reassign on destructor. But it's no real overload assignment.
import copy
global a
class MyClass():
def __init__(self):
a = 1000
# ...
def __del__(self):
a = copy.copy(self)
a = MyClass()
a = 1
I would like to write a custom list class in Python 3 like in this question How would I create a custom list class in python?, but unlike that question I would like to implement __get__ and __set__ methods. Although my class is similar to the list, but there are some magic operations hidden behind these methods. And so I would like to work with this variable like with list, like in main of my program (see below). I would like to know, how to move __get__ and __set__ methods (fget and fset respectively) from Foo class to MyList class to have only one class.
My current solution (also, I added output for each operation for clarity):
class MyList:
def __init__(self, data=[]):
print('MyList.__init__')
self._mylist = data
def __getitem__(self, key):
print('MyList.__getitem__')
return self._mylist[key]
def __setitem__(self, key, item):
print('MyList.__setitem__')
self._mylist[key] = item
def __str__(self):
print('MyList.__str__')
return str(self._mylist)
class Foo:
def __init__(self, mylist=[]):
self._mylist = MyList(mylist)
def fget(self):
print('Foo.fget')
return self._mylist
def fset(self, data):
print('Foo.fset')
self._mylist = MyList(data)
mylist = property(fget, fset, None, 'MyList property')
if __name__ == '__main__':
foo = Foo([1, 2, 3])
# >>> MyList.__init__
print(foo.mylist)
# >>> Foo.fget
# >>> MyList.__str__
# >>> [1, 2, 3]
foo.mylist = [1, 2, 3, 4]
# >>> Foo.fset
# >>> MyList.__init__
print(foo.mylist)
# >>> Foo.fget
# >>> MyList.__str__
# >>> [1, 2, 3, 4]
foo.mylist[0] = 0
# >>> Foo.fget
# >>> MyList.__setitem__
print(foo.mylist[0])
# >>> Foo.fget
# >>> MyList.__getitem__
# >>> 0
Thank you in advance for any help.
How to move __get__ and __set__ methods (fget and fset respectively) from Foo class to MyList class to have only one class?
UPD:
Thanks a lot to #Blckknght! I tried to understand his answer and it works very well for me! It's exactly what I needed. As a result, I get the following code:
class MyList:
def __init__(self, value=None):
self.name = None
if value is None:
self.value = []
else:
self.value = value
def __set_name__(self, owner, name):
self.name = "_" + name
def __get__(self, instance, owner):
return getattr(instance, self.name)
def __set__(self, instance, value):
setattr(instance, self.name, MyList(value))
def __getitem__(self, key):
return self.value[key]
def __setitem__(self, key, value):
self.value[key] = value
def append(self, value):
self.value.append(value)
def __str__(self):
return str(self.value)
class Foo:
my_list = MyList()
def __init__(self):
self.my_list = [1, 2, 3]
print(type(self.my_list)) # <class '__main__.MyList'>
self.my_list = [4, 5, 6, 7, 8]
print(type(self.my_list)) # <class '__main__.MyList'>
self.my_list[0] = 10
print(type(self.my_list)) # <class '__main__.MyList'>
self.my_list.append(7)
print(type(self.my_list)) # <class '__main__.MyList'>
print(self.my_list) # [10, 5, 6, 7, 8, 7]
foo = Foo()
I don't know, that's Pythonic way or not, but it works as I expected.
In a comment, you explained what you actually want:
x = MyList([1])
x = [2]
# and have x be a MyList after that.
That is not possible. In Python, plain assignment to a bare name (e.g., x = ..., in contrast to x.blah = ... or x[0] = ...) is an operation on the name only, not the value, so there is no way for any object to hook into the name-binding process. An assignment like x = [2] works the same way no matter what the value of x is (and indeed works the same way regardless of whether x already has a value or whether this is the first value being assigned to x).
While you can make your MyList class follow the descriptor protocol (which is what the __get__ and __set__ methods are for), you probably don't want to. That's because, to be useful, a descriptor must be placed as an attribute of a class, not as an attribute of an instance. The properties in your Foo class creating separate instances of MyList for each instance. That wouldn't work if the list was defined on the Foo class directly.
That's not to say that custom descriptors can't be useful. The property you're using in your Foo class is a descriptor. If you wanted to, you could write your own MyListAttr descriptor that does the same thing.
class MyListAttr(object):
def __init__(self):
self.name = None
def __set_name__(self, owner, name): # this is used in Pyton 3.6+
self.name = "_" + name
def find_name(self, cls): # this is used on earlier versions that don't support set_name
for name in dir(cls):
if getattr(cls, name) is self:
self.name = "_" + name
return
raise TypeError()
def __get__(self, obj, owner):
if obj is None:
return self
if self.name is None:
self.find_name(owner)
return getattr(obj, self.name)
def __set__(self, obj, value):
if self.name is None:
self.find_name(type(obj))
setattr(obj, self.name, MyList(value))
class Foo(object):
mylist = MyListAttr() # create the descriptor as a class variable
def __init__(self, data=None):
if data is None:
data = []
self.mylist = data # this invokes the __set__ method of the descriptor!
The MyListAttr class is more complicated than it otherwise might be because I try to have the descriptor object find its own name. That's not easy to figure out in older versions of Python. Starting with Python 3.6, it's much easier (because the __set_name__ method will be called on the descriptor when it is assigned as a class variable). A lot of the code in the class could be removed if you only needed to support Python 3.6 and later (you wouldn't need find_name or any of the code that calls it in __get__ and __set__).
It might not seem worth writing a long descriptor class like MyListAttr to do what you were able to do with less code using a property. That's probably correct if you only have one place you want to use the descriptor. But if you may have many classes (or many attributes within a single class) where you want the same special behavior, you will benefit from packing the behavior into a descriptor rather than writing a lot of very similar property getter and setter methods.
You might not have noticed, but I also made a change to the Foo class that is not directly related to the descriptor use. The change is to the default value for data. Using a mutable object like a list as a default argument is usually a very bad idea, as that same object will be shared by all calls to the function without an argument (so all Foo instances not initialized with data would share the same list). It's better to use a sentinel value (like None) and replace the sentinel with what you really want (a new empty list in this case). You probably should fix this issue in your MyList.__init__ method too.
I have code that someone else wrote like this:
class MyClass(object):
def __init__(self, data):
self.data = data
#property
def attribute1(self):
return self.data.another_name1
#property
def attribute2(self):
return self.data.another_name2
and I want to automatically create the corresponding property setters at run time so I don't have to modify the other person's code. The property setters should look like this:
#attribute1.setter
def attribue1(self, val):
self.data.another_name1= val
#attribute2.setter
def attribue2(self, val):
self.data.another_name2= val
How do I dynamically add these setter methods to the class?
You can write a custom Descriptor like this:
from operator import attrgetter
class CustomProperty(object):
def __init__(self, attr):
self.attr = attr
def __get__(self, ins, type):
print 'inside __get__'
if ins is None:
return self
else:
return attrgetter(self.attr)(ins)
def __set__(self, ins, value):
print 'inside __set__'
head, tail = self.attr.rsplit('.', 1)
obj = attrgetter(head)(ins)
setattr(obj, tail, value)
class MyClass(object):
def __init__(self, data):
self.data = data
attribute1 = CustomProperty('data.another_name1')
attribute2 = CustomProperty('data.another_name2')
Demo:
>>> class Foo():
... pass
...
>>> bar = MyClass(Foo())
>>>
>>> bar.attribute1 = 10
inside __set__
>>> bar.attribute2 = 20
inside __set__
>>> bar.attribute1
inside __get__
10
>>> bar.attribute2
inside __get__
20
>>> bar.data.another_name1
10
>>> bar.data.another_name2
20
This is the author of the question. I found out a very jerry-rigged solution, but I don't know another way to do it. (I am using python 3.4 by the way.)
I'll start with the problems I ran into.
First, I thought about overwriting the property entirely, something like this:
Given this class
class A(object):
def __init__(self):
self._value = 42
#property
def value(self):
return self._value
and you can over write the property entirely by doing something like this:
a = A()
A.value = 31 # This just redirects A.value from the #property to the int 31
a.value # Returns 31
The problem is that this is done at the class level and not at the instance level, so if I make a new instance of A then this happens:
a2 = A()
a.value # Returns 31, because the class itself was modified in the previous code block.
I want that to return a2._value because a2 is a totally new instance of A() and therefore shouldn't be influenced by what I did to a.
The solution to this was to overwrite A.value with a new property rather than whatever I wanted to assign the instance _value to. I learned that you can create a new property that instantiates itself from the old property using the special getter, setter, and deleter methods (see here). So I can overwrite A's value property and make a setter for it by doing this:
def make_setter(name):
def value_setter(self, val):
setattr(self, name, val)
return value_setter
my_setter = make_setter('_value')
A.value = A.value.setter(my_setter) # This takes the property defined in the above class and overwrites the setter with my_setter
setattr(A, 'value', getattr(A, 'value').setter(my_setter)) # This does the same thing as the line above I think so you only need one of them
This is all well and good as long as the original class has something extremely simple in the original class's property definition (in this case it was just return self._value). However, as soon as you get more complicated, to something like return self.data._value like I have, things get nasty -- like #BrenBarn said in his comment on my post. I used the inspect.getsourcelines(A.value.fget) function to get the source code line that contains the return value and parsed that. If I failed to parse the string, I raised an exception. The result looks something like this:
def make_setter(name, attrname=None):
def setter(self, val):
try:
split_name = name.split('.')
child_attr = getattr(self, split_name[0])
for i in range(len(split_name)-2):
child_attr = getattr(child_attr, split_name[i+1])
setattr(child_attr, split_name[-1], val)
except:
raise Exception("Failed to set property attribute {0}".format(name))
It seems to work but there are probably bugs.
Now the question is, what to do if the thing failed? That's up to you and sort of off track from this question. Personally, I did a bit of nasty stuff that involves creating a new class that inherits from A (let's call this class B). Then if the setter worked for A, it will work for the instance of B because A is a base class. However, if it didn't work (because the return value defined in A was something nasty), I ran a settattr(B, name, val) on the class B. This would normally change all other instances that were created from B (like in the 2nd code block in this post) but I dynamically create B using type('B', (A,), {}) and only use it once ever, so changing the class itself has no affect on anything else.
There is a lot of black-magic type stuff going on here I think, but it's pretty cool and quite versatile in the day or so I've been using it. None of this is copy-pastable code, but if you understand it then you can write your modifications.
I really hope/wish there is a better way, but I do not know of one. Maybe metaclasses or descriptors created from classes can do some nice magic for you, but I do not know enough about them yet to be sure.
Comments appreciated!
I try to make something like that :
class oObject(object):
def __init__(self, x = 0, y = 0, z = 0):
self.x = x
self.y = y
self.z = z
def asString (self, value):
return str(value)
vector = oObject(5,5,5)
# So i can do
asString(vector.x)
# But I want this kind of syntax
vector.x.asString()
It's just an example, i don't really want to convert integrer into a string. It's more about class into a class.
You could either write a custom method for your oObject class that returns the string of the given key, or maybe you could write a custom Variant class and wrap your values:
class oObject(object):
def __init__(self, x = 0, y = 0, z = 0):
self.x = Variant(x)
self.y = Variant(y)
self.z = Variant(z)
class Variant(object):
def __init__(self, obj):
self._obj = obj
def __repr__(self):
return '<%s: %s>' % (self.__class__.__name__, self.asString())
def __str__(self):
return self.asString()
def asString(self):
return str(self._obj)
def value(self):
return self._obj
Check out this reference as to how PyQt4 does it, with the QVariant class, which is actually from Qt. Normally python wouldn't need this type, but it was necessary for C++ to represent the multiple types.
You cannot shouldn't do this kind of things in Python.
What you can however do is implementing the standard __str__ method in the class and that is the code that will be used when converting an instance to a string using str(instance).
Technically you can play a lot of tricks in python, trying to bend the syntax to whatever you are used to, but this is a bad idea because a lot of efforts have been put on making Python more readable and you are basically destroying that work.
In Python conversion to string is done by str(x), not by calling a method named asString. Using __str__ you can already customize what str is going to return, why adding a method? If you need a way to do a custom string conversion then just define a function dispatching on the object type instead of trying to inject new methods on existing classes:
converters = dict()
def add_converter(klass, f):
converters[klass] = f
def default_converter(x):
return "<%s: %s>" % (x.__class__.__name__, str(x))
def mystr(x):
return converters.get(x.__class__, default_converter)(x)
With this approach there is no "magic" (i.e. surprising) behavior and you are not wrapping things (another approach that may surprise who reads the code).
In the above example I'm not handling converter inheritance, but you can do that by using a more sophisticated lookup if you need and if you really want that (not sure it makes sense to inherit a conversion to string function, it would silently lose information).
Also if you don't understand what a metaclass is for just leave that concept alone, most probably you don't really need it. Metaclasses are a powerful but somewhat complex tool that is not needed really that often...
I think this article is a good general explanation of what metaclasses are and what you can do with them. Note that some gory details are missing and you should use official documentation to dig them.
To have exactly what you are asking for is tricky in Python -
that is because, when you do
"instance.x.method" - Python first retrieves the attribute "x" from "instance", and them
it would try to find "method" as an attribute in the "x" object itself (without any reference to the "instance" which originally had a reference to "x" that could be possibly retrieved from inside the "method" - but for frame introspection).
I said that it "could be done" - and be made to work for most types of x, but could eventually fail, or have collatteral effects, deppending on the type of the attribute "x":
If you write a __setattr__ method for your class that for each attribute set on the instance, it actually creates a dynamic sub-class of that attribute - which would enable the desired methods on the new object. The draw back, is that not all types of objects can be sub-classed, and not all sub-classed objects will behave exactly like their parents. (If "x" is a function, for example). But it would work for most cases:
class Base(object):
def __setattr__(self, name, attr):
type_ = type(attr)
new_dict = {}
for meth_name in dir(self.__class__):
function = getattr(self.__class__, meth_name)
# Assume any methods on the class have the desired behavior and would
# accept the attribute as it's second parameter (the first being self).
# This could be made more robust by making a simple method-decorator
# which would mark the methods that one wishes to be appliable
# to attributes, instead of picking all non "_" starting methods like here:
if not callable(function) or meth_name in new_dict or meth_name.startswith("_"):
continue
def pinner(f):
def auto_meth(se, *args, **kw):
return f(se._container, se, *args, **kw)
return auto_meth
new_dict[meth_name] = pinner(function)
# This could be improved in order to have a class-based cache of derived types
# so that each attribute setting would only create a new_type for
# each different type that is being set
new_type = type(type_.__name__, (type_,), new_dict)
try:
attr.__class__ = new_type
except TypeError:
# here is the main problem withthis approach:
# if the type being stored can't have it's `__class__`dynamically
# changed, we have to build a new instance of it.
# And if the constructor can't take just the base type
# as its building parameter, it won't work. Worse if having another instance
# does have side-effects in the code, we are subject to those.
attr = new_type(attr)
attr._container = self
super(Base, self).__setattr__(name, attr)
class oObject(Base):
def __init__(self, x = 0, y = 0, z = 0):
self.x = x
self.y = y
self.z = z
def asString(self, attr):
return str(attr)
And after loading these in an interactive section:
>>> v = oObject(1,2,3)
>>> v.x.asString()
'1'
>>> v.w = [1,2,3]
>>> v.w.append(3)
>>> v.w.asString()
'[1, 2, 3, 4]'
>>>
As you can see, this can be done with normal class inheritance no need for metaclasses.
Another, more reliable approach for any Parameter type would be to use another separator for the attribute name, and the method - them you could writhe a much simpler __getattribute__ method on a base class, that would dynamically check for the request method and call it for the attribute. This approach requires no dynamic sub-classing, and is about 2 orders of magnitude simpler. The price is that you'd write something like vector.x__asString instead of the dot separator. This is actually the approach taken in the tried and tested SQLALchemy ORM for Python.
# Second approach:
class Base(object):
separator = "__"
def __getattr__(self, attr_name):
if self.__class__.separator in attr_name:
attr_name, method_name = attr_name.split(self.__class__.separator, 1)
method = getattr(self, method_name)
return method(getattr(self, attr_name))
raise AttributeError
And now:
>>> class oObject(Base):
... def __init__(self, x = 0, y = 0, z = 0):
... self.x = x
... self.y = y
... self.z = z
...
... def asString(self, attr):
... return str(attr)
...
>>>
>>>
>>> v = oObject(1,2,3)
>>> v.x__asString
'1'
(Some more code is required if you want more parameters to be passed to the called method, but I think this is enough to get the idea).