I don't understand this python __del__ behaviour - python

Can someone explain why the following code behaves the way it does:
import types
class Dummy():
def __init__(self, name):
self.name = name
def __del__(self):
print "delete",self.name
d1 = Dummy("d1")
del d1
d1 = None
print "after d1"
d2 = Dummy("d2")
def func(self):
print "func called"
d2.func = types.MethodType(func, d2)
d2.func()
del d2
d2 = None
print "after d2"
d3 = Dummy("d3")
def func(self):
print "func called"
d3.func = types.MethodType(func, d3)
d3.func()
d3.func = None
del d3
d3 = None
print "after d3"
The output (note that the destructor for d2 is never called) is this (python 2.7)
delete d1
after d1
func called
after d2
func called
delete d3
after d3
Is there a way to "fix" the code so the destructor is called without deleting the method added? I mean, the best place to put the d2.func = None would be in the destructor!
Thanks
[edit] Based on the first few answers, I'd like to clarify that I'm not asking about the merits (or lack thereof) of using __del__. I tried to create the shortest function that would demonstrate what I consider to be non-intuitive behavior. I'm assuming a circular reference has been created, but I'm not sure why. If possible, I'd like to know how to avoid the circular reference....

You cannot assume that __del__ will ever be called - it is not a place to hope that resources are automagically deallocated. If you want to make sure that a (non-memory) resource is released, you should make a release() or similar method and then call that explicitly (or use it in a context manager as pointed out by Thanatos in comments below).
At the very least you should read the __del__ documentation very closely, and then you should probably not try to use __del__. (Also refer to the gc.garbage documentation for other bad things about __del__)

I'm providing my own answer because, while I appreciate the advice to avoid __del__, my question was how to get it to work properly for the code sample provided.
Short version: The following code uses weakref to avoid the circular reference. I thought I'd tried this before posting the question, but I guess I must have done something wrong.
import types, weakref
class Dummy():
def __init__(self, name):
self.name = name
def __del__(self):
print "delete",self.name
d2 = Dummy("d2")
def func(self):
print "func called"
d2.func = types.MethodType(func, weakref.ref(d2)) #This works
#d2.func = func.__get__(weakref.ref(d2), Dummy) #This works too
d2.func()
del d2
d2 = None
print "after d2"
Longer version:
When I posted the question, I did search for similar questions. I know you can use with instead, and that the prevailing sentiment is that __del__ is BAD.
Using with makes sense, but only in certain situations. Opening a file, reading it, and closing it is a good example where with is a perfectly good solution. You've gone a specific block of code where the object is needed, and you want to clean up the object and the end of the block.
A database connection seems to be used often as an example that doesn't work well using with, since you usually need to leave the section of code that creates the connection and have the connection closed in a more event-driven (rather than sequential) timeframe.
If with is not the right solution, I see two alternatives:
You make sure __del__ works (see this blog for a better
description of weakref usage)
You use the atexit module to run a callback when your program closes. See this topic for example.
While I tried to provide simplified code, my real problem is more event-driven, so with is not an appropriate solution (with is fine for the simplified code). I also wanted to avoid atexit, as my program can be long-running, and I want to be able to perform the cleanup as soon as possible.
So, in this specific case, I find it to be the best solution to use weakref and prevent circular references that would prevent __del__ from working.
This may be an exception to the rule, but there are use-cases where using weakref and __del__ is the right implementation, IMHO.

Instead of del, you can use the with operator.
http://effbot.org/zone/python-with-statement.htm
just like with filetype objects, you could something like
with Dummy('d1') as d:
#stuff
#d's __exit__ method is guaranteed to have been called

del doesn't call __del__
del in the way you are using removes a local variable. __del__ is called when the object is destroyed. Python as a language makes no guarantees as to when it will destroy an object.
CPython as the most common implementation of Python, uses reference counting. As a result del will often work as you expect. However it will not work in the case that you have a reference cycle.
d3 -> d3.func -> d3
Python doesn't detect this and so won't clean it up right away. And its not just reference cycles. If an exception is throw you probably want to still call your destructor. However, Python will typically hold onto to the local variables as part of its traceback.
The solution is not to depend on the __del__ method. Rather, use a context manager.
class Dummy:
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
print "Destroying", self
with Dummy() as dummy:
# Do whatever you want with dummy in here
# __exit__ will be called before you get here
This is guaranteed to work, and you can even check the parameters to see whether you are handling an exception and do something different in that case.

A full example of a context manager.
class Dummy(object):
def __init__(self, name):
self.name = name
def __enter__(self):
return self
def __exit__(self, exct_type, exce_value, traceback):
print 'cleanup:', d
def __repr__(self):
return 'Dummy(%r)' % (self.name,)
with Dummy("foo") as d:
print 'using:', d
print 'later:', d

It seems to me the real heart of the matter is here:
adding the functions is dynamic (at runtime) and not known in advance
I sense that what you are really after is a flexible way to bind different functionality to an object representing program state, also known as polymorphism. Python does that quite well, not by attaching/detaching methods, but by instantiating different classes. I suggest you look again at your class organization. Perhaps you need to separate a core, persistent data object from transient state objects. Use the has-a paradigm rather than is-a: each time state changes, you either wrap the core data in a state object, or you assign the new state object to an attribute of the core.
If you're sure you can't use that kind of pythonic OOP, you could still work around your problem another way by defining all your functions in the class to begin with and subsequently binding them to additional instance attributes (unless you're compiling these functions on the fly from user input):
class LongRunning(object):
def bark_loudly(self):
print("WOOF WOOF")
def bark_softly(self):
print("woof woof")
while True:
d = LongRunning()
d.bark = d.bark_loudly
d.bark()
d.bark = d.bark_softly
d.bark()

An alternative solution to using weakref is to dynamically bind the function to the instance only when it is called by overriding __getattr__ or __getattribute__ on the class to return func.__get__(self, type(self)) instead of just func for functions bound to the instance. This is how functions defined on the class behave. Unfortunately (for some use cases) python doesn't perform the same logic for functions attached to the instance itself, but you can modify it to do this. I've had similar problems with descriptors bound to instances. Performance here probably isn't as good as using weakref, but it is an option that will work transparently for any dynamically assigned function with the use of only python builtins.
If you find yourself doing this often, you might want a custom metaclass that does dynamic binding of instance-level functions.
Another alternative is to add the function directly to the class, which will then properly perform the binding when it's called. For a lot of use cases, this would have some headaches involved: namely, properly namespacing the functions so they don't collide. The instance id could be used for this, though, since the id in cPython isn't guaranteed unique over the life of the program, you'd need to ponder this a bit to make sure it works for your use case... in particular, you probably need to make sure you delete the class function when an object goes out of scope, and thus its id/memory address is available again. __del__ is perfect for this :). Alternatively, you could clear out all methods namespaced to the instance on object creation (in __init__ or __new__).
Another alternative (rather than messing with python magic methods) is to explicitly add a method for calling your dynamically bound functions. This has the downside that your users can't call your function using normal python syntax:
class MyClass(object):
def dynamic_func(self, func_name):
return getattr(self, func_name).__get__(self, type(self))
def call_dynamic_func(self, func_name, *args, **kwargs):
return getattr(self, func_name).__get__(self, type(self))(*args, **kwargs)
"""
Alternate without using descriptor functionality:
def call_dynamic_func(self, func_name, *args, **kwargs):
return getattr(self, func_name)(self, *args, **kwargs)
"""
Just to make this post complete, I'll show your weakref option as well:
import weakref
inst = MyClass()
def func(self):
print 'My func'
# You could also use the types modules, but the descriptor method is cleaner IMO
inst.func = func.__get__(weakref.ref(inst), type(inst))

use eval()
In [1]: int('25.0')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-1-67d52e3d0c17> in <module>
----> 1 int('25.0')
ValueError: invalid literal for int() with base 10: '25.0'
In [2]: int(float('25.0'))
Out[2]: 25
In [3]: eval('25.0')
Out[3]: 25.0

Related

How to check if method exist from within a Python object

Context and intentions: I want to use an object m_o of type My_object as a way of interfacing with another object called s_o of type Stubborn_object. For the sake of easy understanding, they should behave like if My_object inherited from Stubborn_object, in the way that calling an attribute that doesn't exist in My_object should call the attribute in Stubborn_object.
However, the tricky thing is that I wouldn't be asking this question if I could simply inherit My_object from Stubborn_object. It appears that I can't inherit from it, and, for many reasons, I also can't modify the code of the Stubborn_object class, so I have to use it as it is. Please note that trying to inherit isn't the issue of the question here. I know that other solutions exist for my practical problem, but I really want answers to stay on topic for many reasons. I suspect that other users can have different problems than mine and still be unable to inherit a class. Furthermore, not being able to inherit a class is not the only reason that could make someone read this question. In fact, it's quite a general Python object-oriented problem. I also believe the solution of my problem could be useful in other applications, like custom error handling within the object itself when an attribute is not found, or in thread management to lock the instance as soon as an attribute is called.
In addition to the problem of inheritance, let's suppose that I can't use conditions at higher levels to handle these cases, so everything has to be done inside My_object instance or its parents. That means that I can't use hasattr(m_o, attribute_name) to determine if I should call getattr(m_o, attribute_name) or getattr(s_o, attribute_name). This also means that any try/except blocks and other preconditions must be inside the My_object class or its parents. The point of this question is not about detecting exceptions when calling an attribute from outside the My_object instance. A try/catch block normally has to be outside the My_object class, and I previously stated that this can't be allowed.
For the sake of clarity and to provide a complete verifiable example, here is a sample code of the Stubborn_object class. I know that I said I can't inherit from Stubborn_object and the following code includes an inheritable class. Providing an example of an non-inheritable object would only bring confusion and it would'nt be really helpful to the question anyway, so here is a simple example of an inheritable object. The objective of this is to make an easy to understand question, so please just consider that you can't inherit from it:
class Stubborn_object:
def do_something(self):
print("do_something")
def action_to_override():
print("action_to_override")
def action_a(self):
print("action_a")
def action_b(self):
print("action_b")
Objective: Put it simply, I want my class My_object to detect all by itself that a lacking attribute has been called and run some instructions instead of throwing an AttributeError.
Current attempts: Right now, I manually redirect method calls to the Stubborn_object instance like so (it's successful, but not reliable nor scalable because of the use of hardcoding):
class My_object():
def __init__(self, s_o):
self.stubborn_object = s_o
def action_to_override(self):
# Do stuff. This method "overrides" the Stubborn_object.action_to_override method.
print("Here is stuff getting done instead of action_to_override")
def action_a(self):
return self.stubborn_object.action_a()
def action_b(self):
return self.stubborn_object.action_b()
s_o = Stubborn_object()
m_o = My_object(s_o)
m_o.action_to_override() # Executes Stubborn_object.do_something()
m_o.action_a() # Executes Stubborn_object.action_a()
m_o.action_b() # Executes Stubborn_object.action_b()
Executing this code along with the provided Stubborn_object code sample should print:
Here is stuff getting done instead of action_to_override
action_a
action_b
As you can see from methods action_a and action_b, I have to manually call the Stubborn_object methods from whithin the methods in My_object to mimic the attributes of Stubborn_object. This is ineficient, lacks of robustness and will throw an AttributeError exception if we attempt to make an action that wasn't included in the My_object code.
What if I wanted to automatically send method and attribute calls to the Stubborn_object instance without having to rewrite all of its method and attributes in My_object? I believe this can be achieved with detecting if a lacking attribute of My_object instance is called.
Expectations (or sort of): I am open to any solution that allows the My_object class or its parents to determine if the attribute is lacking or not, all within itself. So I believe I am ready to hear extremely original ideas, so go ahead.
On my part, I believe that something that uses parts of this code is the way to go, but it still lacks the "catch any called attribute" part:
class My_object():
def __init__(self, s_o):
# __init__ stays as it was.
self.stubborn_object = s_o
def action_to_override(self):
# This method also stays as it was.
# Do stuff. This method "overrides" the stubborn_object.action_to_override method.
print("Here is stuff getting done instead of action_to_override")
def run_me_when_method_is_not_found(self, method_name, **kwargs):
print("Method " + method_name + " not in 'My_object' class.")
return getattr(self.stubborn_object, method_name)(**kwargs)
So running those lines with the previous code sample
s_o = Stubborn_object()
m_o = My_object(s_o)
m_o.action_to_override() # Executes Stubborn_object.do_something()
m_o.action_a() # Executes Stubborn_object.action_a()
m_o.action_b() # Executes Stubborn_object.action_b()
will print
Here is stuff getting done instead of action_to_override
Method action_a not in 'My_object' class.
action_a
Method action_b not in 'My_object' class.
action_b
Some similar methods will have to be made for getters and setters, however, the idea stays the same. The thing is that this code lacks the ability to detect that an attribute is missing.
Question: How can I run the run_me_when_method_is_not_found when the method is not found in My_object? Especially, how can a My_object instance detect that the method doesn't exists in its class instead of throwing an AttributeError exception?
Thanks a lot.
Seems like overriding __getattribute__ will do exactly what you want: search for attribute in self.stubborn_object if it is missing in self. Put it into My_object class definition:
def __getattribute__(self, attr):
try:
return object.__getattribute__(self, attr)
except AttributeError:
return object.__getattribute__(self.stubborn_object, attr)

Is this accessing private variable? [duplicate]

I'm coming from the Java world and reading Bruce Eckels' Python 3 Patterns, Recipes and Idioms.
While reading about classes, it goes on to say that in Python there is no need to declare instance variables. You just use them in the constructor, and boom, they are there.
So for example:
class Simple:
def __init__(self, s):
print("inside the simple constructor")
self.s = s
def show(self):
print(self.s)
def showMsg(self, msg):
print(msg + ':', self.show())
If that’s true, then any object of class Simple can just change the value of variable s outside of the class.
For example:
if __name__ == "__main__":
x = Simple("constructor argument")
x.s = "test15" # this changes the value
x.show()
x.showMsg("A message")
In Java, we have been taught about public/private/protected variables. Those keywords make sense because at times you want variables in a class to which no one outside the class has access to.
Why is that not required in Python?
It's cultural. In Python, you don't write to other classes' instance or class variables. In Java, nothing prevents you from doing the same if you really want to - after all, you can always edit the source of the class itself to achieve the same effect. Python drops that pretence of security and encourages programmers to be responsible. In practice, this works very nicely.
If you want to emulate private variables for some reason, you can always use the __ prefix from PEP 8. Python mangles the names of variables like __foo so that they're not easily visible to code outside the namespace that contains them (although you can get around it if you're determined enough, just like you can get around Java's protections if you work at it).
By the same convention, the _ prefix means _variable should be used internally in the class (or module) only, even if you're not technically prevented from accessing it from somewhere else. You don't play around with another class's variables that look like __foo or _bar.
Private variables in Python is more or less a hack: the interpreter intentionally renames the variable.
class A:
def __init__(self):
self.__var = 123
def printVar(self):
print self.__var
Now, if you try to access __var outside the class definition, it will fail:
>>> x = A()
>>> x.__var # this will return error: "A has no attribute __var"
>>> x.printVar() # this gives back 123
But you can easily get away with this:
>>> x.__dict__ # this will show everything that is contained in object x
# which in this case is something like {'_A__var' : 123}
>>> x._A__var = 456 # you now know the masked name of private variables
>>> x.printVar() # this gives back 456
You probably know that methods in OOP are invoked like this: x.printVar() => A.printVar(x). If A.printVar() can access some field in x, this field can also be accessed outside A.printVar()... After all, functions are created for reusability, and there isn't any special power given to the statements inside.
As correctly mentioned by many of the comments above, let's not forget the main goal of Access Modifiers: To help users of code understand what is supposed to change and what is supposed not to. When you see a private field you don't mess around with it. So it's mostly syntactic sugar which is easily achieved in Python by the _ and __.
Python does not have any private variables like C++ or Java does. You could access any member variable at any time if wanted, too. However, you don't need private variables in Python, because in Python it is not bad to expose your classes' member variables. If you have the need to encapsulate a member variable, you can do this by using "#property" later on without breaking existing client code.
In Python, the single underscore "_" is used to indicate that a method or variable is not considered as part of the public API of a class and that this part of the API could change between different versions. You can use these methods and variables, but your code could break, if you use a newer version of this class.
The double underscore "__" does not mean a "private variable". You use it to define variables which are "class local" and which can not be easily overridden by subclasses. It mangles the variables name.
For example:
class A(object):
def __init__(self):
self.__foobar = None # Will be automatically mangled to self._A__foobar
class B(A):
def __init__(self):
self.__foobar = 1 # Will be automatically mangled to self._B__foobar
self.__foobar's name is automatically mangled to self._A__foobar in class A. In class B it is mangled to self._B__foobar. So every subclass can define its own variable __foobar without overriding its parents variable(s). But nothing prevents you from accessing variables beginning with double underscores. However, name mangling prevents you from calling this variables /methods incidentally.
I strongly recommend you watch Raymond Hettinger's Python's class development toolkit from PyCon 2013, which gives a good example why and how you should use #property and "__"-instance variables.
If you have exposed public variables and you have the need to encapsulate them, then you can use #property. Therefore you can start with the simplest solution possible. You can leave member variables public unless you have a concrete reason to not do so. Here is an example:
class Distance:
def __init__(self, meter):
self.meter = meter
d = Distance(1.0)
print(d.meter)
# prints 1.0
class Distance:
def __init__(self, meter):
# Customer request: Distances must be stored in millimeters.
# Public available internals must be changed.
# This would break client code in C++.
# This is why you never expose public variables in C++ or Java.
# However, this is Python.
self.millimeter = meter * 1000
# In Python we have #property to the rescue.
#property
def meter(self):
return self.millimeter *0.001
#meter.setter
def meter(self, value):
self.millimeter = value * 1000
d = Distance(1.0)
print(d.meter)
# prints 1.0
There is a variation of private variables in the underscore convention.
In [5]: class Test(object):
...: def __private_method(self):
...: return "Boo"
...: def public_method(self):
...: return self.__private_method()
...:
In [6]: x = Test()
In [7]: x.public_method()
Out[7]: 'Boo'
In [8]: x.__private_method()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-8-fa17ce05d8bc> in <module>()
----> 1 x.__private_method()
AttributeError: 'Test' object has no attribute '__private_method'
There are some subtle differences, but for the sake of programming pattern ideological purity, it's good enough.
There are examples out there of #private decorators that more closely implement the concept, but your mileage may vary. Arguably, one could also write a class definition that uses meta.
As mentioned earlier, you can indicate that a variable or method is private by prefixing it with an underscore. If you don't feel like this is enough, you can always use the property decorator. Here's an example:
class Foo:
def __init__(self, bar):
self._bar = bar
#property
def bar(self):
"""Getter for '_bar'."""
return self._bar
This way, someone or something that references bar is actually referencing the return value of the bar function rather than the variable itself, and therefore it can be accessed but not changed. However, if someone really wanted to, they could simply use _bar and assign a new value to it. There is no surefire way to prevent someone from accessing variables and methods that you wish to hide, as has been said repeatedly. However, using property is the clearest message you can send that a variable is not to be edited. property can also be used for more complex getter/setter/deleter access paths, as explained here: https://docs.python.org/3/library/functions.html#property
Python has limited support for private identifiers, through a feature that automatically prepends the class name to any identifiers starting with two underscores. This is transparent to the programmer, for the most part, but the net effect is that any variables named this way can be used as private variables.
See here for more on that.
In general, Python's implementation of object orientation is a bit primitive compared to other languages. But I enjoy this, actually. It's a very conceptually simple implementation and fits well with the dynamic style of the language.
The only time I ever use private variables is when I need to do other things when writing to or reading from the variable and as such I need to force the use of a setter and/or getter.
Again this goes to culture, as already stated. I've been working on projects where reading and writing other classes variables was free-for-all. When one implementation became deprecated it took a lot longer to identify all code paths that used that function. When use of setters and getters was forced, a debug statement could easily be written to identify that the deprecated method had been called and the code path that calls it.
When you are on a project where anyone can write an extension, notifying users about deprecated methods that are to disappear in a few releases hence is vital to keep module breakage at a minimum upon upgrades.
So my answer is; if you and your colleagues maintain a simple code set then protecting class variables is not always necessary. If you are writing an extensible system then it becomes imperative when changes to the core is made that needs to be caught by all extensions using the code.
"In java, we have been taught about public/private/protected variables"
"Why is that not required in python?"
For the same reason, it's not required in Java.
You're free to use -- or not use private and protected.
As a Python and Java programmer, I've found that private and protected are very, very important design concepts. But as a practical matter, in tens of thousands of lines of Java and Python, I've never actually used private or protected.
Why not?
Here's my question "protected from whom?"
Other programmers on my team? They have the source. What does protected mean when they can change it?
Other programmers on other teams? They work for the same company. They can -- with a phone call -- get the source.
Clients? It's work-for-hire programming (generally). The clients (generally) own the code.
So, who -- precisely -- am I protecting it from?
In Python 3, if you just want to "encapsulate" the class attributes, like in Java, you can just do the same thing like this:
class Simple:
def __init__(self, str):
print("inside the simple constructor")
self.__s = str
def show(self):
print(self.__s)
def showMsg(self, msg):
print(msg + ':', self.show())
To instantiate this do:
ss = Simple("lol")
ss.show()
Note that: print(ss.__s) will throw an error.
In practice, Python 3 will obfuscate the global attribute name. It is turning this like a "private" attribute, like in Java. The attribute's name is still global, but in an inaccessible way, like a private attribute in other languages.
But don't be afraid of it. It doesn't matter. It does the job too. ;)
Private and protected concepts are very important. But Python is just a tool for prototyping and rapid development with restricted resources available for development, and that is why some of the protection levels are not so strictly followed in Python. You can use "__" in a class member. It works properly, but it does not look good enough. Each access to such field contains these characters.
Also, you can notice that the Python OOP concept is not perfect. Smalltalk or Ruby are much closer to a pure OOP concept. Even C# or Java are closer.
Python is a very good tool. But it is a simplified OOP language. Syntactically and conceptually simplified. The main goal of Python's existence is to bring to developers the possibility to write easy readable code with a high abstraction level in a very fast manner.
Here's how I handle Python 3 class fields:
class MyClass:
def __init__(self, public_read_variable, private_variable):
self.public_read_variable_ = public_read_variable
self.__private_variable = private_variable
I access the __private_variable with two underscores only inside MyClass methods.
I do read access of the public_read_variable_ with one underscore
outside the class, but never modify the variable:
my_class = MyClass("public", "private")
print(my_class.public_read_variable_) # OK
my_class.public_read_variable_ = 'another value' # NOT OK, don't do that.
So I’m new to Python but I have a background in C# and JavaScript. Python feels like a mix of the two in terms of features. JavaScript also struggles in this area and the way around it here, is to create a closure. This prevents access to data you don’t want to expose by returning a different object.
def print_msg(msg):
# This is the outer enclosing function
def printer():
# This is the nested function
print(msg)
return printer # returns the nested function
# Now let's try calling this function.
# Output: Hello
another = print_msg("Hello")
another()
https://www.programiz.com/python-programming/closure
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Closures#emulating_private_methods_with_closures
About sources (to change the access rights and thus bypass language encapsulation like Java or C++):
You don't always have the sources and even if you do, the sources are managed by a system that only allows certain programmers to access a source (in a professional context). Often, every programmer is responsible for certain classes and therefore knows what he can and cannot do. The source manager also locks the sources being modified and of course, manages the access rights of programmers.
So I trust more in software than in human, by experience. So convention is good, but multiple protections are better, like access management (real private variable) + sources management.
I have been thinking about private class attributes and methods (named members in further reading) since I have started to develop a package that I want to publish. The thought behind it were never to make it impossible to overwrite these members, but to have a warning for those who touch them. I came up with a few solutions that might help. The first solution is used in one of my favorite Python books, Fluent Python.
Upsides of technique 1:
It is unlikely to be overwritten by accident.
It is easily understood and implemented.
Its easier to handle than leading double underscore for instance attributes.
*In the book the hash-symbol was used, but you could use integer converted to strings as well. In Python it is forbidden to use klass.1
class Technique1:
def __init__(self, name, value):
setattr(self, f'private#{name}', value)
setattr(self, f'1{name}', value)
Downsides of technique 1:
Methods are not easily protected with this technique though. It is possible.
Attribute lookups are just possible via getattr
Still no warning to the user
Another solution I came across was to write __setattr__. Pros:
It is easily implemented and understood
It works with methods
Lookup is not affected
The user gets a warning or error
class Demonstration:
def __init__(self):
self.a = 1
def method(self):
return None
def __setattr__(self, name, value):
if not getattr(self, name, None):
super().__setattr__(name, value)
else:
raise ValueError(f'Already reserved name: {name}')
d = Demonstration()
#d.a = 2
d.method = None
Cons:
You can still overwrite the class
To have variables not just constants, you need to map allowed input.
Subclasses can still overwrite methods
To prevent subclasses from overwriting methods you can use __init_subclass__:
class Demonstration:
__protected = ['method']
def method(self):
return None
def __init_subclass__(cls):
protected_methods = Demonstration.__protected
subclass_methods = dir(cls)
for i in protected_methods:
p = getattr(Demonstration,i)
j = getattr(cls, i)
if not p is j:
raise ValueError(f'Protected method "{i}" was touched')
You see, there are ways to protect your class members, but it isn't any guarantee that users don't overwrite them anyway. This should just give you some ideas. In the end, you could also use a meta class, but this might open up new dangers to encounter. The techniques used here are also very simple minded and you should definitely take a look at the documentation, you can find useful feature to this technique and customize them to your need.

Is it a good practice to keep reference in a class variable to the current instance of it?

I have a class that will always have only 1 object at the time. I'm just starting OOP in python and I was wondering what is a better approach: to assign an instance of this class to the variable and operate on that variable or rather have this instance referenced in the class variable instead. Here is an example of what I mean:
Referenced instance:
def Transaction(object):
current_transaction = None
in_progress = False
def __init__(self):
self.__class__.current_transaction = self
self.__class__.in_progress = True
self.name = 'abc'
self.value = 50
def update(self):
do_smth()
Transaction()
if Transaction.in_progress:
Transaction.current_transaction.update()
print Transaction.current_transaction.name
print Transaction.current_transaction.value
instance in a variable
def Transaction(object):
def __init__(self):
self.name = 'abc'
self.value = 50
def update(self):
do_smth()
current_transaction = Transaction()
in_progress = True
if in_progress:
current_transaction.update()
print current_transaction.name
print current_transaction.value
It's possible to see that you've encapsulated too much in the first case just by comparing the overall readability of the code: the second is much cleaner.
A better way to implement the first option is to use class methods: decorate all your method with #classmethod and then call with Transaction.method().
There's no practical difference in code quality for these two options. However, assuming that the the class is final, that is, without derived classes, I would go for a third choice: use the module as a singleton and kill the class. This would be the most compact and most readable choice. You don't need classes to create sigletons.
I think the first version doesn't make much sense, and the second version of your code would be better in almost all situations. It can sometimes be useful to write a Singleton class (where only one instance ever exists) by overriding __new__ to always return the saved instance (after it's been created the first time). But usually you don't need that unless you're wrapping some external resource that really only ever makes sense to exist once.
If your other code needs to share a single instance, there are other ways to do so (e.g. a global variable in some module or a constructor argument for each other object that needs a reference).
Note that if your instances have a very well defined life cycle, with specific events that should happen when they're created and destroyed, and unknown code running and using the object in between, the context manager protocol may be something you should look at, as it lets you use your instances in with statements:
with Transaction() as trans:
trans.whatever() # the Transaction will be notified if anything raises
other_stuff() # an exception that is not caught within the with block
trans.foo() # (so it can do a rollback if it wants to)
foo() # the Transaction will be cleaned up (e.g. committed) when the indented with block ends
Implementing the context manager protocol requires an __enter__ and __exit__ method.

How to detect if a parameter is not used / not essential for a particular use case?

I have some working code (library) that, in some situations, I only need a small subset of its functional.
Thinking of a simpler case, the code (library) is a class that takes a few parameters when initializing.
For my limited use case, many of those parameters are not vital as they are not directly used in the internal calculation (some parameters are only used when I call particular methods of the object), while it is very hard to prepare those parameters properly.
So, I am wondering, if there is any easy way to know what parameters are essential without fully analyzing the library code (which is too complicated). For example, I may pass fake parameters to the api, And it would raise an exception only if they are actually used.
For example, I can pass in some_parameter = None for some_parameter that I guess won't be used. So whenever the library tries to access some_parameter.some_field an exception would be raised thus I can further look into the issue and replace it by the actually parameter. However, it would change the behavior of the library if the code itself accepts None as a parameter.
Are there any established approach to this problem? I don't mind false positive as I can always look into the problem and manually check if the usage of the fake parameters by the library is trivial.
For those suggestions on reading documentation and code, I don't have documentations! And the code is legacy code left by previous developers.
Update
#sapi:
Yes I would like to use the proxy pattern / object: I will further investigate on such topic.
"A virtual proxy is a placeholder for "expensive to create" objects. The real object is only created when a client first requests/accesses the object."
I am assuming all classes in question are new-style. This is always the case if you are using Python 3; in Python 2, they must extend from object. You can check a class with isinstance(MyClass, type). For the remainder of my answer, I will assume Python 3, since it was not specified. If you are using Python 2, make sure to extend from object where no other base class is specified.
If those conditions hold, you can write a descriptor that raises an exception whenever it is accessed:
class ParameterUsed(Exception):
pass
class UsageDescriptor:
def __init__(self, name):
super(UsageDescriptor, self).__init__()
self.name = name
def __get__(self, instance, owner):
raise ParameterUsed(self.name)
def __set__(self, instance, value):
# Ignore sets if the value is None.
if value is not None:
raise ParameterUsed(self.name)
def __delete__(self, instance):
# Ignore deletes.
pass
I will assume we are using this class as an example:
class Example:
def __init__(self, a, b):
self.a = a
self.b = b
def use_a(self):
print(self.a)
def use_b(self):
print(self.b)
If we want to see if a is used anywhere, extend the class and put an instance of our descriptor on the class:
class ExtExample(Example):
a = UsageDescriptor('a')
Now if we were to try to use the class, we can see which methods use a:
>>> example = ExtExample(None, None)
>>> example.use_a()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ParameterUsed: a
>>> example.use_b()
None
Here, we can see that use_a tried to use a (raising an exception because it did), but use_b did not (it completed successfully).
This approach works more generally than sapi’s does: in particular, sapi’s approach will only detect an attribute being accessed on the object. But there are plenty of things you can do that do not access attributes on that object. This approach, rather than detecting attributes being accessed on that object, detects the object itself being accessed.
Depending on what you're looking to achieve, you may be able to pass in a proxy object which throws an exception when accessed.
For example:
class ObjectUsedException(Exception):
pass
class ErrorOnUseProxy(object):
def __getattr__(self, name):
raise ObjectUsedException('Tried to access %s'%name)
Of course, that approach will fail in two pretty common situations:
if the library itself checks if the attribute exists (eg, to provide some default value)
if it's treated as a primitive (float, string etc), though you could modify this approach to take that into account
I belive the simplest and least intrusive way is to turn the parameters into properties:
class Foo(object):
def __init__(self):
pass
#property
def a(self):
print >>sys.stderr, 'Accesing parameter a'
return 1
bar = Foo()
print bar.a == 1
Will print True in stdout, and Accesing parameter a to stderr. You would have to tweak it to allow the class to change it.

Read only python attribute? Can't print object

I have an instance of a cx_Oracle.Connection called x and I'm trying to print x.clientinfo or x.module and getting:
attribute 'module' of 'cx_Oracle.Connection' objects is not readable
(What's weird is that I can do print x.username)
I can still do dir(x) with success and I don't have time to look at the source code of cx_Oracle (lots of it implemented in C) so I'm wondering how the implementer was able to do this? Was it by rolling descriptors? or something related to __getitem__? What would be the motivation for this?
You can do this pretty easily in Python with a custom descriptor.
Look at the Descriptor Example in the HOWTO. If you just change the __get__ method to raise an AttributeError… that's it. We might as well rename it and strip out the logging stuff to make it simpler.
class WriteOnly(object):
"""A data descriptor that can't be read.
"""
def __init__(self, initval=None, name='var'):
self.val = initval
self.name = name
def __get__(self, obj, objtype):
raise AttributeError("No peeking at attribute '{}'!".format(self.name))
def __set__(self, obj, val):
self.val = val
class MyClass(object):
x = WriteOnly(0, 'x')
m = MyClass()
m.x = 20 # works
print(m.x) # raises AttributeError
Note that in 2.x, if you forget the (object) and create a classic class, descriptors won't work. (I believe descriptors themselves can actually be classic classes… but don't do that.) In 3.x, there are no classic classes, so that's not a problem.
So, if the value is write-only, how would you ever read it?
Well, this toy example is useless. But you could, e.g., set some private attribute on obj rather than on yourself, at which point code that knows where the data are stored can find it, but casual introspection can't.
But you don't even need descriptors. If you want an attribute that's write-only no matter what class you attach it to, that's one thing, but if you just want to block read access to certain members of a particular class, there's an easier way:
class MyClass(object):
def __getattribute__(self, name):
if name in ('x', 'y', 'z'):
raise AttributeError("No! Bad user! You cannot see my '{}'!".format(name))
return super().__getattribute__(self, name)
m = MyClass()
m.x = 20
m.x # same exception
For more details, see the __getattr__ and __getattribute__ documentation from the data model chapter in the docs.
In 2.x, if you leave the (object) off and create a classic class, the rules for attribute lookup are completely different, and not completely documented, and you really don't want to learn them unless you're planning to spend a lot of time in the 90s, so… don't do that. Also, 2.x will obviously need the 2.x-style explicit super call instead of the 3.x-style magic super().
From the C API side, you've got most of the same hooks, but they're a bit different. See PyTypeObjects for details, but basically:
tp_getset lets you automatically build descriptors out of getter and setter functions, which is similar to #property but not identical.
tp_descr_get and tp_descr_set are for building descriptors separately.
tp_getattro and tp_setattro are similar to __getattr__ and __setattr__, except that the rules for when they get called are a little different, and you typically call PyObject_GenericGetAttr instead of delegating to super() when you know you have no base classes that need to hook attribute access.
Still, why would you do that?
Personally, I've done stuff like this to learn more about the Python data model and descriptors, but that's hardly a reason to put it in a published library.
I'm guessing that more often than not, someone does it because they're trying to force a mistaken notion of OO encapsulation (based on the traditional C++ model) on Python—or, worse, trying to build Java-style security-by-encapsulation (which doesn't work without a secure class loader and all that comes with it).
But there could be cases where there's some generic code that uses these objects via introspection, and "tricking" that code could be useful in a way that trying to trick human users isn't. For example, imagine a serialization library that tried to pickle or JSON-ify or whatever all of the attributes. You could easily write it ignore non-readable attributes. (Of course you could just as easily make it, say, ignore attributes prefixed with a _…)
As for why cx_Oracle did it… I've never even looked at it, so I have no idea.

Categories

Resources