Develop Reference

Equivalent of `const` reference to private class members in Python

I understand that in Python there is no real notion of private/protected/public class members. Yet, using underscores this can be achieved to some extents. For example, consider an object that is supposed to count the number of even integers on a stream. The following would be a relatively safe implementation
class EvensCounter:
def __init__(self):
self._count = 0
def __call__(self, n):
if n % 2 == 0:
self._count += 1
#property
def count(self):
return self._count
This way, the user cannot accidentally change count for example in the following way:
counter = EvensCounter()
counter.count = 5 # Gives AttributeError: can't set attribute
I used to see this as an equivalent of defining count as a private member variable in C++ and then having only a getter that returns a const reference to the private member variable.
However, I realize that the above is not exactly correct. Suppose instead of a simple int, count was of a more sophisticated type of MyInt defined as follows:
class MyInt:
def __init__(self):
self._value = 0
def inc(self, n=1):
self._value += n
#property
def value(self):
return self._value
class EvensCounter:
def __init__(self):
self._count = MyInt()
def __call__(self, n):
if n % 2 == 0:
self._count.inc()
#property
def count(self):
return self._count
In this case, the following piece of code would work and effectively modify the count value:
counter = EvensCounter()
counter.count.inc(5)
While the user cannot change _count variable to reference another object, it can still call methods of that object that change its state. (Whereas in C++, because the inc() method is not a const method, it cannot be called on a const reference.)
I was wondering if there is a mechanism in Python that provides same safety guarantees as the const references of C++? Namely, I need a way to expose a reference to a (private) member of the class that only allows the user to 'read' that member variable, but does not permit him to change the state of that member variable.

No, there's no such thing. C++ is the only major language that supports it (by "major", i mean "top 10 in market popularity", don't want to start a flamewar). Python obviously has no const concept, but even in language that do, const prevents you from re-assigning the variable, but not changing the object it refers to.
So, the main distinction in most languages is whether the object returned to is mutable or immutable as your two examples already illustrate. There's nothing you can do but hope that users respect your API.

In general, no. Python's philosophy for preventing users from shooting themselves in the foot is "Hey, try not to shoot your foot please." With some effort, you can add in some extra safeties, but you won't be able to stop users who point at their foot and pull the trigger.
Does your member need to be a reference? Returning a deep copy will at least prevent the private member from being unintentionally modified.
You could also handle the const-ness yourself. For example:
class MyInt:
def __init__(self):
self._value = 0
self._is_locked = True
def inc(self, n=1):
if not self._is_locked:
self._value += n
else:
raise RuntimeError('Stop trying to modify the value outside of EvensCounter, dummy!')
#property
def value(self):
return self._value
class EvensCounter:
def __init__(self):
self._count = MyInt()
def __call__(self, n):
if n % 2 == 0:
self._count._is_locked = False # If I were *actually* writing code like this, I would make this a context manager
self._count.inc()
self._count._is_locked = True
#property
def count(self):
return self._count
You could also maybe return a proxy that uses something like frozendict under the hood, but I think that is susceptible to the same issue that you mentioned where nested members can be modified.

Python object returning integer/float on object reference

I have a class like so:
class NumberGenerator(object):
def __init__(self, seed=0):
random.seed(seed)
self.current = random.random()
def __next__(self):
self.next()
def next(self):
self.current = random.random()
return self.current
Ideally, I would like to do the following via some magic method if available.
>>> numGen = NumberGenerator(99)
>>> numGen.next()
0.15
>>> numGen + 2
2.15
I know __str__ and __repr__ and __add__ etc. But what I want is a magic method to return something whenever the object is referenced other than a pointer to the object. I will not assign another variable to the object reference, downstream, I promise. Is this available in Python?

The Python language doesn't include pointers. You can think of variables in Python as references to some object. When using a variable, you're accessing the object the variable is referring to.
The behavior you want your NumberGenerator class to have, can be implemented by implementing the __add__ magic method. This is the method Python calls when the + operator is being used with an instance of your class on the left side of the expression. In other words:
numGen + 2
Is translated to:
numGen.__add__(2)
Note that if you want to use the + operator regardless of which side of the expression your class instance is on, you need to also implement the __radd__ magic method:
class NumberGenerator(object):
def __init__(self, seed=0):
random.seed(seed)
self.current = random.random()
def __next__(self):
self.next()
def next(self):
self.current = random.random()
return self.current
def __add__(self, value):
return self.current + value
def __radd__(self, value):
return self.__add__(value)
Here is an example of the class being used:
>>> numGen = NumberGenerator(99)
>>> numGen.next()
0.20007544457494542
>>> numGen + 2
2.2000754445749453
>>> 2 + numGen
2.2000754445749453
>>>

How to use python collections for custom classes

Still somewhat perplexed by python and it's magic functional programming, so I tend to find myself writing code that is more towards the Java paradigm of programming as opposed to Idiomatic Python.
My question is somewhat related to: How do I make a custom class a collection in Python
The only difference is I have nested objects (using composition). The VirtualPage object is comprised of a list of PhysicalPage objects. I have a function which can take a list of PhyscialPage objects and coalesce all of the details into a single named tuple I call PageBoundary. Essentially it's a serialization function which can spit out a tuple comprised of an integer range which represents the physical page and the line number in the page. From this I can easily sort and order VirtualPages among one another (that's the idea at least):
PageBoundary = collections.namedtuple('PageBoundary', 'begin end')
I also have a function which can take a PageBoundary namedtuple and de-serialize or expand the tuple into a list of PhysicalPages. It's preferable that these two data storage classes not change as it will break any downstream code.
Here is a snippet of my custom python2.7 class. It is composed of lot things one is list which contains a the object PhysicalPage:
class VirtualPage(object):
def __init__(self, _physical_pages=list()):
self.physcial_pages = _physcial_pages
class PhysicalPage(object):
# class variables: number of digits each attribute gets
_PAGE_PAD, _LINE_PAD = 10, 12
def __init__(self, _page_num=-1):
self.page_num = _page_num
self.begin_line_num = -1
self.end_line_num = -1
def get_cannonical_begin(self):
return int(''.join([str(self.page_num).zfill(PhysicalPage._PAGE_PAD),
str(tmp_line_num).zfill(PhysicalPage._LINE_PAD) ]))
def get_cannonical_end(self):
pass # see get_cannonical_begin() implementation
def get_canonical_page_boundaries(self):
return PageBoundary(self.get_canonical_begin(), self.get_canonical_end())
I would like to leverage some templated collection (from the python collections module) to easily sort and compare as list or set of VirtualPage classes. Also would like some advice on the layout of my data storage classes: VirtualPage and PhysicalPage.
Given either a sequence of VirtualPages or as in the example below:
vp_1 = VirtualPage(list_of_physical_pages)
vp_1_copy = VirtualPage(list_of_physical_pages)
vp_2 = VirtualPage(list_of_other_physical_pages)
I want to easily answer questions like this:
>>> vp_2 in vp_1
False
>>> vp_2 < vp_1
True
>>> vp_1 == vp_1_copy
True
Right off the bat it seems obvious that the VirtualPage class needs to call get_cannonical_page_boundaries or even implement the function itself. At a minimum it should loop over it's PhysicalPage list to implement the required functions (lt() and eq()) so I can compare b/w VirtualPages.
1.) Currently I'm struggling with implementing some of the comparison functions. One big obstacle is how to compare a tuple? Do I create my own lt() function by creating a custom class which extends some type of collection:
import collections as col
import functools
#total_ordering
class AbstractVirtualPageContainer(col.MutableSet):
def __lt__(self, other):
'''What type would other be?
Make comparison by first normalizing to a comparable type: PageBoundary
'''
pass
2.) Should the comparison function implementation exist in the VirtualPage class instead?
I was leaning towards some type of Set data structure as the properties of the data I'm modeling has the concept of uniqueness: i.e. physical page values cannot overlap and to some extend act as a linked list. Also would setter or getter functions, implemented via # decorator functions be of any use here?

I think you want something like the code below. Not tested; certainly not tested for your application or with your data, YMMV, etc.
from collections import namedtuple
# PageBoundary is a subclass of named tuple with special relational
# operators. __le__ and __ge__ are left undefined because they don't
# make sense for this class.
class PageBoundary(namedtuple('PageBoundary', 'begin end')):
# to prevent making an instance dict (See namedtuple docs)
__slots__ = ()
def __lt__(self, other):
return self.end < other.begin
def __eq__(self, other):
# you can put in an assertion if you are concerned the
# method might be called with the wrong type object
assert isinstance(other, PageBoundary), "Wrong type for other"
return self.begin == other.begin and self.end == other.end
def __ne__(self, other):
return not self == other
def __gt__(self, other):
return other < self
class PhysicalPage(object):
# class variables: number of digits each attribute gets
_PAGE_PAD, _LINE_PAD = 10, 12
def __init__(self, page_num):
self.page_num = page_num
# single leading underscore is 'private' by convention
# not enforced by the language
self._begin = self.page_num * 10**PhysicalPage._LINE_PAD + tmp_line_num
#self._end = ...however you calculate this... ^ not defined yet
self.begin_line_num = -1
self.end_line_num = -1
# this serves the purpose of a `getter`, but looks just like
# a normal class member access. used like x = page.begin
#property
def begin(self):
return self._begin
#property
def end(self):
return self._end
def __lt__(self, other):
assert(isinstance(other, PhysicalPage))
return self._end < other._begin
def __eq__(self, other):
assert(isinstance(other, PhysicalPage))
return self._begin, self._end == other._begin, other._end
def __ne__(self, other):
return not self == other
def __gt__(self, other):
return other < self
class VirtualPage(object):
def __init__(self, physical_pages=None):
self.physcial_pages = sorted(physcial_pages) if physical_pages else []
def __lt__(self, other):
if self.physical_pages and other.physical_pages:
return self.physical_pages[-1].end < other.physical_pages[0].begin
else:
raise ValueError
def __eq__(self, other):
if self.physical_pages and other.physical_pages:
return self.physical_pages == other.physical_pages
else:
raise ValueError
def __gt__(self, other):
return other < self
And a few observations:
Although there is no such thing as "private" members in Python classes, it is a convention to begin a variable name with a single underscore, _, to indicate it is not part of the public interface of the class / module/ etc. So, naming method parameters of public methods with an '_', doesn't seem correct, e.g., def __init__(self, _page_num=-1).
Python generally doesn't use setters / getters; just use the attributes directly. If attribute values need to be calculated, or other some other processing is needed use the #property decorator (as shown for PhysicalPage.begin() above).
It's generally not a good idea to initialize a default function argument with a mutable object. def __init__(self, physical_pages=list()) does not initialize physical_pages with a new empty list each time; rather, it uses the same list every time. If the list is modified, at the next function call physical_pages will be initialized with the modified list. See VirtualPages initializer for an alternative.

Python decorator that makes OOP code FP; good or bad idea?

Recently I've been trying to figure out a solution to the 'expression problem' of choosing between implementing my code in OOP or FP (functional programming). The example I used to illustrate my problem was a Vector2D class. I could make a class that contains all the necessary functions for a 2D vector (dot product, magnitude, etc.), or I could make a set of functions that take a 2-tuple representing a vector. Which option do I chose?
To cope with this problem, I thought it might be nice to make a decorator that takes a class's methods and turns them into global functions. This is how I did it:
import types
def function(method):
method._function = True
return method
def make_functions(cls):
for key in cls.__dict__:
method = getattr(cls, key)
if not isinstance(method, types.FunctionType):
continue
if hasattr(method, '_function') and method._function:
globals()[method.__name__] = method
return cls
#make_functions
class Vector2D:
def __init__(self, x, y):
self.x = x
self.y = y
def __repr__(self):
return 'Vector(%g, %g)' % (self.x, self.y)
def __iter__(self):
for component in self.x, self.y:
yield component
def __getitem__(self, key):
return (self.x, self.y)[key]
def __setitem__(self, key, val):
if key == 0:
self.x = val
elif key == 1:
self.y = val
else:
print('not cool man')
def __add__(self, other):
x = self[0] + other[0]
y = self[1] + other[1]
return self.__class__(x, y)
__radd__ = __add__
def __sub__(self, other):
x = self[0] - other[0]
y = self[1] - other[1]
return self.__class__(x, y)
def __rsub__(self, other):
x = other[0] - self[0]
y = other[1] - self[1]
return self.__class__(x, y)
def __mul__(self, other):
x = other * self[0]
y = other * self[1]
return self.__class__(x, y)
__rmul__ = __mul__
#function
def dot_product(self, other):
return self[0]*other[1] + self[1]*other[0]
Now, dot_product is not only a method of the Vector2D class, but it is also a global function that takes in two vectors (or vector-like objects). This satisfies both the functional and object-oriented approaches to implementing an object like this. The only problem I can foresee this approach making is that any class that can be represented as another object like a tuple or a list, must be defined to work in the same ways as the objects which act like it. This is not so bad for a Vector that can also be a tuple, since all we have to do is define the __getitem__ and __iter__ methods, however I can see this getting wildly out of control for classes that have multiple contrasting implementations
Is this a fair solution to the problem? Is it bad practice or technique? Should I solely provide one or the other?

Python has a #staticmethod decorator for using class methods without an instantiation of that class. Simply annotate a class method with the static method wrapper (note the method now does not take a self reference), and you can call it from the class itself.
In your case, for the dot product, simply do:
class Vector2D():
# Magic methods here...
#staticmethod
def dot_product(a, b):
return a[0]*b[1] + a[1]*b[0]
Then, simply call Vector2D.dot_product(my_vector1, my_vector2) to use the function from the Vector2D class itself.
Assigning class methods to global functions sounds like a very dangerous, buggy, complex, and verbose solution. I would avoid it at all costs.

Metaclass and syntax in Python

I try to make something like that :
class oObject(object):
def __init__(self, x = 0, y = 0, z = 0):
self.x = x
self.y = y
self.z = z
def asString (self, value):
return str(value)
vector = oObject(5,5,5)
# So i can do
asString(vector.x)
# But I want this kind of syntax
vector.x.asString()
It's just an example, i don't really want to convert integrer into a string. It's more about class into a class.

You could either write a custom method for your oObject class that returns the string of the given key, or maybe you could write a custom Variant class and wrap your values:
class oObject(object):
def __init__(self, x = 0, y = 0, z = 0):
self.x = Variant(x)
self.y = Variant(y)
self.z = Variant(z)
class Variant(object):
def __init__(self, obj):
self._obj = obj
def __repr__(self):
return '<%s: %s>' % (self.__class__.__name__, self.asString())
def __str__(self):
return self.asString()
def asString(self):
return str(self._obj)
def value(self):
return self._obj
Check out this reference as to how PyQt4 does it, with the QVariant class, which is actually from Qt. Normally python wouldn't need this type, but it was necessary for C++ to represent the multiple types.

You cannot shouldn't do this kind of things in Python.
What you can however do is implementing the standard __str__ method in the class and that is the code that will be used when converting an instance to a string using str(instance).
Technically you can play a lot of tricks in python, trying to bend the syntax to whatever you are used to, but this is a bad idea because a lot of efforts have been put on making Python more readable and you are basically destroying that work.
In Python conversion to string is done by str(x), not by calling a method named asString. Using __str__ you can already customize what str is going to return, why adding a method? If you need a way to do a custom string conversion then just define a function dispatching on the object type instead of trying to inject new methods on existing classes:
converters = dict()
def add_converter(klass, f):
converters[klass] = f
def default_converter(x):
return "<%s: %s>" % (x.__class__.__name__, str(x))
def mystr(x):
return converters.get(x.__class__, default_converter)(x)
With this approach there is no "magic" (i.e. surprising) behavior and you are not wrapping things (another approach that may surprise who reads the code).
In the above example I'm not handling converter inheritance, but you can do that by using a more sophisticated lookup if you need and if you really want that (not sure it makes sense to inherit a conversion to string function, it would silently lose information).
Also if you don't understand what a metaclass is for just leave that concept alone, most probably you don't really need it. Metaclasses are a powerful but somewhat complex tool that is not needed really that often...
I think this article is a good general explanation of what metaclasses are and what you can do with them. Note that some gory details are missing and you should use official documentation to dig them.

To have exactly what you are asking for is tricky in Python -
that is because, when you do
"instance.x.method" - Python first retrieves the attribute "x" from "instance", and them
it would try to find "method" as an attribute in the "x" object itself (without any reference to the "instance" which originally had a reference to "x" that could be possibly retrieved from inside the "method" - but for frame introspection).
I said that it "could be done" - and be made to work for most types of x, but could eventually fail, or have collatteral effects, deppending on the type of the attribute "x":
If you write a __setattr__ method for your class that for each attribute set on the instance, it actually creates a dynamic sub-class of that attribute - which would enable the desired methods on the new object. The draw back, is that not all types of objects can be sub-classed, and not all sub-classed objects will behave exactly like their parents. (If "x" is a function, for example). But it would work for most cases:
class Base(object):
def __setattr__(self, name, attr):
type_ = type(attr)
new_dict = {}
for meth_name in dir(self.__class__):
function = getattr(self.__class__, meth_name)
# Assume any methods on the class have the desired behavior and would
# accept the attribute as it's second parameter (the first being self).
# This could be made more robust by making a simple method-decorator
# which would mark the methods that one wishes to be appliable
# to attributes, instead of picking all non "_" starting methods like here:
if not callable(function) or meth_name in new_dict or meth_name.startswith("_"):
continue
def pinner(f):
def auto_meth(se, *args, **kw):
return f(se._container, se, *args, **kw)
return auto_meth
new_dict[meth_name] = pinner(function)
# This could be improved in order to have a class-based cache of derived types
# so that each attribute setting would only create a new_type for
# each different type that is being set
new_type = type(type_.__name__, (type_,), new_dict)
try:
attr.__class__ = new_type
except TypeError:
# here is the main problem withthis approach:
# if the type being stored can't have it's `__class__`dynamically
# changed, we have to build a new instance of it.
# And if the constructor can't take just the base type
# as its building parameter, it won't work. Worse if having another instance
# does have side-effects in the code, we are subject to those.
attr = new_type(attr)
attr._container = self
super(Base, self).__setattr__(name, attr)
class oObject(Base):
def __init__(self, x = 0, y = 0, z = 0):
self.x = x
self.y = y
self.z = z
def asString(self, attr):
return str(attr)
And after loading these in an interactive section:
>>> v = oObject(1,2,3)
>>> v.x.asString()
'1'
>>> v.w = [1,2,3]
>>> v.w.append(3)
>>> v.w.asString()
'[1, 2, 3, 4]'
>>>
As you can see, this can be done with normal class inheritance no need for metaclasses.
Another, more reliable approach for any Parameter type would be to use another separator for the attribute name, and the method - them you could writhe a much simpler __getattribute__ method on a base class, that would dynamically check for the request method and call it for the attribute. This approach requires no dynamic sub-classing, and is about 2 orders of magnitude simpler. The price is that you'd write something like vector.x__asString instead of the dot separator. This is actually the approach taken in the tried and tested SQLALchemy ORM for Python.
# Second approach:
class Base(object):
separator = "__"
def __getattr__(self, attr_name):
if self.__class__.separator in attr_name:
attr_name, method_name = attr_name.split(self.__class__.separator, 1)
method = getattr(self, method_name)
return method(getattr(self, attr_name))
raise AttributeError
And now:
>>> class oObject(Base):
... def __init__(self, x = 0, y = 0, z = 0):
... self.x = x
... self.y = y
... self.z = z
...
... def asString(self, attr):
... return str(attr)
...
>>>
>>>
>>> v = oObject(1,2,3)
>>> v.x__asString
'1'
(Some more code is required if you want more parameters to be passed to the called method, but I think this is enough to get the idea).