Develop Reference

Switch case like mapping of a dictionary (values = methods)

As I'm fairly new python, I can't decide which of the following two solutions makes more sense, or maybe no sense at all.
Let's say my abstracted object class look like:
class SimpleData(object):
def __init__(self, data):
self.__data = data
def __getData(self):
return self.__data
def __setData(self, data):
self.__data = data
data = property(__getData, __setData)
#classmethod
def create_new(cls, data):
return cls(data)
Objects of this class, that I need frequently (having a 'predifined object payload'), I'd like to simply create by 'assigning' a preset_name to them. Using the preset_name I can create new copies of those specific objects, having that predefined payload, repeatedly.
I could use a dictionary:
class PresetDict(object):
#classmethod
def get_preset(cls, preset_name):
return {
'preset_111': SimpleData.create_new('111'),
'preset_222': SimpleData.create_new('222'),
'preset_333': SimpleData.create_new('333')
}.get(preset_name, None)
or map methods, using getattr:
class PresetMethod(object):
#classmethod
def get_preset(cls, preset_name):
return getattr(cls, preset_name, lambda: None)()
#classmethod
def preset_111(cls):
return SimpleData.create_new('111')
#classmethod
def preset_222(cls):
return SimpleData.create_new('222')
#classmethod
def preset_333(cls):
return SimpleData.create_new('333')
Both solutions do basically the same:
print(PresetDict.get_preset("preset_111").data)
print(PresetDict.get_preset("preset_333").data)
print(PresetDict.get_preset("not present"))
print(PresetMethod.get_preset("preset_111").data)
print(PresetMethod.get_preset("preset_333").data)
print(PresetMethod.get_preset("not present"))
I strongly prefer the dictionary solution, as it is easier to 'read', extend and will be easier to maintain in the future, especially with a big list of presets.
Here's the BUT:
Performace is of importance. Here, I have absolutely no insight, which of those two solutions will perform better, especially if the preset list grows. Especially the dictionary in PresetDict.get_preset looks 'dodgy' to me. Will it create only the SimpleData instance specified via 'preset_name' when called, or will it create all possible instances specified in the dictionary when PresetDict.get_preset is called, then return the instance specified via 'preset_name' and then discard all other instances.
Hope you can enlighten me on this matter. Maybe you know of possible improvements or even a better solution of what I'd like to achieve?
Thx in advance!

you're right, PresetDict.get_preset will create all three objects and then return one. You could just add a class variable to SimpleData that holds the dictionary so it is only created once, and the get_preset can return instances from that
class SimpleData(object):
_presets = {
'preset_111': SimpleData('111'),
'preset_222': SimpleData('222'),
'preset_333': SimpleData('333')
}
#classmethod
def get_preset(cls, preset_name):
return cls._presets.get(preset_name, None)
Note that this isn't really any more efficient, it will just make it easier to create commonly used classes.
Also see functools.lru_cache

How do you keep code consistent with multiple developers?

I know that Python is a dynamically typed language, and that I am likely trying to recreate Java behavior here. However, I have a team of people working on this code base, and my goal with the code is to ensure that they are doing things in a consistent manner. Let me give an example:
class Company:
def __init__(self, j):
self.locations = []
When they instantiate a Company object, an empty list that holds locations is created. Now, with Python anything can be added to the list. However, I would like for this list to only contain Location objects:
class Location:
def __init__(self, j):
self.address = None
self.city = None
self.state = None
self.zip = None
I'm doing this with classes so that the code is self documenting. In other words, "location has only these attributes". My goal is that they do this:
c = Company()
l = Location()
l.city = "New York"
c.locations.append(l)
Unfortunately, nothing is stopping them from simply doing c.locations.append("foo"), and nothing indicates to them that c.locations should be a list of Location objects.
What is the Pythonic way to enforce consistency when working with a team of developers?

An OOP solution is to make sure the users of your class' API do not have to interact directly with your instance attributes.
Methods
One approach is to implement methods which encapsulate the logic of adding a location.
Example
class Company:
def __init__(self, j):
self.locations = []
def add_location(self, location):
if isinstance(location, Location):
self.locations.append(location)
else:
raise TypeError("argument 'location' should be a Location object")
Properties
Another OOP concept you can use is a property. Properties are a simple way to define getter and setters for your instance attributes.
Example
Suppose we want to enforce a certain format for a Location.zip attribute
class Location:
def __init__(self):
self._zip = None
#property
def zip(self):
return self._zip
#zip.setter
def zip(self, value):
if some_condition_on_value:
self._zip = value
else:
raise ValueError('Incorrect format')
#zip.deleter
def zip(self):
self._zip = None
Notice that the attribute Location()._zip is still accessible and writable. While the underscore denotes what should be a private attribute, nothing is really private in Python.
Final word
Due to Python's high introspection capabilities, nothing will ever be totally safe. You will have to sit down with your team and discuss the tools and practice you want to adopt.
Nothing is really private in python. No class or class instance can
keep you away from all what's inside (this makes introspection
possible and powerful). Python trusts you. It says "hey, if you want
to go poking around in dark places, I'm gonna trust that you've got a
good reason and you're not making trouble."
After all, we're all consenting adults here.
--- Karl Fast

You could also define a new class ListOfLocations that make the safety checks. Something like this
class ListOfLocations(list):
def append(self,l):
if not isinstance(l, Location): raise TypeError("Location required here")
else: super().append(l)

python JSON complex objects (accounting for subclassing)

What is the best practice for serializing/deserializing complex python objects into/from JSON, that would account for subclassing and prevent multiple copies of same objects (assuming we know how to distinguish between different instances of same class) to be stored multiple times?
In a nutshell, I'm writing a small scientific library and want people to use it. But after watching Raymond Hettinger talk Python's Class Development Toolkit I've decided that it would be a good exercise for me to implement subclassing-aware behaviour. So far it went fine, but now I hit the JSON serialization task.
Until now I've looked around and found the following about JSON serialization in Python:
python docs about json module
python cookbook about json serialization
dive into python 3 in regards to json
very interesting article from 2009
Two main obstacles that I have are accounting for possible subclassing, single copy per instance.
After multiple different attempts to solve it in pure python, without any changes to the JSON representation of object, I've ended up understanding, that at a time of deserializing JSON, there is now way to know instance of what class heir was serialized before. So some mention about it shall be made, and I've ended up with something like this:
class MyClassJSONEncoder(json.JSONEncoder):
#classmethod
def represent_object(cls, obj):
"""
This is a way to serialize all built-ins as is, and all complex objects as their id, which is hash(obj) in this implementation
"""
if isinstance(obj, (int, float, str, Boolean)) or value is None:
return obj
elif isinstance(obj, (list, dict, tuple)):
return cls.represent_iterable(obj)
else:
return hash(obj)
#classmethod
def represent_iterable(cls, iterable):
"""
JSON supports iterables, so they shall be processed
"""
if isinstance(iterable, (list, tuple)):
return [cls.represent_object(value) for value in iterable]
elif isinstance(iterable, dict):
return [cls.represent_object(key): cls.represent_object(value) for key, value in iterable.items()]
def default(self, obj):
if isinstance(obj, MyClass):
result = {"MyClass_id": hash(obj),
"py__class__": ":".join([obj.__class__.__module, obj.__class__.__qualname__]}
for attr, value in self.__dict__.items():
result[attr] = self.represent_object(value)
return result
return super().default(obj) # accounting for JSONEncoder subclassing
here the accounting for subclassing is done in
"py__class__": ":".join([obj.__class__.__module, obj.__class__.__qualname__]
the JSONDecoder is to be implemented as follows:
class MyClassJSONDecoder(json.JSONDecoder):
def decode(self, data):
if isinstance(data, str):
data = super().decode(data)
if "py__class__" in data:
module_name, class_name = data["py__class__"].split(":")
object_class = getattr(importlib.__import__(module_name, fromlist=[class_name]), class_name)
else:
object_class = MyClass
data = {key, value for key, value in data.items() if not key.endswith("_id") or key != "py__class__"}
return object_class(**data)
As can be seen, here we account for possible subclassing with a "py__class__" attribute in JSON representation of object, and if no such attribute is present (this can be the case, if JSON was generated by another program, say in C++, and they just want to pass us information about the plain MyClass object, and don't really care for inheritance) the default approach to creating an instance of MyClass
is pursued. This is, by the way, the reason why not a single JSONDecoder can be created all objects: it has to have a default class value to create, if no py__class__ is specified.
In terms of a single copy for every instance, this is done by the fact, that object is serialized with a special JSON key myclass_id, and all attribute values are serialized as primitives (lists, tuples, dicts, and built-in are preserved, while when a complex object is a value of some attribute, only its hash is stored). Such approach of storing objects hashes allows one to serialize each object exactly once, and then, knowing the structure of an object to be decoded from json representation, it can look for respective objects and assign them after all. To simply illustrate this the following example can be observed:
class MyClass(object):
json_encoder = MyClassJSONEncoder()
json_decoder = MyClassJSONDecoder()
def __init__(self, attr1):
self.attr1 = attr1
self.attr2 = [complex_object_1, complex_object_2]
def to_json(self, top_level=None):
if top_level is None:
top_level = {}
top_level["my_class"] = self.json_encoder.encode(self)
top_level["complex_objects"] = [obj.to_json(top_level=top_level) for obj in self.attr2]
return top_level
#classmethod
def from_json(cls, data, class_specific_data=None):
if isinstance(data, str):
data = json.loads(data)
if class_specific_data is None:
class_specific_data = data["my_class"] # I know the flat structure of json, and I know the attribute name, this class will be stored
result = cls.json_decoder.decode(class_spcific_data)
# repopulate complex valued attributes with real python objects
# rather than their id aliases
complex_objects = {co_data["ComplexObject_id"]: ComplexObject.from_json(data, class_specific_data=co_data) for co_data in data["complex_objects"]]
result.complex_objects = [c_o for c_o_id, c_o in complex_objects.items() if c_o_id in self.complex_objects]
# finish such repopulation
return result
Is this even a right way to go? Is there a more robust way? Have I missed some programming patter to implement in this very particular situation?
I just really want to understand what is the most correct and pythonic way to implement a JSON serialization that would account for subclassing and also prevent multiple copies of same object to be stored.

Printing all instances of a class

With a class in Python, how do I define a function to print every single instance of the class in a format defined in the function?

I see two options in this case:
Garbage collector
import gc
for obj in gc.get_objects():
if isinstance(obj, some_class):
dome_something(obj)
This has the disadvantage of being very slow when you have a lot of objects, but works with types over which you have no control.
Use a mixin and weakrefs
from collections import defaultdict
import weakref
class KeepRefs(object):
__refs__ = defaultdict(list)
def __init__(self):
self.__refs__[self.__class__].append(weakref.ref(self))
#classmethod
def get_instances(cls):
for inst_ref in cls.__refs__[cls]:
inst = inst_ref()
if inst is not None:
yield inst
class X(KeepRefs):
def __init__(self, name):
super(X, self).__init__()
self.name = name
x = X("x")
y = X("y")
for r in X.get_instances():
print r.name
del y
for r in X.get_instances():
print r.name
In this case, all the references get stored as a weak reference in a list. If you create and delete a lot of instances frequently, you should clean up the list of weakrefs after iteration, otherwise there's going to be a lot of cruft.
Another problem in this case is that you have to make sure to call the base class constructor. You could also override __new__, but only the __new__ method of the first base class is used on instantiation. This also works only on types that are under your control.
Edit: The method for printing all instances according to a specific format is left as an exercise, but it's basically just a variation on the for-loops.

You'll want to create a static list on your class, and add a weakref to each instance so the garbage collector can clean up your instances when they're no longer needed.
import weakref
class A:
instances = []
def __init__(self, name=None):
self.__class__.instances.append(weakref.proxy(self))
self.name = name
a1 = A('a1')
a2 = A('a2')
a3 = A('a3')
a4 = A('a4')
for instance in A.instances:
print(instance.name)

You don't need to import ANYTHING! Just use "self". Here's how you do this
class A:
instances = []
def __init__(self):
self.__class__.instances.append(self)
print('\n'.join(A.instances)) #this line was suggested by #anvelascos
It's this simple. No modules or libraries imported

Very nice and useful code, but it has a big problem: list is always bigger and it is never cleaned-up, to test it just add print(len(cls.__refs__[cls])) at the end of the get_instances method.
Here a fix for the get_instances method:
__refs__ = defaultdict(list)
#classmethod
def get_instances(cls):
refs = []
for ref in cls.__refs__[cls]:
instance = ref()
if instance is not None:
refs.append(ref)
yield instance
# print(len(refs))
cls.__refs__[cls] = refs
or alternatively it could be done using WeakSet:
from weakref import WeakSet
__refs__ = defaultdict(WeakSet)
#classmethod
def get_instances(cls):
return cls.__refs__[cls]

Same as almost all other OO languages, keep all instances of the class in a collection of some kind.
You can try this kind of thing.
class MyClassFactory( object ):
theWholeList= []
def __call__( self, *args, **kw ):
x= MyClass( *args, **kw )
self.theWholeList.append( x )
return x
Now you can do this.
object= MyClassFactory( args, ... )
print MyClassFactory.theWholeList

Python doesn't have an equivalent to Smallktalk's #allInstances as the architecture doesn't have this type of central object table (although modern smalltalks don't really work like that either).
As the other poster says, you have to explicitly manage a collection. His suggestion of a factory method that maintains a registry is a perfectly reasonable way to do it. You may wish to do something with weak references so you don't have to explicitly keep track of object disposal.

It's not clear if you need to print all class instances at once or when they're initialized, nor if you're talking about a class you have control over vs a class in a 3rd party library.
In any case, I would solve this by writing a class factory using Python metaclass support. If you don't have control over the class, manually update the __metaclass__ for the class or module you're tracking.
See http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html for more information.

In my project, I faced a similar problem and found a simple solution that may also work for you in listing and printing your class instances. The solution worked smoothly in Python version 3.7; gave partial errors in Python version 3.5.
I will copy-paste the relevant code blocks from my recent project.
```
instances = []
class WorkCalendar:
def __init__(self, day, patient, worker):
self.day = day
self.patient = patient
self.worker= worker
def __str__(self):
return f'{self.day} : {self.patient} : {self.worker}'
In Python the __str__ method in the end, determines how the object will be interpreted in its string form. I added the : in between the curly brackets, they are completely my preference for a "Pandas DataFrame" kind of reading. If you apply this small __str__ function, you will not be seeing some machine-readable object type descriptions- which makes no sense for human eyes. After adding this __str__ function you can append your objects to your list and print them as you wish.
appointment= WorkCalendar("01.10.2020", "Jane", "John")
instances.append(appointment)
For printing, your format in __str__ will work as default. But it is also possible to call all attributes separately:
for instance in instances:
print(instance)
print(instance.worker)
print(instance.patient)
For detailed reading, you may look at the source: https://dbader.org/blog/python-repr-vs-str

Javascript style dot notation for dictionary keys unpythonic?

I've started to use constructs like these:
class DictObj(object):
def __init__(self):
self.d = {}
def __getattr__(self, m):
return self.d.get(m, None)
def __setattr__(self, m, v):
super.__setattr__(self, m, v)
Update: based on this thread, I've revised the DictObj implementation to:
class dotdict(dict):
def __getattr__(self, attr):
return self.get(attr, None)
__setattr__= dict.__setitem__
__delattr__= dict.__delitem__
class AutoEnum(object):
def __init__(self):
self.counter = 0
self.d = {}
def __getattr__(self, c):
if c not in self.d:
self.d[c] = self.counter
self.counter += 1
return self.d[c]
where DictObj is a dictionary that can be accessed via dot notation:
d = DictObj()
d.something = 'one'
I find it more aesthetically pleasing than d['something']. Note that accessing an undefined key returns None instead of raising an exception, which is also nice.
Update: Smashery makes a good point, which mhawke expands on for an easier solution. I'm wondering if there are any undesirable side effects of using dict instead of defining a new dictionary; if not, I like mhawke's solution a lot.
AutoEnum is an auto-incrementing Enum, used like this:
CMD = AutoEnum()
cmds = {
"peek": CMD.PEEK,
"look": CMD.PEEK,
"help": CMD.HELP,
"poke": CMD.POKE,
"modify": CMD.POKE,
}
Both are working well for me, but I'm feeling unpythonic about them.
Are these in fact bad constructs?

Your DictObj example is actually quite common. Object-style dot-notation access can be a win if you are dealing with ‘things that resemble objects’, ie. they have fixed property names containing only characters valid in Python identifiers. Stuff like database rows or form submissions can be usefully stored in this kind of object, making code a little more readable without the excess of ['item access'].
The implementation is a bit limited - you don't get the nice constructor syntax of dict, len(), comparisons, 'in', iteration or nice reprs. You can of course implement those things yourself, but in the new-style-classes world you can get them for free by simply subclassing dict:
class AttrDict(dict):
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
To get the default-to-None behaviour, simply subclass Python 2.5's collections.defaultdict class instead of dict.

With regards to the DictObj, would the following work for you? A blank class will allow you to arbitrarily add to or replace stuff in a container object.
class Container(object):
pass
>>> myContainer = Container()
>>> myContainer.spam = "in a can"
>>> myContainer.eggs = "in a shell"
If you want to not throw an AttributeError when there is no attribute, what do you think about the following? Personally, I'd prefer to use a dict for clarity, or to use a try/except clause.
class QuietContainer(object):
def __getattr__(self, attribute):
try:
return object.__getattr__(self,attribute)
except AttributeError:
return None
>>> cont = QuietContainer()
>>> print cont.me
None
Right?

This is a simpler version of your DictObj class:
class DictObj(object):
def __getattr__(self, attr):
return self.__dict__.get(attr)
>>> d = DictObj()
>>> d.something = 'one'
>>> print d.something
one
>>> print d.somethingelse
None
>>>

As far as I know, Python classes use dictionaries to store their attributes anyway (that's hidden from the programmer), so it looks to me that what you've done there is effectively emulate a Python class... using a python class.

It's not "wrong" to do this, and it can be nicer if your dictionaries have a strong possibility of turning into objects at some point, but be wary of the reasons for having bracket access in the first place:
Dot access can't use keywords as keys.
Dot access has to use Python-identifier-valid characters in the keys.
Dictionaries can hold any hashable element -- not just strings.
Also keep in mind you can always make your objects access like dictionaries if you decide to switch to objects later on.
For a case like this I would default to the "readability counts" mantra: presumably other Python programmers will be reading your code and they probably won't be expecting dictionary/object hybrids everywhere. If it's a good design decision for a particular situation, use it, but I wouldn't use it without necessity to do so.

The one major disadvantage of using something like your DictObj is you either have to limit allowable keys or you can't have methods on your DictObj such as .keys(), .values(), .items(), etc.

There's a symmetry between this and this answer:
class dotdict(dict):
__getattr__= dict.__getitem__
__setattr__= dict.__setitem__
__delattr__= dict.__delitem__
The same interface, just implemented the other way round...
class container(object):
__getitem__ = object.__getattribute__
__setitem__ = object.__setattr__
__delitem__ = object.__delattr__

Don't overlook Bunch.
It is a child of dictionary and can import YAML or JSON, or convert any existing dictionary to a Bunch and vice-versa. Once "bunchify"'d, a dictionary gains dot notations without losing any other dictionary methods.

I like dot notation a lot better than dictionary fields personally. The reason being that it makes autocompletion work a lot better.

It's not bad if it serves your purpose. "Practicality beats purity".
I saw such approach elserwhere (eg. in Paver), so this can be considered common need (or desire).

Because you ask for undesirable side-effects:
A disadvantage is that in visual editors like eclipse+pyDev, you will see many undefined variable errors on lines using the dot notation. Pydef will not be able to find such runtime "object" definitions. Whereas in the case of a normal dictionary, it knows that you are just getting a dictionary entry.
You would need to 1) ignore those errors and live with red crosses; 2) suppress those warnings on a line by line basis using ##UndefinedVariable or 3) disable undefined variable error entirely, causing you to miss real undefined variable definitions.

If you're looking for an alternative that handles nested dicts:
Recursively transform a dict to instances of the desired class
import json
from collections import namedtuple
class DictTransformer():
#classmethod
def constantize(self, d):
return self.transform(d, klass=namedtuple, klassname='namedtuple')
#classmethod
def transform(self, d, klass, klassname):
return self._from_json(self._to_json(d), klass=klass, klassname=klassname)
#classmethod
def _to_json(self, d, access_method='__dict__'):
return json.dumps(d, default=lambda o: getattr(o, access_method, str(o)))
#classmethod
def _from_json(self, jsonstr, klass, klassname):
return json.loads(jsonstr, object_hook=lambda d: klass(klassname, d.keys())(*d.values()))
Ex:
constants = {
'A': {
'B': {
'C': 'D'
}
}
}
CONSTANTS = DictTransformer.transform(d, klass=namedtuple, klassname='namedtuple')
CONSTANTS.A.B.C == 'D'
Pros:
handles nested dicts
can potentially generate other classes
namedtuples provide immutability for constants
Cons:
may not respond to .keys and .values if those are not provided on your klass (though you can sometimes mimic with ._fields and list(A.B.C))
Thoughts?
h/t to #hlzr for the original class idea