Develop Reference

Undoing a decade of singleton pattern and class-level configuration

Overview
I need to duplicate a whole inheritance tree of classes. Simply deep-copying the class objects does not work; a proper factory pattern involves a huge amount of code changes; I'm not sure how to use metaclasses to accomplish this.
Background
The software I work on implements support for specialized external hardware, connected to the host computer via USB. Many years ago, it was assumed that there would only ever be one type of hardware in use at a time. Consequently, the hardware object is used as a singleton. Along the years, secondary classes were configured based on the currently active hardware class.
At the moment, it is impossible to use this library with two types of hardware at the same time, since the classobjects cannot be configured for both hardware at the same time.
In recent years, we have avoided this issue by creating one python process for each hardware, but this is becoming untenable.
Here is an extremely simplified example of the architecture:
# ----------
# Hardware classes
class HwBase():
def customizeComponent(self, compDict):
compDict['ComponentBase'].hardware = self
class HwA(HwBase):
def customizeComponent(self, compDict):
super().customizeComponent(compDict)
compDict['AnotherComponent'].prop.configure(1,2,3)
class HwB(HwBase):
def customizeComponent(self, compDict):
super().customizeComponent(compDict)
compDict['AnotherComponent'].prop.configure(4,5,6)
# ----------
# Property classes
class SpecialProperty(property):
def __init__(self, fvalidate):
self.fvalidate = fvalidate
# handle fset, fget, etc. here.
# super().__init__()
# ----------
# Component classes
class ComponentBase():
hardware = None
def validateProp(self, val):
return val < self.maxVal
prop = SpecialProperty(fvalidate=validateProp)
class SomeComponent():
"""Users directly instantiate and use this compoent via an interactive shell.
This component does complex operations with the hardware attribute"""
def validateThing(self, val):
return isinstance(val, ComponentBase)
thing = SpecialProperty(fvalidate=validateThing)
class AnotherComponent():
"""Users directly instantiate and use this compoent via an interactive shell
This component does complex operations with the hardware attribute"""
maxVal = 15
# ----------
# Initialization
def initialize():
""" This is only called once perppython instance."""
#activeCls = HwA
activeCls = HwB
allComponents = {
'ComponentBase': ComponentBase,
'SomeComponent': SomeComponent,
'AnotherComponent': AnotherComponent
}
hwInstance = activeCls()
hwInstance.customizeComponent(allComponents)
return allComponents
components = initialize()
# ----------
# User code goes here
someInstance1 = components['SomeComponent']()
someInstance2 = components['SomeComponent']()
someInstance1.prop = 10
someInstance2.prop = 10
The overarching goal would be to interact with both HwA and HwB at the same time. Since most interactions are done via components instead of the Hw objects themselves, I believe the solution involves having multiple versions of the components, e.g.: two separate inheritance trees, for a total of 6 final components, one tree/set configured for each hardware. This is what I need help with.
Potential solutions
Consider that I have around tens different hardware do configure for. Furthermore, there are hundreds of different leaf components classes, with many extra bases and mixin classes.
Move all configuration steps in the component's init method
Not possible due to the use of properties; these need to be set on the class.
Deepcopy the classobjects
Copy all classobjects, swap in the appropriate __bases__. Mutable class variables need to be carefully handled. However, I'm not sure how to deal with properties for this, since classbody references within the property objects (such as fvalidate) need to be updated to that of the copied class.
This requires a significant amount of manual intervention to work. Not impossible, but prone to breaking in the long term.
Factory pattern
Wrap all component definition in a factory function:
def ComponentBaseFactory(hw):
class SomeComponent(cache[hw].ComponentBase):
pass
and have some sort of component cache which would handle creating all classobjects during initialize()
This is what I consider the most architecturally-correct option available. Since the class body is re-executed
on every factory call, the attributes of the properties will reference the appropriate class object.
Downside: huge code footprint. I am familiar with doing codebase-wide changes via sed or python scripts, but this would be quite a lot.
Add metaclasses on components
I am not sure how to proceed for this. Based on the python data model (py3.7), the following happens at class creation (which happens right after the class definition indentation ends):
MRO entries are resolved;
the appropriate metaclass is determined;
the class namespace is prepared;
the class body is executed;
the class object is created.
I would need to redo these steps after the class has been defined (like a factory function!), but i'm not sure how to redo step 4. Specifically, the python documentation states in section 3.3.3.5 that the class body is executed as with a "special?" form of the exec() builtin. How can I re-exec the class body with a different set of locals/globals? Even if I access the class body's code with inspect shenanigans, i'm not sure i'll be able to reproduce the module environment properly.
Even if I mess with __prepare__ and __new__, I don't see how I can fix the cross-references introduced in the class code block regarding the property instantiation.
Components as metaclasses
A metaclass is a class factory, just like a class is an object factory. SomeComponent and AnotherComponent could be declared as metaclasses, then get instantiated with the Hw object during initialize():
SomeComponent = SomeComponentMeta(hw)
This is similar to the factory pattern, but would also require quite a few code changes: a lot of class code would have to be moved to the metaclass __init__.

I'd have to spend a lot more of time here to proper understand what you need, but if your "TL;DR" of executing the class body with different globals/nonlocal variables is the bottom line, the factory approach is a very clean and readable way, as you had considered.
At first, I don't think a metaclass could be a good approach here - although it could be used to customize your special properties (in my first read, I could not figure out what they actually do, and how they should differ between your final classes). If the function as a class factory can specialize your properties, it would work nonetheless.
If what you need is that the properties are independent for Hwa and HwB like in accessing a different list object in HwA than is accessed in HwB, yes, a metaclass could take care of that, by automatically recreating any properties when creating a subclass (so that the property objects themselves are not shared with the supper-classes and across the hierarchy).
If that i what you need, leave a comment, I can write some proof of concept code.
Anyway, it is possible to create a metaclass that, upon instantiating a subclass, will look upon the hierarchy for all SpecialProperty and create new-instances of those for the subclass - so that a base value set on a superclass remains valid for the subclasses, but when configuration runs, each class will have an independent configuration. (as it turns out, no metaclass is needed: we are covered by __init_subclass__ )
Another thing to take care of is that subclassses of property cannot be simply copies with Python's copy.copy (tested empirically), so we need a way to create reliable copies of those. I include one function bellow, but it might need to be improved to work with the actual SpecialProperty class.
from copy import copy
def copy_property(prop):
cls = prop.__class__
new_prop = cls.__new__(cls)
# Initialize the attributes that can't be set from Python code, inplace:
property.__init__(new_prop, prop.fget, prop.fset, prop.fdel)
if hasattr(prop, "__dict__"): # only exists for subclasses of property
# Possible adaptation needed: it may be that for some attributes of
# SpecialProperty, a deepcopy would be needed.
# But for the given example attribute of "fvalidate" a simple copy is better:
new_prop.__dict__ = copy(prop.__dict__)
return new_prop
# Python 3.6 introduced `__init_subclass__` which is called at subclass _creation_
# time. With it, the logic can be inserted in ComponentBase and there is no need for
# a metaclass.
class ComponentBase():
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
for attrname in dir(cls):
attr = getattr(cls, attrname)
if not isinstance(attr, SpecialProperty):
continue
new_prop = copy_property(attr)
setattr(cls, attrname, new_prop)
hardware = None
...
As you see- theres some workarounds that had to be done because your project opted for subclassing property. I am leaving this remark here as a remainder that unless property fits one exact needs, it is more clean to write a new class implementing the Descriptor Protocol - just by implementing __set__, __get__ and __delete__ directly.

Should methods that do not act on object data be made static?

I have a class that includes some auxiliary functions that do not operate on object data. Ordinarily I would leave these methods private, but I am in Python so there is no such thing. In testing, I am finding it slightly goofy to have to instantiate an instance of my class in order to be able to call these methods. Is there a solid theoretical reason to choose to keep these methods non-static or to make them static?

If a method does not need access to the current instance, you may want to make it either a classmethod, a staticmethod or a plain function.
A classmethod will get the current class as first param. This enable it to access the class attributes including other classmethods or staticmethods. This is the right choice if your method needs to call other classmethods or staticmethods.
A staticmethod only gets it's explicit argument - actually it nothing but a function that can be resolved on the class or instance. The main points of staticmethods are specialization - you can override a staticmethod in a child class and have a method (classmethod or instancemethod) of the base class call the overridden version of the staticmethod - and eventually ease of use - you don't need to import the function apart from the class (at this point it's bordering on lazyness but I've had a couple cases with dynamic imports etc where it happened to be handy).
A plain function is, well, just a plain function - no class-based dispatch, no inheritance, no fancy stuff. But if it's only a helper function used internally by a couple classes in the same module and not part of the classes nor module API, it's possibly just what you're looking for.
As a last note: you can have "some kind of" privacy in Python. Mostly, prefixing a name (whether an attribute / method / class or plain function) with a single leading underscore means "this is an implementation detail, it's NOT part of the API, you're not even supposed to know it exists, it might change or disappear without notice, so if you use it and your code breaks then it's your problem".

If you want to keep said methods in the class just for structural reasons, you might as well make them static, by using the #staticmethod decorator:
class Foo():
#staticmethod
def my_static_method(*args, **kwargs):
....
Your first argument will not be interpretted as the object itself, and you can use it from either the class or an object from that class. If you still need to access class attributes in your method though, you can make it a class method:
class Bar():
counter = 0
#classmethod
def my_class_method(cls, *args, **kwargs):
cls.counter += 1
...
Your first argument of the class method will obviously be the class instead of the instance.
If you do not use any class or instance attribute, I can see no "theoretical" reason to not make them static. Some IDE's even highlight this as a soft warning to prompt you to make the method static if it does not use or mutate any class/instance attribute.

Why do we use #staticmethod?

I just can't see why do we need to use #staticmethod. Let's start with an exmaple.
class test1:
def __init__(self,value):
self.value=value
#staticmethod
def static_add_one(value):
return value+1
#property
def new_val(self):
self.value=self.static_add_one(self.value)
return self.value
a=test1(3)
print(a.new_val) ## >>> 4
class test2:
def __init__(self,value):
self.value=value
def static_add_one(self,value):
return value+1
#property
def new_val(self):
self.value=self.static_add_one(self.value)
return self.value
b=test2(3)
print(b.new_val) ## >>> 4
In the example above, the method, static_add_one , in the two classes do not require the instance of the class(self) in calculation.
The method static_add_one in the class test1 is decorated by #staticmethod and work properly.
But at the same time, the method static_add_one in the class test2 which has no #staticmethod decoration also works properly by using a trick that provides a self in the argument but doesn't use it at all.
So what is the benefit of using #staticmethod? Does it improve the performance? Or is it just due to the zen of python which states that "Explicit is better than implicit"?

The reason to use staticmethod is if you have something that could be written as a standalone function (not part of any class), but you want to keep it within the class because it's somehow semantically related to the class. (For instance, it could be a function that doesn't require any information from the class, but whose behavior is specific to the class, so that subclasses might want to override it.) In many cases, it could make just as much sense to write something as a standalone function instead of a staticmethod.
Your example isn't really the same. A key difference is that, even though you don't use self, you still need an instance to call static_add_one --- you can't call it directly on the class with test2.static_add_one(1). So there is a genuine difference in behavior there. The most serious "rival" to a staticmethod isn't a regular method that ignores self, but a standalone function.

Today I suddenly find a benefit of using #staticmethod.
If you created a staticmethod within a class, you don't need to create an instance of the class before using the staticmethod.
For example,
class File1:
def __init__(self, path):
out=self.parse(path)
def parse(self, path):
..parsing works..
return x
class File2:
def __init__(self, path):
out=self.parse(path)
#staticmethod
def parse(path):
..parsing works..
return x
if __name__=='__main__':
path='abc.txt'
File1.parse(path) #TypeError: unbound method parse() ....
File2.parse(path) #Goal!!!!!!!!!!!!!!!!!!!!
Since the method parse is strongly related to the classes File1 and File2, it is more natural to put it inside the class. However, sometimes this parse method may also be used in other classes under some circumstances. If you want to do so using File1, you must create an instance of File1 before calling the method parse. While using staticmethod in the class File2, you may directly call the method by using the syntax File2.parse.
This makes your works more convenient and natural.

I will add something other answers didn't mention. It's not only a matter of modularity, of putting something next to other logically related parts. It's also that the method could be non-static at other point of the hierarchy (i.e. in a subclass or superclass) and thus participate in polymorphism (type based dispatching). So if you put that function outside the class you will be precluding subclasses from effectively overriding it. Now, say you realize you don't need self in function C.f of class C, you have three two options:
Put it outside the class. But we just decided against this.
Do nothing new: while unused, still keep the self parameter.
Declare you are not using the self parameter, while still letting other C methods to call f as self.f, which is required if you wish to keep open the possibility of further overrides of f that do depend on some instance state.
Option 2 demands less conceptual baggage (you already have to know about self and methods-as-bound-functions, because it's the more general case). But you still may prefer to be explicit about self not being using (and the interpreter could even reward you with some optimization, not having to partially apply a function to self). In that case, you pick option 3 and add #staticmethod on top of your function.

Use #staticmethod for methods that don't need to operate on a specific object, but that you still want located in the scope of the class (as opposed to module scope).
Your example in test2.static_add_one wastes its time passing an unused self parameter, but otherwise works the same as test1.static_add_one. Note that this extraneous parameter can't be optimized away.
One example I can think of is in a Django project I have, where a model class represents a database table, and an object of that class represents a record. There are some functions used by the class that are stand-alone and do not need an object to operate on, for example a function that converts a title into a "slug", which is a representation of the title that follows the character set limits imposed by URL syntax. The function that converts a title to a slug is declared as a staticmethod precisely to strongly associate it with the class that uses it.

Dynamically loaded subclass can still instanstiate even without all abstract methods overridden

Setup: Python 3.3
I have a base class, called SourceBase, which defines abstract methods and values:
import abc
class SourceBase(object):
__metaclass__=abc.ABCMeta
pluginid='undefined' #OVERRIDE THIS IN YOUR SUBCLASS. If you don't, the program will ignore your plugin.
#abc.abstractmethod
def get_images(self):
'''This method should return a list of URLs.'''
return
#abc.abstractmethod
def get_source_info(self):
'''This method should return a list containing a human friendly name at index 0, and a human readable url describing the source for this repository.
For example, the EarthPorn subreddit returns a list ['EarthPorn Subreddit', 'http://reddit.com/r/EarthPorn'].
This is used to populate the treeview object with your source information.'''
return
#abc.abstractmethod
def get_pluginid(self):
'''This method should return a string that represents this plugins ID.
The pluginid is used to make calls to this plugin when necessary. It should be unique as ids are in a shared pool,
so make sure the id is unique. The id should remain the same even when updated as some settings with the pluginid
are persisted by the main application, and they will be lost if the id changes.
'''
return
This is the superclass of some python plugins I wrote, which subclass this. They are dynamically loaded at runtime, and all of this works, except that even though I added a new abstract method to my SourceBase, the plugins still load. They shouldn't, since none of them have my new method. (I gave it the #abc.abstractmethod marking).
My google-fu doesn't really show anything, so I'm not sure why I can still instanstiate these plugins even though the superclass says they are abstract.
For example, in SourceBase, I added:
#abc.abstractmethod
def get_dependencies(self):
print('ERROR: THIS PLUGIN SHOULD NOT HAVE LOADED.')
'''This method should return a list of package names. The host program will check if these packages are available.
If they are not available, the plugin will be disabled, but the user will be able to elect to install these packages.'''
return
I did not define this method in my plugins, but I still get this output on the terminal:
....
Screen Height:1080
Screen Width:1920
files: ['bingIOTD.py', 'redditEP.py', 'redditLP.py', '__init__.py']
ERROR: THIS PLUGIN SHOULD NOT HAVE LOADED. <--abstract method
I'm not sure why it is ignoring it, am I missing something...? I've done it before with normal classes that aren't dynamically loaded. Any help is appreciated. I understand I can probably make a workaround (make a default return, check for that), but that doesn't seem to be the right way.
If you need more sourcecode my project is on SourceForge here.

In Python3 the metaclass is specified by
class SourceBase(metaclass=abc.ABCMeta):
not
class SourceBase(object):
__metaclass__=abc.ABCMeta
The code is ignoring the abstractmethod decorator because as far as Python3 is concerned, SourceBase is simply a standard class (instance of type) with an attribute name __metaclass__ rather than being an instance of abc.ABCMeta.

Python: best way to test a single method of a class

I have a class like the following:
class A:
def __init__(self, arg1, arg2, arg3):
self.a=arg1
self.b=arg2
self.c=arg3
# ...
self.x=do_something(arg1, arg2, arg3)
self.y=do_something(arg1, arg2, arg3)
self.m = self.func1(self.x)
self.n = self.func2(self.y)
# ...
def func1(self, arg):
# do something here
def func2(self, arg):
# do something here
As you can see, initializing the class needs to feed in arg1, arg2, and arg3. However, testing func1 and func2 does not directly require such inputs, but rather, it's simply an input/output logic.
In my test, I can of course instantiate and initialize a test object in the regular way, and then test func1 and func2 individually. But the initialization requires input of arg1 arg2, arg3, which is really not relevant to test func1 and func2.
Therefore, I want to test func1 and func2 individually, without first calling __init__. So I have the following 2 questions:
What's the best way of designing such tests? (perferably, in py.test)
I want to test func1 and func2 without invoking __init__. I read from here that A.__new__() can skip invoking __init__ but still having the class instantiated. Is there a better way to achieve what I need without doing this?
NOTE:
There have been 2 questions regarding my ask here:
Is it necessary to test individual member functions?
(for testing purpose) Is it necessary to instantiating a class without initializing the object with __init__?
For question 1, I did a quick google search and find some relevant study or discussion on this:
Unit Testing Non Public Member Functions
(PDF) Incremental Testing of Object-Oriented Class Structures.
We initially test base classes having no parents by designing a test
suite that tests each member function individually and also tests the
interactions among member functions.
For question 2, I'm not sure. But I think it is necessary, as shown in the sample code, func1 and func2 are called in __init__. I feel more comfortable testing them on an class A object that hasn't been called with __init__ (and therefore no previous calls to func1 and func2).
Of course, one could just instantiate a class A object with regular means (testobj = A()) and then perform individual test on func1 and func2. But is it good:)? I'm just discussing here as what's the best way to test such scenario, what's the pros and cons.
On the other hand, one might also argue that from design perspective one should NOT put calls to func1 and func2 in __init__ in the first place. Is this a reasonable design option?

It is not usually useful or even possible to test methods of a class without instantiating the class (including running __init__). Typically your class methods will refer to attributes of the class (e.g., self.a). If you don't run __init__, those attributes won't exist, so your methods won't work. (If your methods don't rely on the attributes of their instance, then why are they methods and not just standalone functions?) In your example, it looks like func1 and func2 are part of the initialization process, so they should be tested as part of that.
In theory it is possible to "quasi-instantiate" the class by using __new__ and then adding just the members that you need, e.g.:
obj = A.__new__(args)
obj.a = "test value"
obj.func1()
However, this is probably not a very good way to do tests. For one thing, it results in you duplicating code that presumably already exists in the initialization code, which means your tests are more likely to get out of sync with the real code. For another, you may have to duplicate many initialization calls this way, since you'll have to manually re-do what would otherwise be done by any base-class __init__ methods called from your class.
As for how to design tests, you can take a look at the unittest module and/or the nose module. That gives you the basics of how to set up tests. What to actually put in the tests obviously depends on what your code is supposed to do.
Edit: The answer to your question 1 is "definitely yes, but not necessarily every single one". The answer to your question 2 is "probably not". Even at the first link you give, there is debate about whether methods that are not part of the class's public API should be tested at all. If your func1 and func2 are purely internal methods that are just part of the initialization, then there is probably no need to test them separately from the initialization.
This gets to your last question about whether it's appropriate to call func1 and func2 from within __init__. As I've stated repeatedly in my comments, it depends on what these functions do. If func1 and func2 perform part of the initialization (i.e., do some "setting-up" work for the instance), then it's perfectly reasonable to call them from __init__; but in that case they should be tested as part of the initialization process, and there is no need to test them independently. If func1 and func2 are not part of the initialization, then yes, you should test them independently; but in that case, why are they in __init__?
Methods that form an integral part of instantiating your class should be tested as part of testing the instantiation of your class. Methods that do not form an integral part of instantiating your class should not be called from within __init__.
If func1 and func2 are "simply an input/output logic" and do not require access to the instance, then they don't need to be methods of the class at all; they can just be standalone functions. If you want to keep them in the class you can mark them as staticmethods and then call them on the class directly without instantiating it. Here's an example:
>>> class Foo(object):
... def __init__(self, num):
... self.numSquared = self.square(num)
...
... #staticmethod
... def square(num):
... return num**2
>>> Foo.square(2) # you can test the square "method" this way without instantiating Foo
4
>>> Foo(8).numSquared
64
It is just imaginable that you might have some monster class which requires a hugely complex initialization process. In such a case, you might find it necessary to test parts of that process individually. However, such a giant init sequence would itself be a warning of an unwieldy designm.

If you have a choice, i'd go for declaring your initialization helper functions as staticmethods and just call them from tests.
If you have different input/output values to assert on, you could look into some parametrizing examples with py.test.
If your class instantiation is somewhat heavy you might want to look into dependency injection and cache the instance like this:
# content of test_module.py
def pytest_funcarg__a(request):
return request.cached_setup(lambda: A(...), scope="class")
class TestA:
def test_basic(self, a):
assert .... # check properties/non-init functions
This would re-use the same "a" instance across each test class. Other possible scopes are "session", "function" or "module". You can also define a command line option to set the scope so that for quick development you use more caching and for Continous-Integration you use more isolated resource setup, without the need to change the test source code.
Personally, in the last 12 years i went from fine-grained unit-testing to more functional/integration types of testing because it eases refactoring and seemed to make better use of my time overall. It's of course crucial to have good support and reports when failures occur, like dropping to PDB, concise tracebacks etc. And for some intricate algorithms i still write very fine-grained unit-tests but then i usually separate the algorithm out into a very independently testable thing.
HTH, holger

I agree with previous comments that it is generally better to avoid this problem by reducing the amount of work done at instantiation, e.g. by moving func1 etc calls into aconfigure(self) method which should be called after instantiation.
If you have strong reasons for keeping calls to self.func1 etc in __init__, there is an approach in pytest which might help.
(1) Put this in the module:
_called_from_test = False
(2) Put the following in conftest.py
import your_module
def pytest_configure(config):
your_module._called_from_test = True
with the appropriate name for your_module.
(3) Insert an if statement to end the execution of __init__ early when you are running tests,
if _called_from_test:
pass
else:
self.func1( ....)
You can then step through the individual function calls, testing them as you go.
The same could be achieved by making _called_from_test an optional argument of __init__.
More context is given in the Detect if running from within a pytest run section of pytest documentation.