Develop Reference

Stop Sphinx from Executing a cached classmethod property

I am writing a package for interacting with dataset and have code that looks something like
from abc import ABC, ABCMeta, abstractmethod
from functools import cache
from pathlib import Path
from warnings import warn
class DatasetMetaClass(ABCMeta):
r"""Meta Class for Datasets"""
#property
#cache
def metaclass_property(cls):
r"""Compute an expensive property (for example: dataset statistics)."""
warn("Caching metaclass property...")
return "result"
# def __dir__(cls):
# return list(super().__dir__()) + ['metaclass_property']
class DatasetBaseClass(metaclass=DatasetMetaClass):
r"""Base Class for datasets that all datasets must subclass"""
#classmethod
#property
#cache
def baseclass_property(cls):
r"""Compute an expensive property (for example: dataset statistics)."""
warn("Caching baseclass property...")
return "result"
class DatasetExampleClass(DatasetBaseClass, metaclass=DatasetMetaClass):
r"""Some Dataset Example."""
Now, the problem is that during make html, sphinx actually executes the baseclass_property which is a really expensive operation. (Among other things: checks if dataset exists locally, if not, downloads it, preprocesses it, computes dataset statistics, mows the lawn and takes out the trash.)
I noticed that this does not happen if I make it a MetaClass property, because the meta-class property does not appear in the classes __dir__ call which may or may not be a bug. Manually adding it to __dir__ by uncommenting the two lines causes sphinx to also process the metaclass property.
Questions:
Is this a bug in Sphinx? Given that #properties are usually handled fine, it seems unintended that it breaks for #classmethod#property.
What is the best option - at the moment - to avoid this problem? Can I somehow tell Sphinx to not parse this function? I hope that there is a possibility to either disable sphinx for a function via comment similarly to # noqa, # type: ignore, # pylint disable= etc. or via some kind of #nodoc decorator.

Everything is working as it should, and there is no "bug" there either in Sphinx, nor in the ABC machinery, and even less in the language.
Sphinx uses th language introspection capabilities to retrieve a class's members and then introspect then for methods. What happens when you combine #classmethod and #property is that, besides it somewhat as a nice surprise actually work, when the class member thus created is accessed by Sphynx, as it must do in search for the doc strings, the code is triggered and runs.
It would actually be less surprising if property and classmethod could not be used in combination actually since both property and classmethod decorators use the descriptor protocol to create a new object with the appropriate methods for the feature they implement.
I think the less surprising thing to go there is to put some explicit guard inside your "classmethod property cache" functions to not run when the file is being processed by sphinx. Since sphinx do not have this feature itself, you can use an environment variable for that, say GENERATING_DOCS. (this does not exist, it can be any name), and then a guard inside your methods like:
...
def baseclass_property(self):
if os.environ.get("GENERATING_DOCS", False):
return
And then you either set this variable manually before running the script, or set it inside Sphinx' conf.py file itself.
If you have several such methods, and don't want to write the guard code in all of them, you could do a decorator, and while at that, just use the same decorator to apply the other 3 decorators at once:
from functools import cache, wraps
import os
def cachedclassproperty(func):
#wraps(func)
def wrapper(*args, **kwargs):
if os.environ.get("GENERATING_DOCS", False):
return
return func(*args, **kwargs)
return classmethod(property(cache(wrapper)))
Now, as for using the property on the metaclass: I advise against it. Metaclasses are for when you really need to customize your class creation process, and it is almost by chance that property on a metaclass works as a class property as well. All that happens in this case, as ou have investigated, is that the property will be hidden from a class' dir, and therefore won't be hit by Sphinx introspection - but even if you are using a metaclass for some other purpose, if you simply add a guard as I had suggested might even not prevent sphinx from properly documenting the class property, if it has a docstring. If you hide it from Sphinx, it will obviously go undocumented.

Is it possible to set a Metaclass globally so it applies to all classes created by default?

I get that a metaclass can be substituted for type and define how a newly created class behaves.
ex:
class NoMixedCase(type):
def __new__(cls,clsname,base,clsdict):
for name in clsdict:
if name.lower() != name:
raise TypeError("Bad name.Don't mix case!")
return super().__new__(cls,clsname,base,clsdict)
class Root(metaclass=NoMixedCase):
pass
class B(Root):
def Foo(self): #type error
pass
However, is there a way of setting NoMixedCase globally, so anytime a new class is created it's behavior is defined by NoMixedCase by default, without havining to inherit from Root?
So if you did...
Class B:
def Foo(self):
pass
...it would still check case on method names.

As for your question, no, it it is not ordinarily - and possibly not even some extra-ordinary thng that will work for this - a lot of CPythons inner things are tied to the type class, and hardcoded to it.
What is possible of trying, without crashing the interpretrer right away, would be to write a wrapper for type.__new__ and use ctypes to replace it directly in type.__new__ slot. (Ordinary assignment won't do it). You'd probably still crash things.
So, in real life, if you decide not to go via a linter program with a plug-in and commit hooks as I suggested in the comment above, the way to go is to have a Base class that uses your metaclass, and get everyone in your project to inherit from that Base.

python: How do I dynamically create bound methods from user supplied source code?

I would like to construct a class in python that supports dynamic updating of methods from user supplied source code.
Instances of class Agent have a method go. At the time an instance is constructed, its .go() method does nothing. For example, if we do a=Agent(), and then a.go() we should get a NotImplementedError or something like that. The user then should be able to interactively define a.go() by supplying source code. A simple source code example would be
mySourceString = "print('I learned how to go!')"
which would be injected into a like this
a.update(mySourceString)
Further invokations of a.go() would then result in "I learned how to go!" being printed to the screen.
I have partially figured out how to do this with the following code:
import types
class Error(Exception):
"""Base class for exceptions in this module."""
pass
class NotImplementedError(Error):
pass
class Agent(object):
def go(self):
raise NotImplementedError()
def update(self,codeString):
#Indent each line of user supplied code
codeString = codeString.replace('\n','\n ')
#Turn code into a function called func
exec "def func(self):\n"+' '+codeString
#Make func a bound method on this instance
self.go = types.MethodType(func, self)
QUESTIONS
Is this implementation sensible?
Will this implementation incur unexpected scope issues?
Is there an obvious way to sandbox the user supplied code to prevent it from touching external objects? I can think of ways to do this by supplying sets of allowed external objects, but this seems not pythonic.
Possibly useful SO posts
What's the difference between eval, exec, and compile in Python?
Adding a Method to an Existing Object
(I am working in python 2.6)

Plugin architecture - Plugin Manager vs inspecting from plugins import *

I'm currently writing an application which allows the user to extend it via a 'plugin' type architecture. They can write additional python classes based on a BaseClass object I provide, and these are loaded against various application signals. The exact number and names of the classes loaded as plugins is unknown before the application is started, but are only loaded once at startup.
During my research into the best way to tackle this I've come up with two common solutions.
Option 1 - Roll your own using imp, pkgutil, etc.
See for instance, this answer or this one.
Option 2 - Use a plugin manager library
Randomly picking a couple
straight.plugin
yapsy
this approach
My question is - on the proviso that the application must be restarted in order to load new plugins - is there any benefit of the above methods over something inspired from this SO answer and this one such as:
import inspect
import sys
import my_plugins
def predicate(c):
# filter to classes
return inspect.isclass(c)
def load_plugins():
for name, obj in inspect.getmembers(sys.modules['my_plugins'], predicate):
obj.register_signals()
Are there any disadvantages to this approach compared to the ones above? (other than all the plugins must be in the same file) Thanks!
EDIT
Comments request further information... the only additional thing I can think to add is that the plugins use the blinker library to provide signals that they subscribe to. Each plugin may subscribe to different signals of different types and hence must have its own specific "register" method.

Since Python 3.6 a new class method __init_subclass__ is added, that is called on a base class, whenever a new subclass is created.
This method can further simplify the solution offered by will-hart above, by removing the metaclass.
The __init_subclass__ method was introduced with PEP 487: Simpler customization of class creation. The PEP comes with a minimal example for a plugin architecture:
It is now possible to customize subclass creation without using a
metaclass. The new __init_subclass__ classmethod will be called on
the base class whenever a new subclass is created:
class PluginBase:
subclasses = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
class Plugin1(PluginBase):
pass
class Plugin2(PluginBase):
pass
The PEP example above stores references to the classes in the Plugin.plugins field.
If you want to store instances of the plugin classes, you can use a structure like this:
class Plugin:
"""Base class for all plugins. Singleton instances of subclasses are created automatically and stored in Plugin.plugins class field."""
plugins = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.plugins.append(cls())
class MyPlugin1(Plugin):
def __init__(self):
print("MyPlugin1 instance created")
def do_work(self):
print("Do something")
class MyPlugin2(Plugin):
def __init__(self):
print("MyPlugin2 instance created")
def do_work(self):
print("Do something else")
for plugin in Plugin.plugins:
plugin.do_work()
which outputs:
MyPlugin1 instance created
MyPlugin2 instance created
Do something
Do something else

The metaclass approach is useful for this issue in Python < 3.6 (see #quasoft's answer for Python 3.6+). It is very simple and acts automatically on any imported module. In addition, complex logic can be applied to plugin registration with very little effort. It requires:
The metaclass approach works like the following:
1) A custom PluginMount metaclass is defined which maintains a list of all plugins
2) A Plugin class is defined which sets PluginMount as its metaclass
3) When an object deriving from Plugin - for instance MyPlugin is imported, it triggers the __init__ method on the metaclass. This registers the plugin and performs any application specific logic and event subscription.
Alternatively if you put the PluginMount.__init__ logic in PluginMount.__new__ it is called whenver a new instance of a Plugin derived class is created.
class PluginMount(type):
"""
A plugin mount point derived from:
http://martyalchin.com/2008/jan/10/simple-plugin-framework/
Acts as a metaclass which creates anything inheriting from Plugin
"""
def __init__(cls, name, bases, attrs):
"""Called when a Plugin derived class is imported"""
if not hasattr(cls, 'plugins'):
# Called when the metaclass is first instantiated
cls.plugins = []
else:
# Called when a plugin class is imported
cls.register_plugin(cls)
def register_plugin(cls, plugin):
"""Add the plugin to the plugin list and perform any registration logic"""
# create a plugin instance and store it
# optionally you could just store the plugin class and lazily instantiate
instance = plugin()
# save the plugin reference
cls.plugins.append(instance)
# apply plugin logic - in this case connect the plugin to blinker signals
# this must be defined in the derived class
instance.register_signals()
Then a base plugin class which looks like:
class Plugin(object):
"""A plugin which must provide a register_signals() method"""
__metaclass__ = PluginMount
Finally, an actual plugin class would look like the following:
class MyPlugin(Plugin):
def register_signals(self):
print "Class created and registering signals"
def other_plugin_stuff(self):
print "I can do other plugin stuff"
Plugins can be accessed from any python module that has imported Plugin:
for plugin in Plugin.plugins:
plugin.other_plugin_stuff()
See the full working example

The approach from will-hart was the most useful one to me!
For i needed more control I wrapped the Plugin Base class in a function like:
def get_plugin_base(name='Plugin',
cls=object,
metaclass=PluginMount):
def iter_func(self):
for mod in self._models:
yield mod
bases = not isinstance(cls, tuple) and (cls,) or cls
class_dict = dict(
_models=None,
session=None
)
class_dict['__iter__'] = iter_func
return metaclass(name, bases, class_dict)
and then:
from plugin import get_plugin_base
Plugin = get_plugin_base()
This allows to add additional baseclasses or switching to another metaclass.

Dynamic sub-classing in Python

I have a number of atomic classes (Components/Mixins, not really sure what to call them) in a library I'm developing, which are meant to be subclassed by applications. This atomicity was created so that applications can only use the features that they need, and combine the components through multiple inheritance.
However, sometimes this atomicity cannot be ensured because some component may depend on another one. For example, imagine I have a component that gives a graphical representation to an object, and another component which uses this graphical representation to perform some collision checking. The first is purely atomic, however the latter requires that the current object already subclassed this graphical representation component, so that its methods are available to it. This is a problem, because we have to somehow tell the users of this library, that in order to use a certain Component, they also have to subclass this other one. We could make this collision component sub class the visual component, but if the user also subclasses this visual component, it wouldn't work because the class is not on the same level (unlike a simple diamond relationship, which is desired), and would give the cryptic meta class errors which are hard to understand for the programmer.
Therefore, I would like to know if there is any cool way, through maybe metaclass redefinition or using class decorators, to mark these unatomic components, and when they are subclassed, the additional dependency would be injected into the current object, if its not yet available. Example:
class AtomicComponent(object):
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent(object):
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
Can someone give me an hint on how I can do this? or if it is even possible...
edit:
Since it is debatable that the meta class solution is the best one, I'll leave this unaccepted for 2 days.
Other solutions might be to improve error messages, for example, doing something like UserClass2 would give an error saying that UnAtomicComponent already extends this component. This however creates the problem that it is impossible to use two UnAtomicComponents, given that they would subclass object on different levels.

"Metaclasses"
This is what they are for! At time of class creation, the class parameters run through the
metaclass code, where you can check the bases and change then, for example.
This runs without error - though it does not preserve the order of needed classes
marked with the "depends" decorator:
class AutoSubclass(type):
def __new__(metacls, name, bases, dct):
new_bases = set()
for base in bases:
if hasattr(base, "_depends"):
for dependence in base._depends:
if not dependence in bases:
new_bases.add(dependence)
bases = bases + tuple(new_bases)
return type.__new__(metacls, name, bases, dct)
__metaclass__ = AutoSubclass
def depends(*args):
def decorator(cls):
cls._depends = args
return cls
return decorator
class AtomicComponent:
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent:
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
(I removed inheritance from "object", as I declared a global __metaclass__ variable. All classs will still be new style class and have this metaclass. Inheriting from object or another class does override the global __metaclass__variable, and a class level __metclass__ will have to be declared)
-- edit --
Without metaclasses, the way to go is to have your classes to properly inherit from their dependencies. Tehy will no longer be that "atomic", but, since they could not work being that atomic, it may be no matter.
In the example bellow, classes C and D would be your User classes:
>>> class A(object): pass
...
>>> class B(A, object): pass
...
>>>
>>> class C(B): pass
...
>>> class D(B,A): pass
...
>>>