I've made a classproperty descriptor and whenever I use a function decorated with it, I get multiple pylint inspection errors.
Here is a sample class with a sample decorated function:
class Bar:
"""
Bar documentation.
"""
# pylint: disable=no-method-argument
#classproperty
def foo():
"""
Retrieve foo.
"""
return "foo"
Thanks to the descriptor, I can call Bar.foo and get the string foo returned.
Unfortunately, whenever I use functions like this with slightly more complex items (e.g. functions which return instances of objects), pylint starts complaining about things such as no-member or unexpected-keyword-arg, simply because it thinks Bar.foo is a method, rather than a wrapped classproperty object.
I would like to disable warnings for any code that uses my function - I definitely can't allow having to write # pylint: disable every single time I use the classproperty-wrapped methods. How can I do it with pylint? Or maybe I should switch to use a different linter instead?
Here is an example of a warning generated because of the reasons above:
class Bar:
"""
Bar documentation.
"""
# pylint: disable=no-method-argument
#classproperty
def foo():
"""
Retrieve an object.
"""
return NotImplementedError("Argument")
print(Bar.foo.args)
pylint complains that E1101: Method 'foo' has no 'args' member (no-member) (even though I know it definitely has), and I would like to completely disable some warnings for any module/class/function that uses Bar.foo.args or similar.
For anyone interested, here is a minimal implementation of a classproperty descriptor:
class classproperty:
"""
Minimal descriptor.
"""
# pylint: disable=invalid-name
def __init__(self, func):
self._func = func
def __get__(self, _obj, _type):
return self._func()
I have managed to create a dirty hack by type-hinting the items as None:
class Bar:
"""
Bar documentation.
"""
# pylint: disable=no-method-argument,function-redefined,too-few-public-methods
foo: None
#classproperty
def foo():
"""
Retrieve an object.
"""
return NotImplementedError("Argument")
I would rather avoid having code like this because I can't actually import the items which should be type-hinted due to the circular imports issue (hence None), but it tricks pylint well.
As far as I know, it's not possible.
I haven't found a way to solve this in pylint's configuration. The closest I could find is the property-classes option, but it only influences the invalid-name checker, so not what we are looking for here:
:property-classes:
List of decorators that produce properties, such as abc.abstractproperty. Add
to this list to register other decorators that produce valid properties.
These decorators are taken in consideration only for invalid-name.
Default: ``abc.abstractproperty``
Maybe it's a question that is worth asking pylint's developers directly.
Seems to me like it's something that could be solved with a transform plugin (Maybe this for inspiration?). Pylint handles #property decorators perfectly fine, so something like the #classproperty suggested here, should be feasible as well.
Aside
(You might know those already)
For properties on classes:
Using property() on classmethods
How to make a class property?
Related
I am trying to provide wrappers for short-cutting every-day commands. Python environments are very useful to do that.
Is it possible to provide all methods of an object to the local namespace within a new environment?
class my_object:
def method_a():
...
class my_environment:
...
def __enter__(self):
some_object = my_object()
# something like `from some_object import *` ??
return(some_object)
...
with my_environment() as some_object:
# standard syntax:
some_object.method_a()
# shortcut:
method_a() # how to make this possible?
It will be rather complex, and IMHO will not be worth it. The problem is that in Python, local variables are local to a function and not to a bloc. So what you are asking for would require that:
__enter__ declares nonlocal variables for all of the methods from some_object and saves their previous value if any
__exit__ restore the previous values if any of those variables, or deletes them if they did not previously existed
Possible but not really Pythonic IMHO (the reason why I have not proposed any code...). After all, inside a method Python requires the object to be explicitely passed, and requires it to be prepended to any internal method call or attribute access. So my advice is to stick to the standard syntax here...
What you are looking for is class hierarchy. On the way, please be careful with the conventions for class names.
class MyObject:
def method_a():
...
class MyEnvironment(MyObject):
...
def __enter__(self):
return self
...
with MyEnvironment() as some_object:
# standard syntax:
some_object.method_a()
The shortcut you are looking doesn't make much sense because the method_a() was defined as a method, therefore it should be called together with the instance.
Maybe #staticmethod can serve your case better.
class MyEnvironment:
#staticmethod
def method_a():
...
MyEnvironment.method_a()
I am writing a package for interacting with dataset and have code that looks something like
from abc import ABC, ABCMeta, abstractmethod
from functools import cache
from pathlib import Path
from warnings import warn
class DatasetMetaClass(ABCMeta):
r"""Meta Class for Datasets"""
#property
#cache
def metaclass_property(cls):
r"""Compute an expensive property (for example: dataset statistics)."""
warn("Caching metaclass property...")
return "result"
# def __dir__(cls):
# return list(super().__dir__()) + ['metaclass_property']
class DatasetBaseClass(metaclass=DatasetMetaClass):
r"""Base Class for datasets that all datasets must subclass"""
#classmethod
#property
#cache
def baseclass_property(cls):
r"""Compute an expensive property (for example: dataset statistics)."""
warn("Caching baseclass property...")
return "result"
class DatasetExampleClass(DatasetBaseClass, metaclass=DatasetMetaClass):
r"""Some Dataset Example."""
Now, the problem is that during make html, sphinx actually executes the baseclass_property which is a really expensive operation. (Among other things: checks if dataset exists locally, if not, downloads it, preprocesses it, computes dataset statistics, mows the lawn and takes out the trash.)
I noticed that this does not happen if I make it a MetaClass property, because the meta-class property does not appear in the classes __dir__ call which may or may not be a bug. Manually adding it to __dir__ by uncommenting the two lines causes sphinx to also process the metaclass property.
Questions:
Is this a bug in Sphinx? Given that #properties are usually handled fine, it seems unintended that it breaks for #classmethod#property.
What is the best option - at the moment - to avoid this problem? Can I somehow tell Sphinx to not parse this function? I hope that there is a possibility to either disable sphinx for a function via comment similarly to # noqa, # type: ignore, # pylint disable= etc. or via some kind of #nodoc decorator.
Everything is working as it should, and there is no "bug" there either in Sphinx, nor in the ABC machinery, and even less in the language.
Sphinx uses th language introspection capabilities to retrieve a class's members and then introspect then for methods. What happens when you combine #classmethod and #property is that, besides it somewhat as a nice surprise actually work, when the class member thus created is accessed by Sphynx, as it must do in search for the doc strings, the code is triggered and runs.
It would actually be less surprising if property and classmethod could not be used in combination actually since both property and classmethod decorators use the descriptor protocol to create a new object with the appropriate methods for the feature they implement.
I think the less surprising thing to go there is to put some explicit guard inside your "classmethod property cache" functions to not run when the file is being processed by sphinx. Since sphinx do not have this feature itself, you can use an environment variable for that, say GENERATING_DOCS. (this does not exist, it can be any name), and then a guard inside your methods like:
...
def baseclass_property(self):
if os.environ.get("GENERATING_DOCS", False):
return
And then you either set this variable manually before running the script, or set it inside Sphinx' conf.py file itself.
If you have several such methods, and don't want to write the guard code in all of them, you could do a decorator, and while at that, just use the same decorator to apply the other 3 decorators at once:
from functools import cache, wraps
import os
def cachedclassproperty(func):
#wraps(func)
def wrapper(*args, **kwargs):
if os.environ.get("GENERATING_DOCS", False):
return
return func(*args, **kwargs)
return classmethod(property(cache(wrapper)))
Now, as for using the property on the metaclass: I advise against it. Metaclasses are for when you really need to customize your class creation process, and it is almost by chance that property on a metaclass works as a class property as well. All that happens in this case, as ou have investigated, is that the property will be hidden from a class' dir, and therefore won't be hit by Sphinx introspection - but even if you are using a metaclass for some other purpose, if you simply add a guard as I had suggested might even not prevent sphinx from properly documenting the class property, if it has a docstring. If you hide it from Sphinx, it will obviously go undocumented.
I am writing a python API/server to allow an external device (microcontroller) to remotely call methods of an object by sending a string with the name of the method. These methods would be stored in a dictionary. e.g. :
class Server:
...
functions = {}
def register(self, func):
self.functions[func.__name__] = func
def call(self, func_name, args):
self.functions[func_name](*args)
...
I know that I could define functions externally to the class definition and register them manually, but I would really like that the registering step would be done automatically. Consider the following class:
class MyServer(Server):
...
def add(self,a,b):
print a+b
def sub(self,a,b):
print a-b
...
It would work by subclassing a server class and by defining methods to be called. How could I get the methods to be automatically registered in the functions dictionary?
One way that I thought it could be done is with a metaclass that look at a pattern in the methods name add if a match is found, add that methods to the functions dictionary. It seems overkill...
Would it be possible to decorate the methods to be registered? Can someone give me a hint to the simplest solution to this problem?
There is no need to construct a dictionary, just use the getattr() built-in function:
def call(self, func_name, args):
getattr(self, func_name)(*args)
Python actually uses a dictionary to access attributes on objects anyway (it's called __dict__, - but using getattr() is better than accessing it directly).
If you really want to construct that dict for some reason, then look at the inspect module:
def __init__(self, ...):
self.functions = dict(inspect.getmembers(self, inspect.ismethod))
If you want to pick specific methods, you could use a decorator to do that, but as BrenBarn points out, the instance doesn't exist at the time the methods are decorated, so you need to use the mark and recapture technique to do what you want.
I am writing a small mocking class to do some tests.
But this class needs to support the idea of having nested attributes.
This example should provide some insight to the problem:
class Foo(object):
def __init__(self):
self.x = True
From the above class, we can have:
f = Foo()
f.x
I know I can add attributes falling back to __getattr__ to avoid an AttributeError, but what if I need something like this to be valid:
f = Foo()
f.x
f.x.y
f.x.y.z()
I know what to return if the object gets called as f.x.y.z() but I just need to find a way to get to z() that makes sense.
You can "mock anything" by returning, on each attribute access, another instance of the "mock anything" class (which must also be callable, if you want to have the .z() part work;-).
E.g.:
class MockAny(object):
# mock special methods by making them noops
def __init__(self, *a, **k): pass
# or returning fixed values
def __len__(self): return 0
# mock attributes:
def getattr(self, name):
return MockAny()
# make it callable, if you need to
def __call__(self, *a, **k):
return MockAny()
The alternative, of course, is to know what it is that you're mocking (by introspection, or by some form of "declarative description", or simply by coding mock for specific things;-) rather than take the catch-all approach; but, the latter is also feasible, as you see in the above (partial) example.
Personally, I'd recommend using an existing mocking framework such as pymox rather than reinventing this particular wheel (also, the source code for such frameworks can be more instructive than a reasonably terse response on SO, like this one;-).
If you are calling something like f.x.y.z() in your unit tests, the chances are you're trying to test too much. Each of these nested attributes should be covered by the unit tests for their particular classes.
Take another look at your Foo class and see if you can test its own behaviour in your unit tests.
Perhaps not the answer you were looking for, but hopefully one that will help in the long run.
I'd like to write a python function that has a dynamically created docstring. In essence for a function func() I want func.__doc__ to be a descriptor that calls a custom __get__ function create the docstring on request. Then help(func) should return the dynamically generated docstring.
The context here is to write a python package wrapping a large number of command line tools in an existing analysis package. Each tool becomes a similarly named module function (created via function factory and inserted into the module namespace), with the function documentation and interface arguments dynamically generated via the analysis package.
You can't do what you're looking to do, in the way you want to do it.
From your description it seems like you could do something like this:
for tool in find_tools():
def __tool(*arg):
validate_args(tool, args)
return execute_tool(tool, args)
__tool.__name__ = tool.name
__tool.__doc__ = compile_docstring(tool)
setattr(module, tool.name, __tool)
i.e. create the documentation string dynamically up-front when you create the function.
Is the a reason why the docstring has to be dynamic from one call to __doc__ to the next?
Assuming there is, you'll have to wrap your function up in a class, using __call__ to trigger the action.
But even then you've got a problem. When help() is called to find the docstring, it is called on the class, not the instance, so this kind of thing:
class ToolWrapper(object):
def __init__(self, tool):
self.tool = tool
self.__name__ = tool.name
def _get_doc(self):
return compile_docstring(self.tool)
__doc__ = property(_get_doc)
def __call__(self, *args):
validate_args(args)
return execute_tool(tool, args)
won't work, because properties are instance, not class attributes. You can make the doc property work by having it on a metaclass, rather than the class itself
for tool in find_tools():
# Build a custom meta-class to provide __doc__.
class _ToolMetaclass(type):
def _get_doc(self):
return create_docstring(tool)
__doc__ = property(_get_doc)
# Build a callable class to wrap the tool.
class _ToolWrapper(object):
__metaclass__ = _ToolMetaclass
def _get_doc(self):
return create_docstring(tool)
__doc__ = property(_get_doc)
def __call__(self, *args):
validate_args(tool, args)
execute_tool(tool, args)
# Add the tool to the module.
setattr(module, tool.name, _ToolWrapper())
Now you can do
help(my_tool_name)
and get the custom docstring, or
my_tool_name.__doc__
for the same thing. The __doc__ property is in the _ToolWrapper class is needed to trap the latter case.
(Python 3 solution)
You could make use of Python's duck typing to implement a dynamic string:
import time
def fn():
pass
class mydoc( str ):
def expandtabs( self, *args, **kwargs ):
return "this is a dynamic strting created on {}".format( time.asctime() ).expandtabs( *args, **kwargs )
fn.__doc__ = mydoc()
help( fn )
Caveats:
This assumes that the help function is calling .expandtabs to get the text from the __doc__ object, which works in Python 3.7. A more robust solution would implement the other str methods in order to have our duck continue acting like a duck even if the help method changes. Also note that our mydoc class derives from str, this is because help, somewhat atypically, enforces strong typing by asserting isinstance(thing.__doc__, str). Like all solutions this is a bit hacky, but whether this is a problem largely depends on the full project requirements.
Instead of messing with the function, why not write your own help function?
my_global=42
def help(func):
print('%s: my_global=%s'%(func.func_name,my_global))
def foo():
pass
help(foo)