I am writing a Python app which will use a config file, so I am delegating the control of the config file to a dedicated module, configmanager, and within it a class, ConfigManager.
Whenever a method within ConfigManager is run, which will change my config file in some way, I will need to get the latest version of the file from the disk. Of course, in the spirit of DRY, I should delegate the opening of the config file to it's own function.
However, I feel as though explicitly calling a method to get and return the config file in each function that edits it is not very "clean".
Is there a recommended way in Python to run a method, and make a value available to other methods in a class, whenever and before a method is run in that class?
In other words:
I create ConfigManager.edit_config().
Whenever ConfigManager.edit_config() is called, another function ConfigManager.get_config_file() is run.
ConfigManager.get_config_file() makes a value available to the method ConfigManager.edit_config().
And ConfigManager.edit_config() now runs, having access to the value given by ConfigManager.get_config_file().
I expect to have many versions of edit_config() methods in ConfigManager, hence the desire to DRY my code.
Is there a recommended way of accomplishing something like this? Or should I just create a function to get the config fine, and manually call it each time?
The natural way to have:
ConfigManager.get_config_file() makes a value available to the method
ConfigManager.edit_config().
is to have get_config_file() return that value.
Just call get_config_file() within edit_config().
If there are going to be many versions of edit_config(), then a decorator might be the way to go:
def config_editor(func):
def wrapped(self, *args, **kwargs):
config_file = self.get_config_file()
func(self, config_file, *args, **kwargs)
return func
class ConfigManager
.
.
.
#config_editor
def edit_config1(self, config_file, arg1):
...
#config_editor
def edit_config2(self, config_file, arg1, arg2):
...
ConfigManager mgr
mgr.edit_config1(arg1)
I don't actually like this:
Firstly, the declaration of edit_config1 takes one more argument than the actual usage needs (because the decorator supplies the additional argument).
Secondly, it doesn't actually save all that much boiler plate over:
def edit_config3(self, arg1):
config_file = self.get_config_file()
In conclusion, I don't think the decorators save enough repetition to be worth it.
Since you get something from disk, you open a file. So, you could use the class with the with "function" of python.
You should check the context managers. With that, you will be able to implement the functionality that you want each time that someone access the config file through the __enter__ method and (if it is needed) implement the functionality for stop using the resource with the __exit__ method.
Related
I have a Python class that requires some data in order to be initialized. This data is usually obtained using a function from another module, which makes calls to an API. One of the parameters my class' initializer takes is the same ID that can be used to obtain the resource with the API.
Calling the API from inside the initializer, and obtaining the data it needs would make for shorter (and cleaner?) initialization. But I am concerned this could make the class harder to test, and introduce a dependency deep inside the code.
I'm trying to devise the best way to implement this in a maintainable and testable way.
Would it be bad to call the API module directly from within the initializer, and obtain the data it needs to complete initialization? Or is it better to just call the API from outside and pass the data to the initializer?
The "normal" way(1) is the pass the dependent function, module, or class, into the constructor itself.
Then, in your production code, pass in the real thing. In your test code, pass in a dummy one that will behave exactly as you desire for the specific test case.
That's actually a half-way measure between the two things you posit.
In other words, something like:
def do_something_with(something_generator):
something = something_generator.get()
print(something)
# Real code.
do_something_with(ProductionGenerator())
# Test code.
class TestGenerator:
def get(self):
return 42
do_something_with(TestGenerator())
If you're reticent to always pass in a dependency, you can get around that with something like a default value and creating it inside the function if not given:
def do_something(something_generator=None):
if something_generator is None:
local_gen = ProductionGenerator()
else:
local_gen = something_generator
something = something_generator.get()
print(something)
# Real code.
do_something()
# Test code.
class TestGenerator:
def get(self):
return 42
do_something(TestGenerator())
(1) Defined, of course, as the way I do it :-)
I am writing a package for interacting with dataset and have code that looks something like
from abc import ABC, ABCMeta, abstractmethod
from functools import cache
from pathlib import Path
from warnings import warn
class DatasetMetaClass(ABCMeta):
r"""Meta Class for Datasets"""
#property
#cache
def metaclass_property(cls):
r"""Compute an expensive property (for example: dataset statistics)."""
warn("Caching metaclass property...")
return "result"
# def __dir__(cls):
# return list(super().__dir__()) + ['metaclass_property']
class DatasetBaseClass(metaclass=DatasetMetaClass):
r"""Base Class for datasets that all datasets must subclass"""
#classmethod
#property
#cache
def baseclass_property(cls):
r"""Compute an expensive property (for example: dataset statistics)."""
warn("Caching baseclass property...")
return "result"
class DatasetExampleClass(DatasetBaseClass, metaclass=DatasetMetaClass):
r"""Some Dataset Example."""
Now, the problem is that during make html, sphinx actually executes the baseclass_property which is a really expensive operation. (Among other things: checks if dataset exists locally, if not, downloads it, preprocesses it, computes dataset statistics, mows the lawn and takes out the trash.)
I noticed that this does not happen if I make it a MetaClass property, because the meta-class property does not appear in the classes __dir__ call which may or may not be a bug. Manually adding it to __dir__ by uncommenting the two lines causes sphinx to also process the metaclass property.
Questions:
Is this a bug in Sphinx? Given that #properties are usually handled fine, it seems unintended that it breaks for #classmethod#property.
What is the best option - at the moment - to avoid this problem? Can I somehow tell Sphinx to not parse this function? I hope that there is a possibility to either disable sphinx for a function via comment similarly to # noqa, # type: ignore, # pylint disable= etc. or via some kind of #nodoc decorator.
Everything is working as it should, and there is no "bug" there either in Sphinx, nor in the ABC machinery, and even less in the language.
Sphinx uses th language introspection capabilities to retrieve a class's members and then introspect then for methods. What happens when you combine #classmethod and #property is that, besides it somewhat as a nice surprise actually work, when the class member thus created is accessed by Sphynx, as it must do in search for the doc strings, the code is triggered and runs.
It would actually be less surprising if property and classmethod could not be used in combination actually since both property and classmethod decorators use the descriptor protocol to create a new object with the appropriate methods for the feature they implement.
I think the less surprising thing to go there is to put some explicit guard inside your "classmethod property cache" functions to not run when the file is being processed by sphinx. Since sphinx do not have this feature itself, you can use an environment variable for that, say GENERATING_DOCS. (this does not exist, it can be any name), and then a guard inside your methods like:
...
def baseclass_property(self):
if os.environ.get("GENERATING_DOCS", False):
return
And then you either set this variable manually before running the script, or set it inside Sphinx' conf.py file itself.
If you have several such methods, and don't want to write the guard code in all of them, you could do a decorator, and while at that, just use the same decorator to apply the other 3 decorators at once:
from functools import cache, wraps
import os
def cachedclassproperty(func):
#wraps(func)
def wrapper(*args, **kwargs):
if os.environ.get("GENERATING_DOCS", False):
return
return func(*args, **kwargs)
return classmethod(property(cache(wrapper)))
Now, as for using the property on the metaclass: I advise against it. Metaclasses are for when you really need to customize your class creation process, and it is almost by chance that property on a metaclass works as a class property as well. All that happens in this case, as ou have investigated, is that the property will be hidden from a class' dir, and therefore won't be hit by Sphinx introspection - but even if you are using a metaclass for some other purpose, if you simply add a guard as I had suggested might even not prevent sphinx from properly documenting the class property, if it has a docstring. If you hide it from Sphinx, it will obviously go undocumented.
I was looking into the following code.
On many occasions the __init__ method is not really used but there is a custom initialize function like in the following example:
def __init__(self):
pass
def initialize(self, opt):
# ...
This is then called as:
data_loader = CustomDatasetDataLoader()
# other instance method is called
data_loader.initialize(opt)
I see the problem that variables, that are used in other instance methods, could still be undefined, if one forgets to call this custom initialize function. But what are the benefits of this approach?
Some APIs out in the wild (such as inside setuptools) have similar kind of thing and they use it to their advantage. The __init__ call could be used for the low level internal API while public constructors are defined as classmethods for the different ways that one might construct objects. For instance, in pkg_resources.EntryPoint, the way to create instances of this class is to make use of the parse classmethod. A similar way can be followed if a custom initialization is desired
class CustomDatasetDataLoader(object):
#classmethod
def create(cls):
"""standard creation"""
return cls()
#classmethod
def create_with_initialization(cls, opt):
"""create with special options."""
inst = cls()
# assign things from opt to cls, like
# inst.some_update_method(opt.something)
# inst.attr = opt.some_attr
return inst
This way users of the class will not need two lines of code to do what a single line could do, they can just simply call CustomDatasetDataLoader.create_with_initialization(some_obj) if that is what they want, or call the other classmethod to construct an instance of this class.
Edit: I see, you had an example linked (wish underlining links didn't go out of fashion) - that particular usage and implementation I feel is a poor way, when a classmethod (or just rely on the standard __init__) would be sufficient.
However, if that initialize function were to be an interface with some other system that receives an object of a particular type to invoke some method with it (e.g. something akin to the visitor pattern) it might make sense, but as it is it really doesn't.
I'm looking to add a decorator that runs a video recorder on certain tests like so:
#decorators.video(self)
def test_1234(self):
...
I'm having trouble passing the self variable into the decorator as it is need for some attributes. How can I do this?
theodox answer is generally good, but for decorators you should use functools.wraps function, like in an example below:
from functools import wraps
def enable_video(fn)
'''Decorate the function to start video, call the function, stop video.'''
#wraps(fn)
def inner(*args, **kwargs):
# could be just `def inner(self):` if only intended to use
# with methods without arguments and keyword arguments
do_stuff_before()
fn(*args, **kwargs)
do_stuff_after()
return inner
It will persist original docstrings, original function name (and more). You can read more about it in Python docs.
Then, assuming that previous code is in decorators module, you should use it as follows:
class MyTestCase(unittests.TestCase);
#decorators.enable_video
def testSomeVideoFunction(self):
do_test_stuff()
Note that in the code example it's just #decorators.enable_video, not #decorators.enable_video(self). As like in jonrsharpe's comment to your question, reference to a self is not present at a decoration time.
Are you sure you need the self reference at all?
More commonly you'd do something like this
def enable_video(fn):
'''decorate the test function so it starts the video, runs the test, and stop the video'''
def video_aware_test(self_refrence):
start_video_recorder()
try:
fn()
finally:
stop_video_recorder()
return video_aware_test
And you'd apply it like this:
#enable_video
def test_something(self)
If for some reason the decorator actually needed the self reference, you can see where you'd grab it. This version doesn't include configuring the video recorder in any way, to that you'd use a class rather than a function decorator and pass the configuration as arguments.
Is there a reasonably natural way of converting python function to standalone scripts? Something like:
def f():
# some long and involved computation
script = function_to_script(f) # now script is some sort of closure,
# which can be run in a separate process
# or even shipped over the network to a
# different host
and NOT like:
script = open("script.py", "wt")
script.write("#!/usr/bin/env python")
...
You can turn any "object" into a function by defining the __call__ method on it (see here.) Hence, if you want to compartmentalize some state with the computations, as long as what you've provided from the very top to the bottom of a class can be pickled, then that object can be pickled.
class MyPickledFunction(object):
def __init__(self, *state):
self.__state = state
def __call__(self, *args, **kwargs):
#stuff in here
That's the easy cheater way. Why pickling? Anything that can be pickled can be sent to another process without fear. You're forming a "poor man's closure" by using an object like this.
(There's a nice post about the "marshal" library here on SO if you want to truly pickle a function.)