I have a decorator which simply caches return values (called #cached in my example) and I wish to use it in conjunction with #property. This works just fine normally. The problem I am facing occurs when I try and use an expire attribute added by #cached.
def cached(f):
cache = [None]
def inner(*args, **kwargs):
if cache[0]:
cache[0] = f(*args, **kwargs)
return cache[0]
def expire():
cache[0] = None
inner.expire = expire
return inner
class Example(object):
#property
#cached
def something_expensive(self):
print("expensive")
return "hello"
e = Example()
e.something_expensive
e.something_expensive.expire()
How am I able to get access to the expire function? added to the function after its replaced by #property. I understand why this doesn't work I am interested in a way of working around that problem.
Some restrictions:
I cannot change the #cached decorator its in a library I don't control
I would really rather not remove #property because I want to expire in my unit tests and they make my code much nicer to use.
One solution that I think is rather bad is (because In reality I have a lot of properties that I want to do this for):
class Example(object):
#cached
def _something_expensive(self):
return "hello"
#property
def something_expensive(self):
return self._something_expensive()
You can access it using the class dictionary:
type(e).__dict__['something_expensive'].fget.expire()
In general e.something_expensive is equivalent to:
type(e).__dict__['something_expensive'].__get__(e, type(e))
For more details read up: Descriptor HowTo Guide
Note that inside the expiry function you're not setting cache from the outer function cached function as None, you're simply creating a new local variable. You may want to do something like this:
def expire():
del cache[:]
cache.append(None)
In Python 3 it's even easier to update cache using the nonlocal keyword.
Related
I need to create/declare 50 functions inside my Class. But, I need to do it dynamically with custom function names and cutom urls in the function body. Something as the following:
for item_nr in range(1, 51):
#task(1)
def view_item_with_id_{item_nr}(self, item_nr=item_nr):
self.client.get(
url=f"/my-url/id=SEPT24_00{item_nr}",
verify=False,
auth=(os.environ['USERNAME'], os.environ['PASSWORD'])
)
P.S since it's inside a class- I cannot really use another function to generate it as suggested in some other threads, because the 'self' parameter will not be visible then. Example (this will not work):
def function_builder(args):
def function(more_args):
#do stuff based on the values of args
return function
Any help is appreciated
This is an interesting question, but is there some reason you cant just use a single #task with a loop? (this solution is of course specific to Locust)
#task
def view_item_with_id(self):
for item_nr in range(1, 51):
self.client.get(
url=f"/my-url/id=SEPT24_00{item_nr}",
verify=False,
auth=(os.environ['USERNAME'], os.environ['PASSWORD'])
)
The other suggested answers might be closer to your original question, but this is so much simpler (it can of course easily be adjusted to pick items randomly, if that is your preference)
You could use setattr to define new class methods:
class Test:
def __init__(self):
for item_nr in range(1, 51):
setattr(self, f"view_item_with_id_{item_nr}", self.itemViewer)
#task(1)
def itemViewer(self, item_nr=None):
"""
self.client.get(
url=f"/my-url/id=SEPT24_00{item_nr}",
verify=False,
auth=(os.environ['USERNAME'], os.environ['PASSWORD'])
)
"""
print(f"/my-url/id=SEPT24_00{item_nr}")
t = Test()
t.view_item_with_id_14(14)
t.view_item_with_id_44(44)
Out:
/my-url/id=SEPT24_0014
/my-url/id=SEPT24_0044
I have to mention: Fundamentally, wanting to do this in the first place is almost certainly a mistake. You should surely, instead, modify the part of the code that calls these functions to just call one function and pass in a parameter, or otherwise work around the problem.
There are two parts to this question: how to generate the functions, and how to wrap them in decorators.
Create methods programmatically
To create methods programmatically, you need to use setattr(). This allows you to add attributes to an object specified by string rather than explicitly in your code. However, for a function added in this way to be treated as a method (i.e. your object is automatically passed as the self parameter), it must set on the class, not directly on the object.
class Foo:
pass
def f(self, val):
self.x = val
setattr(Foo, "f", f)
def g(self):
print(self.x)
setattr(Foo, "g", g)
obj = Foo()
obj.f(3)
obj.g() # prints 3
Call decorator manually
The second part is the easy bit. A decorator is just some syntactic sugar on a function that receives and returns another function. So these are equivalent:
#task(1)
def foo():
pass
def bar():
pass
bar = task(1)(bar)
In your case, you can simply call the decorator directly rather than having to use anything like the #task(1) notation.
Putting it together
Putting the two ideas together, here's what you want. (By the way, your default parameter technique is fine for creating the function, but functools.partialmethod is a bit cleaner because the result doesn't allow the caller to override the parameter, so I've used that below.)
def MyClass
pass # Whatever else you need
def fn(self, item_nr):
self.client.get(
url=f"/my-url/id=SEPT24_00{item_nr}",
verify=False,
auth=(os.environ['USERNAME'], os.environ['PASSWORD'])
)
for item_nr in range(1, 51):
this_fn = functools.partialmethod(fn, item_nr)
decorated_fn = task(1)(this_fn)
setattr(MyClass, f"view_item_with_id_{item_nr}", decorated_fn)
I have a function foo that takes a parameter stuff
Stuff can be something in a database and I'd like to create a function that takes a stuff_id, get the stuff from the db, execute foo.
Here's my attempt to solve it:
1/ Create a second function with suffix from_stuff_id
def foo(stuff):
do something
def foo_from_stuff_id(stuff_id):
stuff = get_stuff(stuff_id)
foo(stuff)
2/ Modify the first function
def foo(stuff=None, stuff_id=None):
if stuff_id:
stuff = get_stuff(stuff_id)
do something
I don't like both ways.
What's the most pythonic way to do it ?
Assuming foo is the main component of your application, your first way. Each function should have a different purpose. The moment you combine multiple purposes into a single function, you can easily get lost in long streams of code.
If, however, some other function can also provide stuff, then go with the second.
The only thing I would add is make sure you add docstrings (PEP-257) to each function to explain in words the role of the function. If necessary, you can also add comments to your code.
I'm not a big fan of type overloading in Python, but this is one of the cases where I might go for it if there's really a need:
def foo(stuff):
if isinstance(stuff, int):
stuff = get_stuff(stuff)
...
With type annotations it would look like this:
def foo(stuff: Union[int, Stuff]):
if isinstance(stuff, int):
stuff = get_stuff(stuff)
...
It basically depends on how you've defined all these functions. If you're importing get_stuff from another module the second approach is more Pythonic, because from an OOP perspective you create functions for doing one particular purpose and in this case when you've already defined the get_stuff you don't need to call it within another function.
If get_stuff it's not defined in another module, then it depends on whether you are using classes or not. If you're using a class and you want to use all these modules together you can use a method for either accessing or connecting to the data base and use that method within other methods like foo.
Example:
from some module import get_stuff
MyClass:
def __init__(self, *args, **kwargs):
# ...
self.stuff_id = kwargs['stuff_id']
def foo(self):
stuff = get_stuff(self.stuff_id)
# do stuff
Or if the functionality of foo depends on the existence of stuff you can have a global stuff and simply check for its validation :
MyClass:
def __init__(self, *args, **kwargs):
# ...
_stuff_id = kwargs['stuff_id']
self.stuff = get_stuff(_stuff_id) # can return None
def foo(self):
if self.stuff:
# do stuff
else:
# do other stuff
Or another neat design pattern for such situations might be using a dispatcher function (or method in class) that delegates the execution to different functions based on the state of stuff.
def delegator(stff, stuff_id):
if stuff: # or other condition
foo(stuff)
else:
get_stuff(stuff_id)
I have some class with a field spent_times. spent_times is a list and all methods of this class write some information, which is valuable for logging.
Also, I have a decorator, which calculate execution time for every function and write it to spent_times.
This is realization of my decorator:
def timing(message):
def wrap(function):
def called(*args, **kwargs):
time_start = timer()
spent_time = round(timer() - time_start, 5)
if not args:
return function(*args, **kwargs), spent_time
obj = args[0]
if hasattr(obj, "spent_times"):
obj.spent_times.append("{}={:.5f}".format(message, spent_time))
return function(*args, **kwargs)
else:
logging.warning('Decorator allows to set spent_time attribute!')
return called
return wrap
As you can see in my decorator there is a check, if the calling function has attribute self.
If it has, than I can write needed info in list spent_times on the spot, if it does not have, than decorator returns time spent on execution and function itself.
I am using this decorator in one single module and second case (when no self found) belongs to some other functions in this module, which does not belong to class, where spent_time list is defined, but I execute them inside my class, so I am able to realize for example the following structure:
This is declaration of "outer" function
def calc_users(requests, priority):
# .....
And inside my class I execute it and update my spent_time list this way:
response, spent_time = calc_users(requests, priority)
self.class_obj.spent_times.append("user_calculation={:.5f}".format(spent_time))
which is not very nice, but it is working at least.
Now, I moved a few functions of my class in different new module and I would like to use the same decorator timing.
Can someone help me to implement this realization of timing in new module. I do not know, what can I do to update my spent_times list now.
These two modules will work at the same time and I cannot create object of class and pass it as an argument to new module, because (as far as I understand it) there will be two objects and spent_times will not be updated correctly.
Maybe there is a way to pass a reference to spent_times somehow, but I do not want to change arguments of my functions in new module, since I think in this case principle of shared responsibility will be broken (decorator is responsible for logging, function for its action).
So how to improve decorator or how to pass spent_times list to a new module?
Any help will be greatly appreciate!
P.S.
Maybe make spent_times a global variable? (in the very worst case)
A global list seems fine but you can also use a class and create a singleton by deleting the class after instantiation. This prevents from creating another another instance:
# mymodule.py
from timeit import default_timer as timer
class Timing(object):
def __init__(self):
self.spent_times = []
def __call__(self, message):
def wrap(function):
def called(*args, **kwargs):
time_start = timer()
spent_time = round(timer() - time_start, 5)
self.spent_times.append("{}={:.5f}".format(message, spent_time))
return function(*args, **kwargs)
return called
return wrap
timing = Timing()
del Timing # prevent another instance
Now import in another module:
from mymodule import timing
#timing('Hello')
def add(a, b):
return a + b
The special method __call__ makes an instance of a class behave like a function, i.e. it is callable with ().
The advantage it that you can use self.attr instead of a global variable.
The deletion of the class after instantiation prevents from creating another instance. This is called a singleton. Now all your timings end up in the same list no matter how often you use timing as a decorator.
I am trying to make a class that will make api requests, retrying based on configuration options passed in to the retrying.retry decorator, and handle different error codes in the correct way for each job.
Here is my code:
from retrying import retry
class APIRequester:
def __init__(self, url, **kwargs):
self.url = url
self.retry_kwargs = kwargs
#retry(**self.retry_kwargs) # Obviously doesn't know what self is
def make_request(self):
pass
How can I pass in parameters to this method decorator? I tried making them a class attribute, which didn't work either.
A couple of notes/questions:
The #retry decorator will be applied to the make_request method at the time the class is created, while retry_kwargs will only become available when an instance of the class is created, and thus the former must precede the latter.
In which case, the former cannot depend on information that becomes available in the latter, ... as long as you use the decorator syntax ...
The decorator syntax
#decorator
def xxx(...):
...
is just syntax sugar for
def xxx(...):
...
xxx = decorate(xxx)
which means that, along with the fact that Python is very dynamic, you could force the issue by doing something like
class APIRequester:
def __init__(self, url, **kwargs):
self.url = url
self.retry_kwargs = kwargs
APIRequester.make_request = retry(**kwargs)(APIRequester.make_request)
def make_request(self):
pass
Whether this particular decorator chokes on the self parameter or not, I cannot tell you.
Will you have more than one instance of APIRequester? If so, note that the method will be re-decorated each time a new instance is created: can this work sensibly? (I doubt it.) But see the edit below ...
If you do not have more that one instance, then you probably don't need to rely on information that becomes availale at the singleton's construction time.
The above were some general Python principles. I doubt that you really want to force the issue in this case. It seems to me that you are trying to use the decorator in a way that it was not designed to be used.
Edit: instancemethods
If you replace the line that does the decorating in the constructor with
self.make_request = retry(**kwargs)(self.make_request)
then each instance will get its own decorated version of the function. This should avoid any problems with re-decoration of the same function. There may will still be problems with self getting in the way. In that case, you could remove the self parameter from the definition and wrap it with staticmethod:
self.make_request = retry(**kwargs)(staticmethod(self.make_request))
Or better still, use decorator syntax to apply staticmethod to make_request at the place where you define it, the way Guido inteded it.
Like this, it even stands a chance of working! :-)
Decorator is just a syntax sugar for func=decorator(func). You can do the assignment yourself:
class APIRequester:
def __init__(self, url, **kwargs):
self.url = url
self.make_request = retry(**kwargs)(self.make_request)
def make_request(self):
pass
This will internally replace a method (descriptor) by a function, but it will work as expected.
Of course self is available in the decorator at the call time. See answers to How to decorate a method inside a class? , on which I based my answer here:
def my_retry(fn):
from functools import wraps
#wraps(fn)
def wrapped(self):
print(self.retry_kwargs)
for i in range(self.retry_kwargs["times"]):
# you have total control
fn(self)
# around your method. Can even call it multiple times,
# call with original retry:
retry(**self.retry_kwargs)(fn)(self)
# check exceptions, return some value (here None), etc
#
return wrapped
class APIRequester(object):
def __init__(self, url, **kwargs):
self.url = url
self.retry_kwargs = kwargs
#my_retry
def make_request(self):
print("method")
a = APIRequester('http://something', times=3)
a.make_request()
That is, original decorator is wrapped with a new, configuration-aware decorator. No need to change the constructor, syntax remains simple.
Retry decorator doesn't support class method, because instance of the class is implicitly passed to function.
Please decorate normal function.
If you want to wrap function into class, please decorate static method.
Im trying to create a class with some formatting options. But i can't figure out how to do it properly...
The code produced the following error:
AttributeError: 'NoneType' object has no attribute 'valuesonly'
class Testings(object):
def format_as_values_only(self,somedata):
buildstring=somedata.values()
return buildstring
def format_as_keys_only(self):
pass
def format_as_numbers(self):
pass
def get_data_method(self):
self.data= {'2_testkey':'2_testvalue',"2_testkey2":"2_testvalue2"}
#property
def valuesonly(self):
return format_as_values_only(self.data)
test=Testings()
print test.get_data_method().valuesonly
The important thing for me is to be able to get the formatters like: class.method.formatter or so...
Thanks a lot for any hints!
get_data_method has no return value, so the result of test.get_data_method() is None. That's why you're getting that exception.
If you really want to do something like test.get_data_method().valuesonly, either define the valuesonly property on Testings, and have get_data_method return self, or have get_data_method return some new object with the properties that you want defined.
You can't do things this way. Methods are just functions defined directly inside a class block. Your function is inside another function, so it's not a method. The property decorator is useless except in a class block.
But, more fundamentally, function definitions just create local names, the same as variable assignments or anything else. Your valuesonly function is not accessible at all from outside the get_data_method function, because nothing from within a function is accessible except its return value. What you have done is no different than:
def get_data_method(self):
a = 2
. . . and then expecting to be able to access the local variable a from outside the function. It won't work. When you call get_data_method(), you get the value None, because get_data_method doesn't return anything. Anything you subsequently do with the result of get_data_method() is just operating on that same None value.
If you want to access things using the syntax you describe, you will need to make get_data_method return an object that has properties like valuesonly. In other words, write another class that provides a valuesonly property, and have get_data_method return an instance of that class. A rough outline (untested):
class DataMethodGetter(object):
def __init__(self, parent):
self.parent = parent
#property
def valuesonly(self):
return format_as_values_only(self.parent.data)
class Testings(object):
# rest of class def here
def get_data_method(self):
self.data = {'blah': 'blah'}
return DataMethodGetter(self)
However, you should think about why you want to do this. It's likely to be simpler to set it up to just call valuesonly directly on the Testing object, or to pass a flag to get_data_method, doing something like get_data_method(valuesonly=True).