In the print_foo function, there is a part of the code that changes from project to project.
And there is a part of the code which is always the same. It is responsible for working with the Counter class.
What's the best way to change the code to just move the unchanged part of the function from project to project?
Is it possible to implement this through the decorator? If so, how can the decorator query the attributes of an object, in this case foobar. And how would it save them to an object of the Counter class?
class Counter(object):
def __init__(self):
self.foo = 0
self.status = 'Off'
def counter_foo(self, n):
self.foo += n
def set_status(self, status):
self.status = status
class First(object):
def __init__(self):
self.counter = Counter()
self.status = 'On'
def print_foo(self, n=1):
# This part is changing in every project
print('foo' * n)
# This part doesn't change. And this part I want take out separately
self.counter.counter_foo(n)
if self.status == 'On':
self.counter.set_status(self.status)
def main():
foobar = First()
foobar.print_foo(8)
I want in the First, just write the function
def print_foo(self, n=1):
print('foo' * n)
But with the preservation of the functionality of this part
self.counter.counter_foo(n)
if self.status == 'On':
self.counter.set_status(self.status)
In a real project, it is 6 functions and tens of lines. I can't copy them every time by hand.
You could declare a decorator as follows and still use it for your scope since the self name is just a name at the end...
from functools import wraps
def increment_counter(func):
#wraps(func)
def wrapper_func(self, *args, **kwargs):
result = func(self, *args, **kwargs)
self.counter.counter_foo(n)
if self.status == 'On':
self.counter.set_status(self.status)
return result
return wrapper_func
You can then decorate print_foo with it as:
class First:
#increment_counter
def print_foo(self, n=1):
# This part is changing in every project
print('foo' * n)
Related
I have a number of classes that implement smoothing techniques on a series of prices.
I am trying to figure the best way of implementing any of these smoothing classes within the __call__ function of another class, where some operation is then performed on the smoothed series:
i.e.
class NoSmoother:
def __init__(self, prices):
self.prices = prices
def __call__(self):
return self.prices
class MASmoother:
def __init__(self, prices):
self.prices = prices
def __call__(self, window):
return self.prices.rolling(window).mean().to_frame("price")
class ThatDoesSomethingWithSmoothedPrices():
def __init__(self, prices):
self.prices = prices
def __call__(self, smoother=ma_smoother, window=3)
smoothed_prices = SomeBuilderClassThatCallsTheCorrectSmootherClass()
As you can see, I would need the factory/builder class to implement NoSmoother if say smoother = None, otherwise, it would call the appropriate smoothing class. Of course, the returned object can vary from simple to complex, for example, if I smooth the prices using a Kalman Filter, the class can expect many more parameters.
Currently, my code instantiates class ThatDoesSomethingWithSmoothedPrices() with a price series, then calls by passing a **config.
Desired Output:
Ideally, I would like to be able to call any smoothing class from within the call function of
class ThatDoesSomethingWithSmoothedPrices().
Example implementation:
configs = {'smoother': MASmoother, 'window': 3}
processor = ThatDoesSomethingWithSmoothedPrices(prices)
output = processor(**config)
My attempt:
class Smoother:
def __init__(self, prices):
self.prices = prices
def __call__(self, smoother, *args, **kwargs):
return partial(smoother, **kwargs)
def ma_smoother(self, window: int = 3):
return self.prices.rolling(window).mean().to_frame("price")
def no_smoother(self):
return self.prices
class ThatDoesSomethingWithSmoothedPrices:
def __init__(self, prices):
self.prices = prices
def __call__(self, smooth_method = 'no_smoother'):
smoother = Smoother(prices)
prices_smoothed = smoother(**configs)
# do other things
if __name__ == '__main__':
configs = {'smoother': 'ma_smoother', window=3}
label = ThatDoesSomethingWithSmoothedPrices(**configs)
Any help greatly appreciated.
For simplicity, if you don't have a lot of state, I'd just use regular functions.
You can use functools.partial() to partially apply a function, i.e. in this case set the MA window:
from functools import partial
def no_smoother(values):
return values
def ma_smoother(values, *, window):
return values.rolling(window).mean().to_frame("price")
def get_prices():
...
def get_smoothed_prices(smoother):
prices = get_prices()
return smoother(prices)
get_smoothed_prices(smoother=no_smoother)
get_smoothed_prices(smoother=partial(ma_smoother, window=3))
EDIT
Based on the edit in the question:
configs = {'smoother': MASmoother, 'window': 3}
processor = ThatDoesSomethingWithSmoothedPrices(prices)
output = processor(**config)
would be expressed as something like
def construct_smoother(smoother, **kwargs):
return partial(smoother, **kwargs)
smoother = construct_smoother(**configs)
# ...
Given your latest update you need to pass in the configs as well:
configs = {'smoother': MASmoother, 'window': 3}
processor = ThatDoesSomethingWithSmoothedPrices(configs, prices)
output = processor(**config)
So that means ThatDoesSomethingWithSmoothedPrices could be like this:
def ThatDoesSomethingWithSmoothedPrices(configs, prices):
smoother = configs['smoother'](prices)
return smoother
To create a factory, you can create a class method inside a class.
# declaration
class Factory:
# class variables (static)
id = 0
#classmethod
def new(cls):
return object()
# usage
obj = Factory.new()
If you have a class that needs arguments in the constructor, then you can pass variable number of arguments. Basically * removes brackets around the list or dict you are passing.
# declaration
def function(var1, var2, *args, **kwargs):
# transfer variable arguments to other function
return SomeObejct(*args, **kwargs)
# usage
obj = function(1, 2, a, b, c, key=value)
In your case, you would do something like this:
# declaration
# you can also pass classes as arguments
def __call__(self, smoother=MASmoother, window=3, *args, **kwargs)
smoothed_prices = smoother(*args, **kwargs)
return smoothed_prices
# usage
smth = ThatDoesSomethingWithSmoothedPrices()
smoother1 = smth()
smoother2 = smth(NoSmoother, 2)
smoother3 = smth(NoSmoother, 2, arg1, arg2, key=value)
def memoize(fn):
memory=dict()
def inner(inpt):
if not inpt in memory:
memory[inpt] = fn(inpt)
return memory[inpt]
return inner
#memoize
def expected_rounds(picks_to_go):
#algorithm....
ans += expected_rounds(new_picks_to_go) #some recursion
#algorithm....
return ans
How to access (and even print) variable memory in memoize decorator after function will successfully end execution and return answer?
my idea that I created after posting question it uses class but I would like to stay with function decorator:
#class callable that can be used as decorator
class Memoize:
def __init__(self):
self.memory = {}
# self.fn = None
def set_fn(self, fn):
self.fn = fn
return self # this line is important
def __call__(self, *args):
if args not in self.memo:
self.memory[args] = self.fn(*args)
return self.memory[args]
memo1 = Memoize()
#memo1.set_fn
def expected_rounds(picks_to_go):
#something
return ans
expected_rounds(some_input)
memo1.memory # <- access to memory possible
I have to model a warrior and the different kinds of attacks he can perform. The idea is to use mixins to contain the attack logic. I have my classes defined in the following way:
class Warrior:
def __init__(self, energy):
self.energy = energy
class TemplarKnight(Warrior, HandToHandCombatMixin):
pass
class CombatMixin:
def __init__(self):
self.attacks_cost = {}
def attack(self, attacker, attack_cost):
if attacker.energy < attack_cost:
print('Not enough energy to attack')
else:
attacker.energy -= attack_cost
print('Attack!')
class HandToHandCombatMixin(CombatMixin):
def __init__(self):
super().__init__()
self.attacks_cost['sword_spin'] = 10
def sword_spin(self, attacker):
return self.attack(attacker, self.attacks_cost['sword_spin'])
But the problem comes when I try to test this setup. When I do
class TestTemplarKnight(unittest.TestCase):
def setUp(self):
self.templar = TemplarKnight(energy=100)
def test_templar_knight_can_sword_spin(self):
self.templar.sword_spin(self.warrior)
self.assertEquals(self.templar.energy, 90)
I get
def sword_spin(self, attacker):
return self.attack(
> attacker, self.attacks_cost['sword_spin'])
E AttributeError: 'TemplarKnight' object has no attribute 'attacks_cost'
It seems that Python thinks that the parameter self.attacks_cost (when calling self.attack() inside the sword_spin() method of the HandToHandCombatMixin class) belongs to the TemplarKnight class instead of the HandToHandCombatMixin.
How should I have written this code to make Python look for self.attacks_cost inside HandToHandCombatMixin?
To use super correctly, all the classes involved need to use it. Right now, Warrior.__init__ is called first, but it doesn't use super, so HandToHandCombatMixin.__init__ is never called.
Make the following additions:
class Warrior:
def __init__(self, energy, **kwargs):
super().__init__(**kwargs)
self.energy = energy
class TemplarKnight(Warrior, HandToHandCombatMixin):
pass
class CombatMixin:
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.attacks_cost = {}
def attack(self, attacker, attack_cost):
if attacker.energy < attack_cost:
print('Not enough energy to attack')
else:
attacker.energy -= attack_cost
print('Attack!')
class HandToHandCombatMixin(CombatMixin):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.attacks_cost['sword_spin'] = 10
def sword_spin(self, attacker):
return self.attack(attacker, self.attacks_cost['sword_spin'])
Now when you instantiate TemplarKnight, you'll guarantee that all the __init__ methods are called, and in the correct order. Eventually, once of the calls to super() will cause object.__init__ to be called, at which point the chain finally ends. If you are correctly handling the keyword arguments, **kwargs will be empty by the time that happens.
I have been trying to understand better the decorators and closures.
I am trying to decorate the function to achieve:
remembering previously passed values,
counting how many times the function was called.
I want to make it using two separate decorators - for science :)
So I managed to create this working code (I used some snippet for the counting - I admit)
class countcalls(object):
"Decorator that keeps track of the number of times a function is called."
__instances = {}
def __init__(self, f):
self.__f = f
self.__numcalls = 0
countcalls.__instances[f] = self
def __call__(self, *args, **kwargs):
self.__numcalls += 1
return self.__f(*args, **kwargs)
def count(self):
"Return the number of times the function f was called."
return countcalls.__instances[self.__f].__numcalls
#staticmethod
def counts():
"Return a dict of {function: # of calls} for all registered functions."
return dict([(f.__name__, countcalls.__instances[f].__numcalls) for f in countcalls.__instances])
def wrapper(x):
past=[]
#countcalls
def inner(y):
print x
print inner.count()
past.append(y)
print past
return inner
def main():
foo = wrapper("some constant")
foo(5)
foo("something")
if __name__ == '__main__':
main()
output:
some constant
1
[5]
some constant
2
[5, 'something']
Now I want to change the memoize function to a neat pythonic decorator. Here is what I came out with so far:
class countcalls(object):
"Decorator that keeps track of the number of times a function is called."
__instances = {}
def __init__(self, f):
self.__f = f
self.__numcalls = 0
countcalls.__instances[f] = self
def __call__(self, *args, **kwargs):
self.__numcalls += 1
return self.__f(*args, **kwargs)
def count(self):
"Return the number of times the function f was called."
return countcalls.__instances[self.__f].__numcalls
#staticmethod
def counts():
"Return a dict of {function: # of calls} for all registered functions."
return dict([(f.__name__, countcalls.__instances[f].__numcalls) for f in countcalls.__instances])
class memoize(object):
past=[]
def __init__(self, f):
past = []
self.__f = f
def __call__(self, *args, **kwargs):
self.past.append(*args)
return self.__f(*args, **kwargs)
def showPast(self):
print self.past
#memoize
#countcalls
def dosth(url):
print dosth._memoize__f.count() ## <-- this is so UGLY
dosth.showPast()
def main():
dosth("one")
dosth("two")
if __name__ == '__main__':
main()
And here is the output:
1
['one']
2
['one', 'two']
How to get rid of the "ugly" line ( print dosth._memoize__f.count() ) ? In other words - how can I directly call the methods of the stacked decorators? (without adding a method to the decorators to call the methods of other decorators - that is not my point)
You'll still have to unwrap the decorators if you want to access specific decorator results, but the 'default' attribute for this is __wrapped__. The Python 3 version of the #functools.wraps() decorator sets this attribute for you (via the functools.update_wrapper() utility function, starting with Python 3.4), but you can do the same manually in your own decorators:
class memoize(object):
past=[]
def __init__(self, f):
past = []
self.__wrapped__ = self.__f = f
Now you can access the wrapped callable with:
dosth.__wrapper__.count()
wich is the standard Pythonic way of unwrapping decorators.
I have created a decorator memoization class that I am actively using for cache my calls. There are already many excellent suggestions on how to implement python memoization.
The class that I have created currently uses get and set method calls to set the cacheTimeOut. They are called getCacheTimeOut() and setCacheTimeOut(). While this is an adequate solution. I was hoping to use the #property and #cacheTimeOut.setter decorators to enable the functions to be called directly as for example cacheTimeOut=120
The problem is in the details. I do not know how to make these properties accessible in the __get__ method. The __get__ method assigns the different function calls defined within the class to functions.partial.
Here is my script example designed for Python 2.7
import time
from functools import partial
import cPickle
class memoize(object):
def __init__(self, func):
self.func = func
self._cache = {}
self._timestamps = {}
self._cacheTimeOut = 120
self.objtype = None
def __new__(cls, *args, **kwargs):
return object.__new__(cls,*args, **kwargs)
def __get__(self, obj, objtype=None):
"""Used for object methods where decorator has been placed before methods."""
self.objtype = objtype
fn = partial(self, obj)
fn.resetCache = self.resetCache
fn.getTimeStamps = self.getTimeStamps
fn.getCache = self.getCache
fn._timestamps = self._timestamps
fn.setCacheTimeOut = self.setCacheTimeOut
fn.getCacheTimeOut = self.getCacheTimeOut
return fn
def __argsToKey(self, *args, **kwargs):
args = list(args)
for x, arg in enumerate(args): # remove instance from
if self.objtype:
if isinstance(arg, self.objtype):
args.remove(arg)
str = cPickle.dumps(args, 1)+cPickle.dumps(kwargs, 1)
return str
def __call__(self, *args, **kwargs):
"""Main calling function of decorator."""
key = self.__argsToKey(*args, **kwargs)
now = time.time() # get current time to query for key
if self._timestamps.get(key, now) > now:
return self._cache[key]
else:
value = self.func(*args, **kwargs)
self._cache[key] = value
self._timestamps[key] = now + self._cacheTimeOut
return value
def __repr__(self):
'''Return the function's docstring.'''
return self.func.__doc__
def resetCache(self):
"""Resets the cache. Currently called manually upon request."""
self._cache = {}
self._timestamps = {}
def getCacheTimeOut(self):
"""Get the cache time out used to track stale data."""
return self._cacheTimeOut
def setCacheTimeOut(self, timeOut):
"""Set the cache timeout to some other value besides 120. Requires an integer value. If you set timeOut to zero you are ignoring the cache"""
self._cacheTimeOut = timeOut
def getCache(self):
"""Returns the cache dictionary."""
return self._cache
def getTimeStamps(self):
"""Returns the encapsulated timestamp dictionary."""
return self._timestamps
#property
def cacheTimeOut(self):
"""Get cacheTimeOut."""
return self._cacheTimeOut
#cacheTimeOut.setter
def cacheTimeOut(self, timeOut):
"""Set cacheTimeOut."""
self._cacheTimeOut = timeOut
memoize
def increment(x):
increment.count+=1
print("increment.count:%d, x:%d"%(increment.count, x))
x+=1
return x
increment.count = 0 # Define the count to track whether calls to increment vs cache
class basic(object):
def __init__(self):
self.count = 0
#memoize
def increment(self, x):
self.count+=1
print("increment.count:%d, x:%d"%(increment.count, x))
x+=1
return x
def main():
print increment(3)
print increment(3)
# What I am actually doing
print increment.getCacheTimeOut() # print out default of 120
increment.setCacheTimeOut(20) # set to 20
print increment.getCacheTimeOut() # verify that is has been set to 120
# What I would like to do and currently does not work
print increment.cacheTimeOut
# Assign to property
increment.cacheTimeOut = 20
myObject = basic()
print myObject.increment(3)
print myObject.count
print myObject.increment(3)
print myObject.count
print myObject.increment(4)
print myObject.count
####### Unittest code.
import sys
import time
import unittest
from memoize import memoize
class testSampleUsages(unittest.TestCase):
# """This series of unit tests is to show the user how to apply memoize calls."""
def testSimpleUsageMemoize(self):
#memoize
def increment(var=0):
var += 1
return var
increment(3)
increment(3)
def testMethodBasedUsage(self):
"""Add the #memoize before method call."""
class myClass(object):
#memoize
def increment(self,var=0):
var += 1
return var
#memoize
def decrement(self, var=0):
var -=1
return var
myObj = myClass()
myObj.increment(3)
myObj.increment(3)
myObj.decrement(6)
myObj.decrement(6)
def testMultipleInstances(self):
#memoize
class myClass(object):
def __init__(self):
self.incrementCountCalls = 0
self.decrementCountCalls = 0
self.powCountCall = 0
# #memoize
def increment(self,var=0):
var += 1
self.incrementCountCalls+=1
return var
# #memoize
def decrement(self, var=0):
self.decrementCountCalls+=1
var -=1
return var
def pow(self, var=0):
self.powCountCall+=1
return var*var
obj1 = myClass() # Memoizing class above does not seem to work.
obj2 = myClass()
obj3 = myClass()
obj1.increment(3)
obj1.increment(3)
#obj2.increment(3)
#obj2.increment(3)
#obj3.increment(3)
#obj3.increment(3)
obj1.pow(4)
obj2.pow(4)
obj3.pow(4)
There's no way to attach a property to a single instance. Being descriptors, propertys must be part of a class definition in order to function. That means you can't easily add them to the partial object you create in __get__.
Now, you could create a class of your own to reimplement the behavior of partial with your added property. However, I suspect the limitation is actually to your benefit. If memo is applied to a method, its state is shared by all instances of the class (and perhaps even instances of subclasses). If you allow the caching details to be adjusted through instances, you might confuse users with cases like:
obj1 = basic()
print obj1.increment.getCacheTimeout() # prints the initial value, e.g. 120
obj2 = basic()
obj2.increment.setCacheTimeOut(20) # change the timeout value via another instance
print obj1.increment.getCacheTimeout() # the value via the first instance now prints 20
I suggest that you make the memoization-related interfaces of decorated methods accessible only through the class, not through instances. To make that work, you need to update your __get__ method to work if obj is None. It can simply return self:
def __get__(self, obj, objtype=None):
if obj is None:
return self
self.objtype = objtype
return partial(self, obj) # no need to attach our methods to the partial anymore
With this change, using a property on the memo via the class works:
basic.increment.cacheTimeOut = 20 # set property of the "unbound" method basic.increment
There is actually a way to accomplish this - by rebinding the decorator as instance-object with a call-method
class Helper(object):
def __init__(self, d, obj):
self.d = d
self.obj = obj
self.timeout = 0
def __call__(self, *args, **kwargs):
print self, self.timeout
return self.d.func(self.obj, *args, **kwargs)
class decorator(object):
def __init__(self, func):
self.func = func
self.name = func.__name__
def __get__(self, obj, clazz):
if object is not None:
obj.__dict__[self.name] = Helper(self, obj)
return obj.__dict__[self.name]
class Foo(object):
#decorator
def bar(self, args):
return args * 2
f = Foo()
g = Foo()
f.bar.timeout = 10
g.bar.timeout = 20
print f.bar(10)
print g.bar(20)
HTH