I have a module called preparation.py which verifies the arguments passed to a function, if it isn't present, instead of using a pre-stabilished value as a keyword argument, it searches the argument as an attribute of an object. The code is the following:
def prep(argslist, argsprovided, attributes):
argsout = []
for name in argslist:
if name in argsprovided:
argsout.append(argsprovided[name])
else:
argsout.append(getattr(attributes,name))
return argsout
A simple use of this would be:
import preparation as prep
class example(object):
def __init__(self,x,y):
self.x = x
self.y = y
E = example(1,1)
def foo(**kwargs):
[x,y] = prep.prep(['x','y'],kwargs,E)
return x + y
print( foo())
print( foo(x = 2))
Since almost every function in my code does this check everytime it's called, I want to know if the time spent on it is considerable. The time spent on a single time when this module is called can't be measured using time.time() method, so I just can't sum a bunch of smaller intervals. Is there a way of doing this?
Related
Let's say I have a simple class, with an attribute, x:
class A:
def __init__(self):
self.x = random.randint(-5, 5) # not the most efficient, but it serves purposes well
I'll also have a list, with hundreds of instances of this class:
Az = []
for i in range(150):
Az.append(A())
Now, let's say I want to loop through all the As in Az, and run a function on the classes who's x attribute is equivalent to less than one. This is one way, but alas, is very inefficient:
for cls in Az:
if cls.x<1:
func1(cls) # A random function that accepts a class as a parameter, and does something to it
So, to wrap it up, my question: How to optimize the speed of the checking?
Optimizing only the third step is tricky. Why not start at the second step by saving list ids of classes where attribute x is <1?
Az = []
ids = []
for i, id in enumerate(range(150)):
cls = A()
if cls.x < 1:
ids.append(id)
Az.append(cls)
And then modify the third step:
for id in ids:
func1(Az[id])
I know functions in Python are 1st class citizens, meaning that they are objects of the function class similar to how 5 is an object of the int class. This means that at some point in their life-time a constructor is called. For most functions I would expect this to happen when they are defined (as most functions are presumably defined only once) so that we only pay the construction price once no matter how many times we use it.
But what about nested functions? Those are redefined every time their parent is called. Does this mean we re-construct the object every time? If yes, isn't that grossly inefficient? Wouldn't a private method (assuming the parent function is a method) or another function be much more efficient? I am ignoring scoping arguments in favour of nesting for this discussion.
I run a simple experiment that seems to support my aforementioned argument as the inner function edition is slower:
import time
def f(x, y):
def defined_inside(x, y):
return x + y
return defined_inside(x, y)
def defined_outside(x, y):
return x + y
def g(x, y):
return defined_outside(x, y)
start = time.time()
for i in range(10000000):
_ = f(3, 4)
end = time.time()
print("Using inner function it took {} seconds".format(end - start))
start = time.time()
for i in range(10000000):
_ = g(3, 4)
end = time.time()
print("Using outer function it took {} seconds".format(end - start))
Results:
Using inner function it took 2.494696855545044 seconds
Using outer function it took 1.8862690925598145 seconds
Bonus question: In case the above is true, how does the situation relate to compiled languages, such as Scala? I grew into a huge fun of nesting and higher order functions, it would be terrible if the trick is as inefficient as it seems.
I have a very expensive method that returns 2 values, and it is called by class A. Since it is expensive, I made the 2 values lazy evaluated, using properties. Since I don't want to call the very_expensive_function 2 times, the first time the user wants to access one of the 2 values, I save both.
So far I wrote this:
class A:
def __init__(self):
self._attr1 = None
self._attr2 = None
#property
def attr1(self):
self.calculate_metrics()
return self._attr1
#property
def attr2(self):
self.calculate_metrics()
return self._attr2
def calculate_metrics():
if self._attr1 is None:
attr1, attr2 = very_expensive_call()
self._attr1 = attr1
self._attr2 = attr2
As you can see, the first time the user access to attr1 or attr2, I save both. Is it correct or is it possible in another way? It seems very strange to have that calculate_metrics() copy-pasted every time.
Memoization is, simply put, remembering if you have already called a function with particular arguments. If you have it simply returns the already calculated return value rather than calculating it again.
import time
def long_calculation(x, y, memo={}):
try:
result = memo[x, y] # already calculated!
except KeyError:
# make long_calculation take a long time!
time.sleep(2)
result = x * y
memo[x, y] = result
return result
The dictionary memo is able to remember calls to the function because it is evaluated when the function is first loaded: every call to long_calculation shares the same memo dictionary.
To test this try:
# Note that (2,2) (7,8) and (10,10) are repeated here:
test_values = ((2,2),(4,5),(2,2),(7,8),(2,3),(7,8),(10,11),(4,5),(10,10),(10,10))
for values in test_values:
start = time.time()
res = long_calculation(*values)
end = time.time()
elapsed = end-start
print(values,' calculated in ',elapsed, "seconds")
It should be fairly easy to insert this kind of code into your class. If you always need the attributes calculated then you can put the call in __init__.
I have a class where the shared variable self.a is obtained after a very heavy computation which requires a lot of time:
class MyClass(object):
def __init__(self):
# ----------------------------------
# function computationally demanding
out = demanding_function() # In this example, the output is a list of characters ['A','X','R','N','L']
# ----------------------------------
self.a = out
def fun1(self, string):
out = []
for letter in self.a:
out.append(string+letter)
return out
def fun2(self, number):
out = []
for letter in self.a:
out.append(str(number)+letter)
return out
o = MyClass()
x = o.fun1('Hello ')
y = o.fun2(2)
As you can see, self.a is used by the functions fun1 and fun2.
Here is my question is the following: if I call those 2 functions, is the demanding_function() executed multiple times or just once?
Note: this is a generic example and the variables don't have any specific meaning
The function is called just once, when the class instance is initialised i.e. when the __init__ of the class is called. Every other time you access self.a, the already assigned value is used; so no worries.
__init__ is only called once, when you instantiate the object. Any subsequent method calls using that instantiated object will use the already-computed values of instance varaibles
Let's say I've got a variable A that is the result of a function/expression F. F in it's turn has a number of other variables in it, let's say X,Y and Z.
Is it possible to bind A to F so that whenever X,Y or Z changes, A will be updated automatically?
What I want to avoid is that everytime X,Y and Z changes, I have to remember to update A explicitly in the code. I also don't want to call the function everytime I want to use the A.
Example (as per requested): I've got the following function:
def calcHits():
return sum(hitDiceRolls,level*modList['con'])
and in my program (outside of the function), I've got a variable called hitPoints (yes, it's a roleplaying game program). Whenever the variables that's used in the function is changed, I want hitPoints to change as well.
The typical way to do this in Python would be to use a class:
class ExpressionBinder:
def __init__(self, f):
self.f = f
self.x = 0
self.y = 0
self.z = 0
#property
def result(self):
return self.f(self.x, self.y, self.z)
You can use it like this:
def f(x, y, z):
return x**3 + y**2 + z
b = ExpressionBinder(f)
b.x = 1
b.y = 2
b.z = 3
print(b.result)
There is no way in Python to automatically rebind a name in global or local scope in response to other names being rebound. However, it should be possible to make a class that can keep track of some values and have a member function that returns the value you called A. And, as #Alok pointed out, you can use property descriptors to make a member name that implicitly calls a function to return its value, so you can hide the function and treat the name like a plain old name.
class Trk(object):
"""Track some values and compute a function if any change"""
def __init__(self, name, fn, **objects_to_track):
def _trk_fn(self):
if any(self.__dict__[x] != self.original_objects[x] for x in self.original_objects):
self.value = self.saved_fn(self.__dict___)
# now that self.value is updated, also update self.original_objects
for x in self.original_objects:
self.original_objects[x] = self.__dict__[x]
return self.value
self.original_objects = objects_to_track # make reference copy
self.__dict__.update(objects_to_track)
self.name = name
self.saved_fn = fn
self.fn = self._trk_fn()
self.value = self.fn()
I'm sorry but I am very tired right now, and I canot finish this example. I didn't test it either. But this shows one way to track values, and if they are different, do something different. You use it like this:
# want to track x, y, z
trk = Trk(x, y, z)
trk.fn() # returns up-to-date value
trk.x = new_value
trk.fn() #detects that trk.x changed and computes new trk.value
If the above works, you can use the property descriptor stuff to bind a name such that an attempt to read a value from the name will call self.fn()
EDIT: Oh, it's important that when self.value is updated, self.original_objects should be updated. I've added code to do that.
And now I'm going to sleep!