Related
I have a wrapper to time the execution of certain functions in a list. Most of these functions have one and the same parameter: era. I run the functions like displayed below. However, some functions require an extra parameter, e.g. the function dummy_function(). I've been looking for a way to be able to add this parameter in a Pythonic way. I found some solutions but they are very ugly and not quite scalable. Any help or suggestions would be tremendously appreciated!
def dummy_function(self, period, letter='A'):
""" Debugging purposes only """
print(f'This function prints the letter {letter}.')
from time import sleep
sleep(3)
def timed_execution(callbacks, era):
for callback in callbacks:
start_time = time.time()
callback(era)
end_time = time.time()
print(f'{callback.__name__} took {end_time-start_time:.3f}s')
def calculate_insights(era):
timed_execution([
dummy_function,
another_function,
yet_another_function,
], era)
calculate_insights(era)
Perhaps the best way is to actually pass the arguments for their respective function or just try to use a wrapper to calculate the time of a function.
Code taken from another question
from functools import wraps
from time import time
def timing(f):
#wraps(f)
def wrap(*args, **kw):
ts = time()
result = f(*args, **kw)
te = time()
print 'func:%r args:[%r, %r] took: %2.4f sec' % \
(f.__name__, args, kw, te-ts)
return result
return wrap
Then you can do something along the lines of
#timming
def dummy_function(self, period, letter='A'):
""" Debugging purposes only """
print(f'This function prints the letter {letter}.')
from time import sleep
sleep(3)
def calculate_insights():
dummy_function(era)
or you could just a dict with all the parameters passed into each callback but that doesn't sounds to pythonic for me.
Pretty simple question - I have searched but not found an answer to this question.
It might be a bit silly to do this but I was curious if it is possible to hook into the print(*arg, **kwarg) function of python 3.X and override it / add time.sleep(var) at the end of the invoke.
Of course I could just def another method and wrap it with a time.sleep(var) but I was just curious how one would go about overriding the pre-built functions.
If you want to patch any function globally, e.g. for testing / debugging purposes, the safest way is to use unittest.mock.patch():
def x():
'''the code under test'''
print('Hello, world!')
...
from unittest.mock import patch
orig_print = print
with patch('builtins.print') as m:
def my_print(*args, **kwargs):
orig_print('PATCHED:', *args, **kwargs)
m.side_effect = my_print
x() # prints 'PATCHED: Hello, world!'
# prints 'Hello, world!', because the patch context is exited
# and the original function is restored:
x()
You could also try this one liner
out = print
print = lambda *args, **kwargs: [time.sleep(1), out(*args, **kwargs)]
You can, you could do something like:
def new_print(*args, **kwargs):
# Your new print function here
pass
print = new_print
It's recommended that you save the old print function, and you'll need it if you want to use it inside your print function.
You could do so like so
old_print = print
def new_print(*args, **kwargs):
old_print(*args, **kwargs)
print = new_print
If you now want to add sleeping to that, you can do so by just putting it in the new function
import time
old_print = print
def new_print(*args, **kwargs):
old_print(*args, **kwargs)
time.sleep(5)
print = new_print
Just to show that it would work, here an example. It is definitely not recommended though as you know.
import sys
def new_print(msg):
sys.stdout.write("I always add this text\n")
sys.stdout.write(msg)
print = new_print
print("Test")
I always add this text
Test
Python: How to get the caller's method name in the called method?
Assume I have 2 methods:
def method1(self):
...
a = A.method2()
def method2(self):
...
If I don't want to do any change for method1, how to get the name of the caller (in this example, the name is method1) in method2?
inspect.getframeinfo and other related functions in inspect can help:
>>> import inspect
>>> def f1(): f2()
...
>>> def f2():
... curframe = inspect.currentframe()
... calframe = inspect.getouterframes(curframe, 2)
... print('caller name:', calframe[1][3])
...
>>> f1()
caller name: f1
this introspection is intended to help debugging and development; it's not advisable to rely on it for production-functionality purposes.
Shorter version:
import inspect
def f1(): f2()
def f2():
print 'caller name:', inspect.stack()[1][3]
f1()
(with thanks to #Alex, and Stefaan Lippen)
This seems to work just fine:
import sys
print sys._getframe().f_back.f_code.co_name
I would use inspect.currentframe().f_back.f_code.co_name. Its use hasn't been covered in any of the prior answers which are mainly of one of three types:
Some prior answers use inspect.stack but it's known to be too slow.
Some prior answers use sys._getframe which is an internal private function given its leading underscore, and so its use is implicitly discouraged.
One prior answer uses inspect.getouterframes(inspect.currentframe(), 2)[1][3] but it's entirely unclear what [1][3] is accessing.
import inspect
from types import FrameType
from typing import cast
def demo_the_caller_name() -> str:
"""Return the calling function's name."""
# Ref: https://stackoverflow.com/a/57712700/
return cast(FrameType, cast(FrameType, inspect.currentframe()).f_back).f_code.co_name
if __name__ == '__main__':
def _test_caller_name() -> None:
assert demo_the_caller_name() == '_test_caller_name'
_test_caller_name()
Note that cast(FrameType, frame) is used to satisfy mypy.
Acknowlegement: comment by 1313e for an answer.
I've come up with a slightly longer version that tries to build a full method name including module and class.
https://gist.github.com/2151727 (rev 9cccbf)
# Public Domain, i.e. feel free to copy/paste
# Considered a hack in Python 2
import inspect
def caller_name(skip=2):
"""Get a name of a caller in the format module.class.method
`skip` specifies how many levels of stack to skip while getting caller
name. skip=1 means "who calls me", skip=2 "who calls my caller" etc.
An empty string is returned if skipped levels exceed stack height
"""
stack = inspect.stack()
start = 0 + skip
if len(stack) < start + 1:
return ''
parentframe = stack[start][0]
name = []
module = inspect.getmodule(parentframe)
# `modname` can be None when frame is executed directly in console
# TODO(techtonik): consider using __main__
if module:
name.append(module.__name__)
# detect classname
if 'self' in parentframe.f_locals:
# I don't know any way to detect call from the object method
# XXX: there seems to be no way to detect static method call - it will
# be just a function call
name.append(parentframe.f_locals['self'].__class__.__name__)
codename = parentframe.f_code.co_name
if codename != '<module>': # top level usually
name.append( codename ) # function or a method
## Avoid circular refs and frame leaks
# https://docs.python.org/2.7/library/inspect.html#the-interpreter-stack
del parentframe, stack
return ".".join(name)
Bit of an amalgamation of the stuff above. But here's my crack at it.
def print_caller_name(stack_size=3):
def wrapper(fn):
def inner(*args, **kwargs):
import inspect
stack = inspect.stack()
modules = [(index, inspect.getmodule(stack[index][0]))
for index in reversed(range(1, stack_size))]
module_name_lengths = [len(module.__name__)
for _, module in modules]
s = '{index:>5} : {module:^%i} : {name}' % (max(module_name_lengths) + 4)
callers = ['',
s.format(index='level', module='module', name='name'),
'-' * 50]
for index, module in modules:
callers.append(s.format(index=index,
module=module.__name__,
name=stack[index][3]))
callers.append(s.format(index=0,
module=fn.__module__,
name=fn.__name__))
callers.append('')
print('\n'.join(callers))
fn(*args, **kwargs)
return inner
return wrapper
Use:
#print_caller_name(4)
def foo():
return 'foobar'
def bar():
return foo()
def baz():
return bar()
def fizz():
return baz()
fizz()
output is
level : module : name
--------------------------------------------------
3 : None : fizz
2 : None : baz
1 : None : bar
0 : __main__ : foo
You can use decorators, and do not have to use stacktrace
If you want to decorate a method inside a class
import functools
# outside ur class
def printOuterFunctionName(func):
#functools.wraps(func)
def wrapper(self):
print(f'Function Name is: {func.__name__}')
func(self)
return wrapper
class A:
#printOuterFunctionName
def foo():
pass
you may remove functools, self if it is procedural
An alternative to sys._getframe() is used by Python's Logging library to find caller information. Here's the idea:
raise an Exception
immediately catch it in an Except clause
use sys.exc_info to get Traceback frame (tb_frame).
from tb_frame get last caller's frame using f_back.
from last caller's frame get the code object that was being executed in that frame.
In our sample code it would be method1 (not method2) being executed.
From code object obtained, get the object's name -- this is caller method's name in our sample.
Here's the sample code to solve example in the question:
def method1():
method2()
def method2():
try:
raise Exception
except Exception:
frame = sys.exc_info()[2].tb_frame.f_back
print("method2 invoked by: ", frame.f_code.co_name)
# Invoking method1
method1()
Output:
method2 invoked by: method1
Frame has all sorts of details, including line number, file name, argument counts, argument type and so on. The solution works across classes and modules too.
Code:
#!/usr/bin/env python
import inspect
called=lambda: inspect.stack()[1][3]
def caller1():
print "inside: ",called()
def caller2():
print "inside: ",called()
if __name__=='__main__':
caller1()
caller2()
Output:
shahid#shahid-VirtualBox:~/Documents$ python test_func.py
inside: caller1
inside: caller2
shahid#shahid-VirtualBox:~/Documents$
I found a way if you're going across classes and want the class the method belongs to AND the method. It takes a bit of extraction work but it makes its point. This works in Python 2.7.13.
import inspect, os
class ClassOne:
def method1(self):
classtwoObj.method2()
class ClassTwo:
def method2(self):
curframe = inspect.currentframe()
calframe = inspect.getouterframes(curframe, 4)
print '\nI was called from', calframe[1][3], \
'in', calframe[1][4][0][6: -2]
# create objects to access class methods
classoneObj = ClassOne()
classtwoObj = ClassTwo()
# start the program
os.system('cls')
classoneObj.method1()
Hey mate I once made 3 methods without plugins for my app and maybe that can help you, It worked for me so maybe gonna work for you too.
def method_1(a=""):
if a == "method_2":
print("method_2")
if a == "method_3":
print("method_3")
def method_2():
method_1("method_2")
def method_3():
method_1("method_3")
method_2()
Basically I want to do something like this:
How can I hook a function in a python module?
but I want to call the old function after my own code.
like
import whatever
oldfunc = whatever.this_is_a_function
def this_is_a_function(parameter):
#my own code here
# and call original function back
oldfunc(parameter)
whatever.this_is_a_function = this_is_a_function
Is this possible?
I tried copy.copy, copy.deepcopy original function but it didn't work.
Something like this? It avoids using globals, which is generally a good thing.
import whatever
import functools
def prefix_function(function, prefunction):
#functools.wraps(function)
def run(*args, **kwargs):
prefunction(*args, **kwargs)
return function(*args, **kwargs)
return run
def this_is_a_function(parameter):
pass # Your own code here that will be run before
whatever.this_is_a_function = prefix_function(
whatever.this_is_a_function, this_is_a_function)
prefix_function is a function that takes two functions: function and prefunction. It returns a function that takes any parameters, and calls prefunction followed by function with the same parameters. The prefix_function function works for any callable, so you only need to program the prefixing code once for any other hooking you might need to do.
#functools.wraps makes it so that the docstring and name of the returned wrapper function is the same.
If you need this_is_a_function to call the old whatever.this_is_a_function with arguments different than what was passed to it, you could do something like this:
import whatever
import functools
def wrap_function(oldfunction, newfunction):
#functools.wraps(function)
def run(*args, **kwargs):
return newfunction(oldfunction, *args, **kwargs)
return run
def this_is_a_function(oldfunc, parameter):
# Do some processing or something to customize the parameters to pass
newparams = parameter * 2 # Example of a change to newparams
return oldfunc(newparams)
whatever.this_is_a_function = wrap_function(
whatever.this_is_a_function, this_is_a_function)
There is a problem that if whatever is a pure C module, it's typically impossible (or very difficult) to change its internals in the first place.
So, here's an example of monkey-patching the time function from the time module.
import time
old_time = time.time
def time():
print('It is today... but more specifically the time is:')
return old_time()
time.time = time
print time.time()
# Output:
# It is today... but more specifically the time is:
# 1456954003.2
However, if you are trying to do this to C code, you will most likely get an error like cannot overwrite attribute. In that case, you probably want to subclass the C module.
You may want to take a look at this question.
This is the perfect time to tout my super-simplistic Hooker
def hook(hookfunc, oldfunc):
def foo(*args, **kwargs):
hookfunc(*args, **kwargs)
return oldfunc(*args, **kwargs)
return foo
Incredibly simple. It will return a function that first runs the desired hook function (with the same parameters, mind you) and will then run the original function that you are hooking and return that original value. This also works to overwrite a class method. Say we have static method in a class.
class Foo:
#staticmethod
def bar(data):
for datum in data:
print(datum, end="") # assuming python3 for this
print()
But we want to print the length of the data before we print out its elements
def myNewFunction(data):
print("The length is {}.".format(len(data)))
And now we simple hook the function
Foo.bar(["a", "b", "c"])
# => a b c
Foo.bar = hook(Foo.bar, myNewFunction)
Foo.bar(["x", "y", "z"])
# => The length is 3.
# => x y z
Actually, you can replace the target function's func_code. The example below
# a normal function
def old_func():
print "i am old"
# a class method
class A(object):
def old_method(self):
print "i am old_method"
# a closure function
def make_closure(freevar1, freevar2):
def wrapper():
print "i am old_clofunc, freevars:", freevar1, freevar2
return wrapper
old_clofunc = make_closure('fv1', 'fv2')
# ===============================================
# the new function
def new_func(*args):
print "i am new, args:", args
# the new closure function
def make_closure2(freevar1, freevar2):
def wrapper():
print "i am new_clofunc, freevars:", freevar1, freevar2
return wrapper
new_clofunc = make_closure2('fv1', 'fv2')
# ===============================================
# hook normal function
old_func.func_code = new_func.func_code
# hook class method
A.old_method.im_func.func_code = new_func.func_code
# hook closure function
# Note: the closure function's `co_freevars` count should be equal
old_clofunc.func_code = new_clofunc.func_code
# ===============================================
# call the old
old_func()
A().old_method()
old_clofunc()
output:
i am new, args: ()
i am new, args: (<__main__.A object at 0x0000000004A5AC50>,)
i am new_clofunc, freevars: fv1 fv2
Im running several machine learning algorithms with sklearn in a for loop and want to see how long each of them takes. The problem is I also need to return a value and DONT want to have to run it more than once because each algorithm takes so long. Is there a way to capture the return value 'clf' using python's timeit module or a similar one with a function like this...
def RandomForest(train_input, train_output):
clf = ensemble.RandomForestClassifier(n_estimators=10)
clf.fit(train_input, train_output)
return clf
when I call the function like this
t = Timer(lambda : RandomForest(trainX,trainy))
print t.timeit(number=1)
P.S. I also dont want to set a global 'clf' because I might want to do multithreading or multiprocessing later.
For Python 3.5 you can override the value of timeit.template
timeit.template = """
def inner(_it, _timer{init}):
{setup}
_t0 = _timer()
for _i in _it:
retval = {stmt}
_t1 = _timer()
return _t1 - _t0, retval
"""
unutbu's answer works for python 3.4 but not 3.5 as the _template_func function appears to have been removed in 3.5
The problem boils down to timeit._template_func not returning the function's return value:
def _template_func(setup, func):
"""Create a timer function. Used if the "statement" is a callable."""
def inner(_it, _timer, _func=func):
setup()
_t0 = _timer()
for _i in _it:
_func()
_t1 = _timer()
return _t1 - _t0
return inner
We can bend timeit to our will with a bit of monkey-patching:
import timeit
import time
def _template_func(setup, func):
"""Create a timer function. Used if the "statement" is a callable."""
def inner(_it, _timer, _func=func):
setup()
_t0 = _timer()
for _i in _it:
retval = _func()
_t1 = _timer()
return _t1 - _t0, retval
return inner
timeit._template_func = _template_func
def foo():
time.sleep(1)
return 42
t = timeit.Timer(foo)
print(t.timeit(number=1))
returns
(1.0010340213775635, 42)
The first value is the timeit result (in seconds), the second value is the function's return value.
Note that the monkey-patch above only affects the behavior of timeit when a callable is passed timeit.Timer. If you pass a string statement, then you'd have to (similarly) monkey-patch the timeit.template string.
Funnily enough, I'm also doing machine-learning, and have a similar requirement ;-)
I solved it as follows, by writing a function, that:
runs your function
prints the running time, along with the name of your function
returns the results
Let's say you want to time:
clf = RandomForest(train_input, train_output)
Then do:
clf = time_fn( RandomForest, train_input, train_output )
Stdout will show something like:
mymodule.RandomForest: 0.421609s
Code for time_fn:
import time
def time_fn( fn, *args, **kwargs ):
start = time.clock()
results = fn( *args, **kwargs )
end = time.clock()
fn_name = fn.__module__ + "." + fn.__name__
print fn_name + ": " + str(end-start) + "s"
return results
If I understand it well, after python 3.5 you can define globals at each Timer instance without having to define them in your block of code. I am not sure if it would have the same issues with parallelization.
My approach would be something like:
clf = ensemble.RandomForestClassifier(n_estimators=10)
myGlobals = globals()
myGlobals.update({'clf'=clf})
t = Timer(stmt='clf.fit(trainX,trainy)', globals=myGlobals)
print(t.timeit(number=1))
print(clf)
As of 2020, in ipython or jupyter notebook it is
t = %timeit -n1 -r1 -o RandomForest(trainX, trainy)
t.best
If you don't want to monkey-patch timeit, you could try using a global list, as below. This will also work in python 2.7, which doesn't have globals argument in timeit():
from timeit import timeit
import time
# Function to time - plaigiarised from answer above :-)
def foo():
time.sleep(1)
return 42
result = []
print timeit('result.append(foo())', setup='from __main__ import result, foo', number=1)
print result[0]
will print the time and then the result.
An approach I'm using it is to "append" the running time to the results of the timed function. So, I write a very simple decorator using the "time" module:
def timed(func):
def func_wrapper(*args, **kwargs):
import time
s = time.clock()
result = func(*args, **kwargs)
e = time.clock()
return result + (e-s,)
return func_wrapper
And then I use the decorator for the function I want to time.
For Python 3.X I use this approach:
# Redefining default Timer template to make 'timeit' return
# test's execution timing and the function return value
new_template = """
def inner(_it, _timer{init}):
{setup}
_t0 = _timer()
for _i in _it:
ret_val = {stmt}
_t1 = _timer()
return _t1 - _t0, ret_val
"""
timeit.template = new_template