I'm struggling with this using timeit and was wondering if anyone had any tips
Basically I have a function(that I pass a value to) that I want to test the speed of and created this:
if __name__=='__main__':
from timeit import Timer
t = Timer(superMegaIntenseFunction(10))
print t.timeit(number=1)
but when I run it, I get weird errors like coming from the timeit module.:
ValueError: stmt is neither a string nor callable
If I run the function on its own, it works fine. Its when I wrap it in the time it module, I get the errors(I have tried using double quotes and without..sameoutput).
any suggestions would be awesome!
Thanks!
Make it a callable:
if __name__=='__main__':
from timeit import Timer
t = Timer(lambda: superMegaIntenseFunction(10))
print(t.timeit(number=1))
Should work
Timer(superMegaIntenseFunction(10)) means "call superMegaIntenseFunction(10), then pass the result to Timer". That's clearly not what you want. Timer expects either a callable (just as it sounds: something that can be called, such as a function), or a string (so that it can interpret the contents of the string as Python code). Timer works by calling the callable-thing repeatedly and seeing how much time is taken.
Timer(superMegaIntenseFunction) would pass the type check, because superMegaIntenseFunction is callable. However, Timer wouldn't know what values to pass to superMegaIntenseFunction.
The simple way around this, of course, is to use a string with the code. We need to pass a 'setup' argument to the code, because the string is "interpreted as code" in a fresh context - it doesn't have access to the same globals, so you need to run another bit of code to make the definition available - see #oxtopus's answer.
With lambda (as in #Pablo's answer), we can bind the parameter 10 to a call to superMegaIntenseFunction. All that we're doing is creating another function, that takes no arguments, and calls superMegaIntenseFunction with 10. It's just as if you'd used def to create another function like that, except that the new function doesn't get a name (because it doesn't need one).
You should be passing a string. i.e.
t = Timer('superMegaIntenseFunction(10)','from __main__ import superMegaIntenseFunction')
One way to do it would be by using partial so that the function, 'superMegaIntenseFunction' is used as a callable (ie without the ()) in the timer or directly inside timeit.timeit. Using partial will pass the argument to the function when it will be call by the timer.
from functools import partial
from timeit import timeit
print(timeit(partial(superMegaIntenseFunction, 10), number=1))
A note for future visitors. If you need to make it work in pdb debugger, and superMegaIntenseFunction is not in the global scope, you can make it work by adding to globals:
globals()['superMegaIntenseFunction'] = superMegaIntenseFunction
timeit.timeit(lambda: superMegaIntenseFunction(x))
Note that the timing overhead is a little larger in this case because of the extra function calls. [source]
Related
I explain:
I would like to know how I can pass method or functions as arguments.
For example, in Python will be:
from MyFile import MyClass
MyClass().my_method_click(function) # without parentheses
In this example, in Python you send the functions or method without
parentheses, if I do:
from MyFile import MyClass
MyClass().my_method_click(function()) # with parentheses
I call the function but don't send it.
In Ruby, when you call a method or function, you can do it with or
without parentheses.
if I do this in Ruby:
require_relative "MyClass"
MyClass.new.my_method_click(function) # without parentheses
Just call it without send it.
Of course, is for a Button, that when I click it, run this operation.
How I can do it in Ruby??
Thanks!
Basically, you want to pass a runnable block of code. I haven't looked into Python yet, but I am sure it supports closures as well.
Anyhow, in Ruby, a "general" way of passing a runnable code is to use blocks (lambdas and procs).
function = lambda { # your code }
MyClass.new.my_method_click(function)
# or a shorter way
MyClass.new.my_method_click(-> { # your code })
# to run a block
def my_method_click(&block)
#you can either `yield` from your receiving method
yield
# or call `.call` method on your lambda/proc instance
block.call
end
You can also get an instance of your class' method or create a new method one using Method.new. But, you'd end up dealing with bindings and binding to the correct instance types, etc. So, it's much easier with Lambdas and Procs.
Suppose I have the following code:
def my_func(input_line):
is_skip_line = self.is_skip_line(input_line) # parse input line check if skip line
if is_skip_line:
# do something...
# do more ...
if is_skip_line:
# do one last thing
So we have a check for is_skip_line (if is_skip_line:) that appears twice. Does it mean that due to lazy evaluation the method self.is_skip_line(input_line) will be called twice?
If so, what is the best work around, given that self.is_skip_line(input_line) is time consuming? Do I have to "immediately invoke" it, like below?
is_skip_line = (lambda x: self.is_skip_line(x))(input_line)
Thanks.
The misconception here is that this statement is not being immediately invoked:
is_skip_line = self.is_skip_line(input_line)
...when in fact, it is.
The method self.is_skip_line will only ever be invoked once. Since you assign it to a variable, you can use that variable as many times as you like in any context you like.
If you're concerned about the performance of it, then you could use cProfile to really test the performance of the method it's called in with respect to the method it's calling.
You want to write unittest-cases for a function like that:
def test_me(a):
for b in c:
print do_something(a,b)
At first I thought about just collecting the outputs of do_something in a string and then returning it, to print and test the whole output together. But it's not always convinient because such loops could cause your buffer string to get very big, depending on the circumstances. So what can you do to test the output, when it is printed and not returned?
print prints to sys.stdout, which you can reassign to your own object if you wish. The only thing your object needs is a write function which takes a single string argument.
Since Python 2.6 you may also change print to be a function rather than a language construct by adding from __future__ import print_function to the top of your script. This way you can override print with your own function.
In Python 3 it's easy to use unittest.mock on the builtin print function:
from unittest.mock import patch, call
#patch('builtins.print')
def test_print(mocked_print):
print('foo')
print()
assert mocked_print.mock_calls == [call('foo'), call()]
I have a situation where I am attempting to port some big, complex python routines to a threaded environment.
I want to be able to, on a per-call basis, redirect the output from the function's print statement somewhere else (a logging.Logger to be specific).
I really don't want to modify the source for the code I am compiling, because I need to maintain backwards compatibility with other software that calls these modules (which is single threaded, and captures output by simply grabbing everything written to sys.stdout).
I know the best option is to do some rewriting, but I really don't have a choice here.
Edit -
Alternatively, is there any way I can override the local definition of print to point to a different function?
I could then define the local print = system print unless overwritten by a kwarg, and would only involve modify a few lines at the beginning of each routine.
In Python2.6 (and 2.7), you can use
from __future__ import print_function
Then you can change the code to use the print() function as you would for Python3
This allows you to create a module global or local function called print which will be used in preference to the builtin function
eg.
from __future__ import print_function
def f(x, print=print):
print(x*x)
f(5)
L=[]
f(6, print=L.append)
print(L)
Modifying the source code doesn't need to imply breaking backward compatibility.
What you need to do is first replace every print statement with a call to a function that does the same thing:
import sys
def _print(*args, **kw):
sep = kw.get('sep', ' ')
end = kw.get('end', '\n')
file = kw.get('file', sys.stdout)
file.write(sep.join(args))
file.write(end)
def foo():
# print "whatever","you","want"
_print("whatever","you","want")
Then the second step is to stop using the _print function directly and make it a keyword argument:
def foo(_print=_print):
...
and make sure to change all internal function calls to pass the _print function around.
Now all the existing code will continue to work and will use print, but you can pass in whatever _print function you want.
Note that the signature of _print is exactly that of the print function in more recent versions of Python, so as soon as you upgrade you can just change it to use print(). Also you may get away with using 2to3 to migrate the print statements in the existing code which should reduce the editing required.
Someone in the sixties had an idea about how to solve this but it requires a bit of alien technology. Unfortunately python has no "current environment" concept and this means you cannot provide context unless specifying it in calls as a parameter.
For handling just this specific problem what about replacing stdout with a file-like object that behaves depending on a thread-specific context ? This way the source code remains the same but for example you can get a separate log for each thread. It's even easy to do this on a specific per-call way... for example:
class MyFakeStdout:
def write(self, s):
try:
separate_logs[current_thread()].write(s)
except KeyError:
old_stdout.write(s)
and then having a function to set a logger locally to a call (with)
PS: I saw the "without touching stdout" in the title but I thought this was because you wanted only some thread to be affected. Touching it while still allowing other threads to work unaffected seems to me compatible with the question.
I want to profile my Python code. I am well-aware of cProfile, and I use it, but it's too low-level. (For example, there isn't even a straightforward way to catch the return value from the function you're profiling.)
One of the things I would like to do: I want to take a function in my program and set it to be profiled on the fly while running the program.
For example, let's say I have a function heavy_func in my program. I want to start the program and have the heavy_func function not profile itself. But sometime during the runtime of my program, I want to change heavy_func to profile itself while it's running. (If you're wondering how I can manipulate stuff while the program is running: I can do it either from the debug probe or from the shell that's integrated into my GUI app.)
Is there a module already written which does stuff like this? I can write it myself but I just wanted to ask before so I won't be reinventing the wheel.
It may be a little mind-bending, but this technique should help you find the "bottlenecks", it that's what you want to do.
You're pretty sure of what routine you want to focus on.
If that's the routine you need to focus on, it will prove you right.
If the real problem(s) are somewhere else, it will show you where they are.
If you want a tedious list of reasons why, look here.
I wrote my own module for it. I called it cute_profile. Here is the code. Here are the tests.
Here is the blog post explaining how to use it.
It's part of GarlicSim, so if you want to use it you can install garlicsim and do from garlicsim.general_misc import cute_profile.
If you want to use it on Python 3 code, just install the Python 3 fork of garlicsim.
Here's an outdated excerpt from the code:
import functools
from garlicsim.general_misc import decorator_tools
from . import base_profile
def profile_ready(condition=None, off_after=True, sort=2):
'''
Decorator for setting a function to be ready for profiling.
For example:
#profile_ready()
def f(x, y):
do_something_long_and_complicated()
The advantages of this over regular `cProfile` are:
1. It doesn't interfere with the function's return value.
2. You can set the function to be profiled *when* you want, on the fly.
How can you set the function to be profiled? There are a few ways:
You can set `f.profiling_on=True` for the function to be profiled on the
next call. It will only be profiled once, unless you set
`f.off_after=False`, and then it will be profiled every time until you set
`f.profiling_on=False`.
You can also set `f.condition`. You set it to a condition function taking
as arguments the decorated function and any arguments (positional and
keyword) that were given to the decorated function. If the condition
function returns `True`, profiling will be on for this function call,
`f.condition` will be reset to `None` afterwards, and profiling will be
turned off afterwards as well. (Unless, again, `f.off_after` is set to
`False`.)
`sort` is an `int` specifying which column the results will be sorted by.
'''
def decorator(function):
def inner(function_, *args, **kwargs):
if decorated_function.condition is not None:
if decorated_function.condition is True or \
decorated_function.condition(
decorated_function.original_function,
*args,
**kwargs
):
decorated_function.profiling_on = True
if decorated_function.profiling_on:
if decorated_function.off_after:
decorated_function.profiling_on = False
decorated_function.condition = None
# This line puts it in locals, weird:
decorated_function.original_function
base_profile.runctx(
'result = '
'decorated_function.original_function(*args, **kwargs)',
globals(), locals(), sort=decorated_function.sort
)
return locals()['result']
else: # decorated_function.profiling_on is False
return decorated_function.original_function(*args, **kwargs)
decorated_function = decorator_tools.decorator(inner, function)
decorated_function.original_function = function
decorated_function.profiling_on = None
decorated_function.condition = condition
decorated_function.off_after = off_after
decorated_function.sort = sort
return decorated_function
return decorator