Related
Here I came up with the solution to the other question asked by me on how to remove all costly calling to debug output function scattered over the function code (slowdown was 25 times with using empty function lambda *p: None).
The solution is to edit function code dynamically and prepend all function calls with comment sign #.
from __future__ import print_function
DEBUG = False
def dprint(*args,**kwargs):
'''Debug print'''
print(*args,**kwargs)
def debug(on=False,string='dprint'):
'''Decorator to comment all the lines of the function code starting with string'''
def helper(f):
if not on:
import inspect
source = inspect.getsource(f)
source = source.replace(string, '#'+string) #Beware! Swithces off the whole line after dprint statement
with open('temp_f.py','w') as file:
file.write(source)
from temp_f import f as f_new
return f_new
else:
return f #return f intact
return helper
def f():
dprint('f() started')
print('Important output')
dprint('f() ended')
f = debug(DEBUG,'dprint')(f) #If decorator #debug(True) is used above f(), inspect.getsource somehow includes #debug(True) inside the code.
f()
The problems I see now are these:
# commets all line to the end; but there may be other statements separated by ;. This may be addressed by deleting all pprint calls in f, not commenting, still it may be not that trivial, as there may be nested parantheses.
temp_f.py is created, and then new f code is loaded from it. There should be a better way to do this without writing to hard drive. I found this recipe, but haven't managed to make it work.
if decorator is applied with special syntax used #debug, then inspect.getsource includes the line with decorator to the function code. This line can be manually removed from string, but it may lead to bugs if there are more than one decorator applied to f. I solved it with resorting to old-style decorator application f=decorator(f).
What other problems do you see here?
How can all these problems be solved?
What are upsides and downsides of this approach?
What can be improved here?
Is there any better way to do what I try to achieve with this code?
I think it's a very interesting and contentious technique to preprocess function code before compilation to byte-code. Strange though that nobody got interested in it. I think the code I gave may have a lot of shaky points.
A decorator can return either a wrapper, or the decorated function unaltered. Use it to create a better debugger:
from functools import wraps
def debug(enabled=False):
if not enabled:
return lambda x: x # Noop, returns decorated function unaltered
def debug_decorator(f):
#wraps(f)
def print_start(*args, **kw):
print('{0}() started'.format(f.__name__))
try:
return f(*args, **kw)
finally:
print('{0}() completed'.format(f.__name__))
return print_start
return debug_decorator
The debug function is a decorator factory, when called it produces a decorator function. If debugging is disabled, it simply returns a lambda that returns it argument unchanged, a no-op decorator. When debugging is enabled, it returns a debugging decorator that prints when a decorated function has started and prints again when it returns.
The returned decorator is then applied to the decorated function.
Usage:
DEBUG = True
#debug(DEBUG)
def my_function_to_be_tested():
print('Hello world!')
To reiterate: when DEBUG is set to false, the my_function_to_be_tested remains unaltered, so runtime performance is not affected at all.
Here is the solution I came up with after composing answers from another questions asked by me here on StackOverflow.
This solution don't comment anything and just deletes standalone dprint statements. It uses ast module and works with Abstract Syntax Tree, it lets us avoid parsing source code. This idea was written in the comment here.
Writing to temp_f.py is replaced with execution f in necessary environment. This solution was offered here.
Also, the last solution addresses the problem of decorator recursive application. It's solved by using _blocked global variable.
This code solves the problem asked to be solved in the question. But still, it's suggested not to be used in real projects:
You are correct, you should never resort to this, there are so many
ways it can go wrong. First, Python is not a language designed for
source-level transformations, and it's hard to write it a transformer
such as comment_1 without gratuitously breaking valid code. Second,
this hack would break in all kinds of circumstances - for example,
when defining methods, when defining nested functions, when used in
Cython, when inspect.getsource fails for whatever reason. Python is
dynamic enough that you really don't need this kind of hack to
customize its behavior.
from __future__ import print_function
DEBUG = False
def dprint(*args,**kwargs):
'''Debug print'''
print(*args,**kwargs)
_blocked = False
def nodebug(name='dprint'):
'''Decorator to remove all functions with name 'name' being a separate expressions'''
def helper(f):
global _blocked
if _blocked:
return f
import inspect, ast, sys
source = inspect.getsource(f)
a = ast.parse(source) #get ast tree of f
class Transformer(ast.NodeTransformer):
'''Will delete all expressions containing 'name' functions at the top level'''
def visit_Expr(self, node): #visit all expressions
try:
if node.value.func.id == name: #if expression consists of function with name a
return None #delete it
except(ValueError):
pass
return node #return node unchanged
transformer = Transformer()
a_new = transformer.visit(a)
f_new_compiled = compile(a_new,'<string>','exec')
env = sys.modules[f.__module__].__dict__
_blocked = True
try:
exec(f_new_compiled,env)
finally:
_blocked = False
return env[f.__name__]
return helper
#nodebug('dprint')
def f():
dprint('f() started')
print('Important output')
dprint('f() ended')
print('Important output2')
f()
Other relevant links:
Switching off debug prints
I noticed pylint doesn't handle well the case of:
#property
def foo(self):
return self._bar.foo
#foo.setter
def foo(self, foo_val):
self._bar.foo = foo_val
Though it's a perfectly valid case syntax since python2.6
It says I defined foo twice, and doesn't understand the ".setter" syntax (Gives E1101 & E0102).
Is there a workaround for that without having to change the code? I don't want to disable the errors as they are important for other places.
Is there any other tool I can use that handles it better? I already checked pyflakes and it behaves the same way. PyDev's code analysis seems to handle this specific case better, but it doesn't check for conventions, refactoring, and other cool features pylint does, and I can't run it from an external script (or can I??)
Thanks!
If you don't want to disable the errors globally, you can disable them for these specific lines, for example:
def foo(self, foo_val): # pylint: disable-msg=E0102
This is ticket http://www.logilab.org/ticket/51222 on the pylint project. Monitor it's status.
Huh. Annoying. And all the major tools I could find (pyflakes, pylint, pychecker) exhibit this problem. It looks like the problem starts in the byte code, but I can't get dis to give me any byte code for object properties.
It looks like you would be better off if you used this syntax:
# Changed to longer member names to reduce pylint grousing
class HughClass(object):
def __init__(self, init_value):
self._hugh = init_value
def hugh_setter(self):
return self._hugh * 2
def hugh_getter(self, value):
self._hugh = value / 2
hugh = property(hugh_getter, hugh_setter)
Here's a nice blog article on it. LOL-quote:
Getters and setters belong to the sad
world of Java and C++.
This was reported as a bug in pyflakes, and it appears to be fixed in current trunk. So I guess the answer (now) is: pyflakes!
There is an eval() function in Python I stumbled upon while playing around. I cannot think of a case when this function is needed, except maybe as syntactic sugar. Can anyone give an example?
eval and exec are handy quick-and-dirty way to get some source code dynamically, maybe munge it a bit, and then execute it -- but they're hardly ever the best way, especially in production code as opposed to "quick-and-dirty" prototypes &c.
For example, if I had to deal with such dynamic Python sources, I'd reach for the ast module -- ast.literal_eval is MUCH safer than eval (you can call it directly on a string form of the expression, if it's a one-off and relies on simple constants only, or do node = ast.parse(source) first, then keep the node around, perhaps munge it with suitable visitors e.g. for variable lookup, then literal_eval the node) -- or, once having put the node in proper shape and vetted it for security issues, I could compile it (yielding a code object) and build a new function object out of that. Far less simple (except that ast.literal_eval is just as simple as eval for the simplest cases!) but safer and preferable in production-quality code.
For many tasks I've seen people (ab-)use exec and eval for, Python's powerful built-ins, such as getattr and setattr, indexing into globals(), &c, provide preferable and in fact often simpler solutions. For specific uses such as parsing JSON, library modules such as json are better (e.g. see SilentGhost's comment on tinnitus' answer to this very question). Etc, etc...
The Wikipedia article on eval is pretty informative, and details various uses.
Some of the uses it suggests are:
Evaluating mathematical expressions
Compiler bootstrapping
Scripting (dynamic languages in general are very suitable to this)
Language tutors
You may want to use it to allow users to enter their own "scriptlets": small expressions (or even small functions), that can be used to customize the behavior of a complex system.
In that context, and if you do not have to care too much for the security implications (e.g. you have an educated userbase), then eval() may be a good choice.
In the past I have used eval() to add a debugging interface to my application. I created a telnet service which dropped you into the environment of the running application. Inputs were run through eval() so you can interactively run Python commands in the application.
In a program I once wrote, you had an input file where you could specify geometric parameters both as values and as python expressions of the previous values, eg:
a=10.0
b=5.0
c=math.log10(a/b)
A python parser read this input file and obtained the final data evaluating the values and the expressions using eval().
I don't claim it to be good programming, but I did not have to drive a nuclear reactor.
I use it as a quick JSON parser ...
r='''
{
"glossary": {
"title": "example glossary"
}
}
'''
print eval(r)['glossary']['title']
You can use eval in a decorator:
#this replaces the original printNumber with a lambda-function,
#which takes no arguments and which calls the old function with
#the number 10
#eval("lambda fun: lambda: fun(10)")
def printNumber(i: int) -> None:
print("The number is %i", i)
#call
printNumber()
while you cannot use complex expressions like
#lambda fun: lambda: fun(10)
def ...
nor
#(lambda fun: lambda: fun(10))
def ...
You cannot use a lambda-expression there, because the decorator should either be an identifier:
#myModule.functionWithOneArg
or a function call:
#functionReturningFunctionWithOneArg(any, "args")
You see that the call of the function eval with a string has valid syntax here, but the lambda-expression not. (-> https://docs.python.org/3/reference/compound_stmts.html#function-definitions)
eval() is not normally very useful. One of the few things I have used it for (well, it was exec() actually, but it's pretty similar) was allowing the user to script an application that I wrote in Python. If it were written in something like C++, I would have to embed a Python interpreter in the application.
Eval is a way to interact with the Python interpreter from within a program. You can pass literals to eval and it evaluates them as python expressions.
For example -
print eval("__import__('os').getcwd()")
would return the current working directory.
cheers
eval() is for single sentence, while exec() is for multiple ones.
usually we use them to add or visit some scripts just like bash shell.
because of they can run some byte scripts in the memory, if you have some important data or script you can decode and unzip your 'secret' then do everything you wanna.
I just came across a good use of eval. I was writing a test suite for some code, and created a Test class, where every method was a test to be run. I wanted a way so that I could run all the test methods without having to call each method individually. So, I wrote something rather dirty.
class Test:
def __init__(self, *args):
#bs
def test1(self):
#bs
def test2(self):
#bs
if __name__ == "__main__":
import argparse
#argparse bs
test = Test(*bs_args)
for func in (i for i in dir(test) if i[0] != '_' and i not in test.__dict__):
print(eval('test.{func}()'.format(func = func)))
Dynamic evaluation of arbitrary test cases is pretty cool. I just have to write the method, and after saving I can include the method in my test suite. As for the code, I basically just inspect the methods defined in the test object, and make sure they aren't default python "magic" methods or attributes to the Test object. After that I can assume they are methods and can be evaluated.
I used it to input variable values to the main program:
test.py var1=2 var2=True
...
var1=0
var2=False
for arg in sys.argv[1:]:
exec(arg)
A crude way to allow keyword args in the main program. If there's a better way let me know!
I had a case where I used eval in combination with an informix database. For some reason the query returned a string formed like this
query_result = "['1', '2', '3']"
I just used eval on the query result so python interpreted it as a list of strings.
[int(i) for i in eval(query_result)]
> [1,2,3]
I could not change the db so this was a quick (and dirty) way to get the integers.
I use exec to create a system of plugins in Python.
try:
exec ("from " + plugin_name + " import Plugin")
myplugin = Plugin(module_options, config=config)
except ImportError, message:
fatal ("No such module " + plugin_name + \
" (or no Plugin constructor) in my Python path: " + str(message))
except Exception:
fatal ("Module " + plugin_name + " cannot be loaded: " + \
str(sys.exc_type) + ": " + str(sys.exc_value) + \
".\n May be a missing or erroneous option?")
With a plugin like:
class Plugin:
def __init__ (self):
pass
def query(self, arg):
...
You will be able to call it like:
result = myplugin.query("something")
I do not think you can have plugins in Python without exec/eval.
I have the following code:
def causes_exception(lamb):
try:
lamb()
return False
except:
return True
I was wondering if it came already in any built-in library?
/YGA
Edit: Thx for all the commentary. It's actually impossible to detect whether code causes an exception without running it -- otherwise you could solve the halting problem (raise an exception if the program halts). I just wanted a syntactically clean way to filter a set of identifiers for those where the code didn't except.
No, as far as I know there is no such function in the standard library. How would it be useful? I mean, presumably you would use it like this:
if causes_exception(func):
# do something
else:
# do something else
But instead, you could just do
try:
func()
except SomeException:
# do something else
else:
# do something
There's assertRaises(exception, callable) in unittest module and this is probably the only place where such check makes sense.
In regular code you can never be 100% sure that causes_exception you suggested are not causing any side effects.
I'm not aware of that function, or anything similar, in the Python standard library.
It's rather misleading - if I saw it used, I might think it told you without calling the function whether the function could raise an exception.
I'm trying to write a freeze decorator for Python.
The idea is as follows :
(In response to the two comments)
I might be wrong but I think there is two main use of
test case.
One is the test-driven development :
Ideally , developers are writing case before writing the code.
It usually helps defining the architecture because this discipline
forces to define the real interfaces before development.
One may even consider that in some case the person who
dispatches job between dev is writing the test case and
use it to illustrate efficiently the specification he has in mind.
I don't have any experience of the use of test case like that.
The second is the idea that all project with a decent
size and a several programmers is suffering from broken code.
Something that use to work may get broken from a change
that looked like an innocent refactoring.
Though good architecture, loose couple between component may
help to fight against this phenomenon ; you will sleep better
at night if you have written some test case to make sure
that nothing will break your program's behavior.
HOWEVER,
Nobody can deny the overhead of writting test cases. In the
first case one may argue that test case is actually guiding
development and is therefore not to be considered as an overhead.
Frankly speaking, I'm a pretty young programmer and if I were
you, my word on this subject is not really valuable...
Anyway, I think that mosts company/projects are not working
like that, and that unit tests are mainly used in the second
case...
In other words, rather than ensuring that the program is
working correctly, it is aiming at checking that it will
work the same in the future.
This needs can be met without the cost of writing tests,
by using this freezing decorator.
Let's say you have a function
def pow(n,k):
if n == 0: return 1
else: return n * pow(n,k-1)
It is perfectly nice, and you want to rewrite it as an optimized version.
It is part of a big project. You want it to give back the same result
for a few value.
Rather than going through the pain of test cases, one could use some
kind of freeze decorator.
Something such that the first time the decorator is run,
the decorator run the function with the defined args (below 0, and 7)
and saves the result in a map ( f --> args --> result )
#freeze(2,0)
#freeze(1,3)
#freeze(3,5)
#freeze(0,0)
def pow(n,k):
if n == 0: return 1
else: return n * pow(n,k-1)
Next time the program is executed, the decorator will load this map and check
that the result of this function for these args as not changed.
I already wrote quickly the decorator (see below), but hurt a few problems about
which I need your advise...
from __future__ import with_statement
from collections import defaultdict
from types import GeneratorType
import cPickle
def __id_from_function(f):
return ".".join([f.__module__, f.__name__])
def generator_firsts(g, N=100):
try:
if N==0:
return []
else:
return [g.next()] + generator_firsts(g, N-1)
except StopIteration :
return []
def __post_process(v):
specialized_postprocess = [
(GeneratorType, generator_firsts),
(Exception, str),
]
try:
val_mro = v.__class__.mro()
for ( ancestor, specialized ) in specialized_postprocess:
if ancestor in val_mro:
return specialized(v)
raise ""
except:
print "Cannot accept this as a value"
return None
def __eval_function(f):
def aux(args, kargs):
try:
return ( True, __post_process( f(*args, **kargs) ) )
except Exception, e:
return ( False, __post_process(e) )
return aux
def __compare_behavior(f, past_records):
for (args, kargs, result) in past_records:
assert __eval_function(f)(args,kargs) == result
def __record_behavior(f, past_records, args, kargs):
registered_args = [ (a, k) for (a, k, r) in past_records ]
if (args, kargs) not in registered_args:
res = __eval_function(f)(args, kargs)
past_records.append( (args, kargs, res) )
def __open_frz():
try:
with open(".frz", "r") as __open_frz:
return cPickle.load(__open_frz)
except:
return defaultdict(list)
def __save_frz(past_records):
with open(".frz", "w") as __open_frz:
return cPickle.dump(past_records, __open_frz)
def freeze_behavior(*args, **kvargs):
def freeze_decorator(f):
past_records = __open_frz()
f_id = __id_from_function(f)
f_past_records = past_records[f_id]
__compare_behavior(f, f_past_records)
__record_behavior(f, f_past_records, args, kvargs)
__save_frz(past_records)
return f
return freeze_decorator
Dumping and Comparing of results is not trivial for all type. Right now I'm thinking about using a function (I call it postprocess here), to solve this problem.
Basically instead of storing res I store postprocess(res) and I compare postprocess(res1)==postprocess(res2), instead of comparing res1 res2.
It is important to let the user overload the predefined postprocess function.
My first question is :
Do you know a way to check if an object is dumpable or not?
Defining a key for the function decorated is a pain. In the following snippets
I am using the function module and its name.
** Can you think of a smarter way to do that. **
The snippets below is kind of working, but opens and close the file when testing and when recording. This is just a stupid prototype... but do you know a nice way to open the file, process the decorator for all function, close the file...
I intend to add some functionalities to this. For instance, add the possibity to define
an iterable to browse a set of argument, record arguments from real use, etc.
Why would you expect from such a decorator?
In general, would you use such a feature, knowing its limitation... Especially when trying to use it with POO?
"In general, would you use such a feature, knowing its limitation...?"
Frankly speaking -- never.
There are no circumstances under which I would "freeze" results of a function in this way.
The use case appears to be based on two wrong ideas: (1) that unit testing is either hard or complex or expensive; and (2) it could be simpler to write the code, "freeze" the results and somehow use the frozen results for refactoring. This isn't helpful. Indeed, the very real possibility of freezing wrong answers makes this a bad idea.
First, on "consistency vs. correctness". This is easier to preserve with a simple mapping than with a complex set of decorators.
Do this instead of writing a freeze decorator.
print "frozen_f=", dict( (i,f(i)) for i in range(100) )
The dictionary object that's created will work perfectly as a frozen result set. No decorator. No complexity to speak of.
Second, on "unit testing".
The point of a unit test is not to "freeze" some random results. The point of a unit test is to compare real results with results developed another (simpler, more obvious, poorly-performing way). Usually unit tests compare hand-developed results. Other times unit tests use obvious but horribly slow algorithms to produce a few key results.
The point of having test data around is not that it's a "frozen" result. The point of having test data is that it is an independent result. Done differently -- sometimes by different people -- that confirms that the function works.
Sorry. This appears to me to be a bad idea; it looks like it subverts the intent of unit testing.
"HOWEVER, Nobody can deny the overhead of writting test cases"
Actually, many folks would deny the "overhead". It isn't "overhead" in the sense of wasted time and effort. For some of us, unittests are essential. Without them, the code may work, but only by accident. With them, we have ample evidence that it actually works; and the specific cases for which it works.
Are you looking to implement invariants or post conditions?
You should specify the result explicitly, this wil remove most of you problems.