passing data with callback - python

I'm creating an async child process with gobject.spawn_async, which generates data on stdout that I want to use when the child exits. So I create two callbacks (minimal example):
output = ""
def child_read(source, cb_condition):
for line in source: output += line + "\n"
def child_done(pid, condition, user_data):
print(user_data)
cpid, cout = gobject.spawn_async(['/bin/ls'],
flags = gobject.SPAWN_DO_NOT_REAP_CHILD,
standard_output=True)
gobject.child_watch_add(pid=cpid, function=child_done, data=output)
gobject.io_add_watch(os.fdopen(cout, 'r'), gobject.IO_IN | gobject.IO_PRI, child_read)
The obvious defect here is that child_done will always print nothing since output is reallocated in child_read. Now the question is, how do I do this in a syntactically nice and readable (i.e. self-documenting) way? Sure, I could just read output in child_done, but then the child_watch_add call doesn't document which data are used in the callback. Plus the callback can't be used for anything else. I'm really missing C/C++ pointer semantics here, since that would do just what I want.
I'm also aware that I could create a wapper class that emulates pointer semantics, but that kinda bloats syntax, too. So, any proposals for doing this "pythonic", i.e. elegantly, in a nice and readable way?

I haven't used gobject, but four hours is a long time for a Python question to remain unanswered on SO, so I'll give it a shot.
Make output a list of strings instead of a string. Something like this, perhaps:
output = []
def child_read(source, cb_condition):
output.extend(source)
def child_done(pid, condition, user_data):
output_str = '\n'.join(user_data)
print(output_str)
The rest of the code is unchanged (i.e. still use data=output in the call to child_watch_add().

Related

Is it possible to "hack" Python's print function?

Note: This question is for informational purposes only. I am interested to see how deep into Python's internals it is possible to go with this.
Not very long ago, a discussion began inside a certain question regarding whether the strings passed to print statements could be modified after/during the call to print has been made. For example, consider the function:
def print_something():
print('This cat was scared.')
Now, when print is run, then the output to the terminal should display:
This dog was scared.
Notice the word "cat" has been replaced by the word "dog". Something somewhere somehow was able to modify those internal buffers to change what was printed. Assume this is done without the original code author's explicit permission (hence, hacking/hijacking).
This comment from the wise #abarnert, in particular, got me thinking:
There are a couple of ways to do that, but they're all very ugly, and
should never be done. The least ugly way is to probably replace the
code object inside the function with one with a different co_consts
list. Next is probably reaching into the C API to access the str's
internal buffer. [...]
So, it looks like this is actually possible.
Here's my naive way of approaching this problem:
>>> import inspect
>>> exec(inspect.getsource(print_something).replace('cat', 'dog'))
>>> print_something()
This dog was scared.
Of course, exec is bad, but that doesn't really answer the question, because it does not actually modify anything during when/after print is called.
How would it be done as #abarnert has explained it?
First, there's actually a much less hacky way. All we want to do is change what print prints, right?
_print = print
def print(*args, **kw):
args = (arg.replace('cat', 'dog') if isinstance(arg, str) else arg
for arg in args)
_print(*args, **kw)
Or, similarly, you can monkeypatch sys.stdout instead of print.
Also, nothing wrong with the exec … getsource … idea. Well, of course there's plenty wrong with it, but less than what follows here…
But if you do want to modify the function object's code constants, we can do that.
If you really want to play around with code objects for real, you should use a library like bytecode (when it's finished) or byteplay (until then, or for older Python versions) instead of doing it manually. Even for something this trivial, the CodeType initializer is a pain; if you actually need to do stuff like fixing up lnotab, only a lunatic would do that manually.
Also, it goes without saying that not all Python implementations use CPython-style code objects. This code will work in CPython 3.7, and probably all versions back to at least 2.2 with a few minor changes (and not the code-hacking stuff, but things like generator expressions), but it won't work with any version of IronPython.
import types
def print_function():
print ("This cat was scared.")
def main():
# A function object is a wrapper around a code object, with
# a bit of extra stuff like default values and closure cells.
# See inspect module docs for more details.
co = print_function.__code__
# A code object is a wrapper around a string of bytecode, with a
# whole bunch of extra stuff, including a list of constants used
# by that bytecode. Again see inspect module docs. Anyway, inside
# the bytecode for string (which you can read by typing
# dis.dis(string) in your REPL), there's going to be an
# instruction like LOAD_CONST 1 to load the string literal onto
# the stack to pass to the print function, and that works by just
# reading co.co_consts[1]. So, that's what we want to change.
consts = tuple(c.replace("cat", "dog") if isinstance(c, str) else c
for c in co.co_consts)
# Unfortunately, code objects are immutable, so we have to create
# a new one, copying over everything except for co_consts, which
# we'll replace. And the initializer has a zillion parameters.
# Try help(types.CodeType) at the REPL to see the whole list.
co = types.CodeType(
co.co_argcount, co.co_kwonlyargcount, co.co_nlocals,
co.co_stacksize, co.co_flags, co.co_code,
consts, co.co_names, co.co_varnames, co.co_filename,
co.co_name, co.co_firstlineno, co.co_lnotab,
co.co_freevars, co.co_cellvars)
print_function.__code__ = co
print_function()
main()
What could go wrong with hacking up code objects? Mostly just segfaults, RuntimeErrors that eat up the whole stack, more normal RuntimeErrors that can be handled, or garbage values that will probably just raise a TypeError or AttributeError when you try to use them. For examples, try creating a code object with just a RETURN_VALUE with nothing on the stack (bytecode b'S\0' for 3.6+, b'S' before), or with an empty tuple for co_consts when there's a LOAD_CONST 0 in the bytecode, or with varnames decremented by 1 so the highest LOAD_FAST actually loads a freevar/cellvar cell. For some real fun, if you get the lnotab wrong enough, your code will only segfault when run in the debugger.
Using bytecode or byteplay won't protect you from all of those problems, but they do have some basic sanity checks, and nice helpers that let you do things like insert a chunk of code and let it worry about updating all offsets and labels so you can't get it wrong, and so on. (Plus, they keep you from having to type in that ridiculous 6-line constructor, and having to debug the silly typos that come from doing so.)
Now on to #2.
I mentioned that code objects are immutable. And of course the consts are a tuple, so we can't change that directly. And the thing in the const tuple is a string, which we also can't change directly. That's why I had to build a new string to build a new tuple to build a new code object.
But what if you could change a string directly?
Well, deep enough under the covers, everything is just a pointer to some C data, right? If you're using CPython, there's a C API to access the objects, and you can use ctypes to access that API from within Python itself, which is such a terrible idea that they put a pythonapi right there in the stdlib's ctypes module. :) The most important trick you need to know is that id(x) is the actual pointer to x in memory (as an int).
Unfortunately, the C API for strings won't let us safely get at the internal storage of an already-frozen string. So screw safely, let's just read the header files and find that storage ourselves.
If you're using CPython 3.4 - 3.7 (it's different for older versions, and who knows for the future), a string literal from a module that's made of pure ASCII is going to be stored using the compact ASCII format, which means the struct ends early and the buffer of ASCII bytes follows immediately in memory. This will break (as in probably segfault) if you put a non-ASCII character in the string, or certain kinds of non-literal strings, but you can read up on the other 4 ways to access the buffer for different kinds of strings.
To make things slightly easier, I'm using the superhackyinternals project off my GitHub. (It's intentionally not pip-installable because you really shouldn't be using this except to experiment with your local build of the interpreter and the like.)
import ctypes
import internals # https://github.com/abarnert/superhackyinternals/blob/master/internals.py
def print_function():
print ("This cat was scared.")
def main():
for c in print_function.__code__.co_consts:
if isinstance(c, str):
idx = c.find('cat')
if idx != -1:
# Too much to explain here; just guess and learn to
# love the segfaults...
p = internals.PyUnicodeObject.from_address(id(c))
assert p.compact and p.ascii
addr = id(c) + internals.PyUnicodeObject.utf8_length.offset
buf = (ctypes.c_int8 * 3).from_address(addr + idx)
buf[:3] = b'dog'
print_function()
main()
If you want to play with this stuff, int is a whole lot simpler under the covers than str. And it's a lot easier to guess what you can break by changing the value of 2 to 1, right? Actually, forget imagining, let's just do it (using the types from superhackyinternals again):
>>> n = 2
>>> pn = PyLongObject.from_address(id(n))
>>> pn.ob_digit[0]
2
>>> pn.ob_digit[0] = 1
>>> 2
1
>>> n * 3
3
>>> i = 10
>>> while i < 40:
... i *= 2
... print(i)
10
10
10
… pretend that code box has an infinite-length scrollbar.
I tried the same thing in IPython, and the first time I tried to evaluate 2 at the prompt, it went into some kind of uninterruptable infinite loop. Presumably it's using the number 2 for something in its REPL loop, while the stock interpreter isn't?
Monkey-patch print
print is a builtin function so it will use the print function defined in the builtins module (or __builtin__ in Python 2). So whenever you want to modify or change the behavior of a builtin function you can simply reassign the name in that module.
This process is called monkey-patching.
# Store the real print function in another variable otherwise
# it will be inaccessible after being modified.
_print = print
# Actual implementation of the new print
def custom_print(*args, **options):
_print('custom print called')
_print(*args, **options)
# Change the print function globally
import builtins
builtins.print = custom_print
After that every print call will go through custom_print, even if the print is in an external module.
However you don't really want to print additional text, you want to change the text that is printed. One way to go about that is to replace it in the string that would be printed:
_print = print
def custom_print(*args, **options):
# Get the desired seperator or the default whitspace
sep = options.pop('sep', ' ')
# Create the final string
printed_string = sep.join(args)
# Modify the final string
printed_string = printed_string.replace('cat', 'dog')
# Call the default print function
_print(printed_string, **options)
import builtins
builtins.print = custom_print
And indeed if you run:
>>> def print_something():
... print('This cat was scared.')
>>> print_something()
This dog was scared.
Or if you write that to a file:
test_file.py
def print_something():
print('This cat was scared.')
print_something()
and import it:
>>> import test_file
This dog was scared.
>>> test_file.print_something()
This dog was scared.
So it really works as intended.
However, in case you only temporarily want to monkey-patch print you could wrap this in a context-manager:
import builtins
class ChangePrint(object):
def __init__(self):
self.old_print = print
def __enter__(self):
def custom_print(*args, **options):
# Get the desired seperator or the default whitspace
sep = options.pop('sep', ' ')
# Create the final string
printed_string = sep.join(args)
# Modify the final string
printed_string = printed_string.replace('cat', 'dog')
# Call the default print function
self.old_print(printed_string, **options)
builtins.print = custom_print
def __exit__(self, *args, **kwargs):
builtins.print = self.old_print
So when you run that it depends on the context what is printed:
>>> with ChangePrint() as x:
... test_file.print_something()
...
This dog was scared.
>>> test_file.print_something()
This cat was scared.
So that's how you could "hack" print by monkey-patching.
Modify the target instead of the print
If you look at the signature of print you'll notice a file argument which is sys.stdout by default. Note that this is a dynamic default argument (it really looks up sys.stdout every time you call print) and not like normal default arguments in Python. So if you change sys.stdout print will actually print to the different target even more convenient that Python also provides a redirect_stdout function (from Python 3.4 on, but it's easy to create an equivalent function for earlier Python versions).
The downside is that it won't work for print statements that don't print to sys.stdout and that creating your own stdout isn't really straightforward.
import io
import sys
class CustomStdout(object):
def __init__(self, *args, **kwargs):
self.current_stdout = sys.stdout
def write(self, string):
self.current_stdout.write(string.replace('cat', 'dog'))
However this also works:
>>> import contextlib
>>> with contextlib.redirect_stdout(CustomStdout()):
... test_file.print_something()
...
This dog was scared.
>>> test_file.print_something()
This cat was scared.
Summary
Some of these points have already be mentioned by #abarnet but I wanted to explore these options in more detail. Especially how to modify it across modules (using builtins/__builtin__) and how to make that change only temporary (using contextmanagers).
A simple way to capture all output from a print function and then process it, is to change the output stream to something else, e.g. a file.
I'll use a PHP naming conventions (ob_start, ob_get_contents,...)
from functools import partial
output_buffer = None
print_orig = print
def ob_start(fname="print.txt"):
global print
global output_buffer
print = partial(print_orig, file=output_buffer)
output_buffer = open(fname, 'w')
def ob_end():
global output_buffer
close(output_buffer)
print = print_orig
def ob_get_contents(fname="print.txt"):
return open(fname, 'r').read()
Usage:
print ("Hi John")
ob_start()
print ("Hi John")
ob_end()
print (ob_get_contents().replace("Hi", "Bye"))
Would print
Hi John
Bye John
Let's combine this with frame introspection!
import sys
_print = print
def print(*args, **kw):
frame = sys._getframe(1)
_print(frame.f_code.co_name)
_print(*args, **kw)
def greetly(name, greeting = "Hi")
print(f"{greeting}, {name}!")
class Greeter:
def __init__(self, greeting = "Hi"):
self.greeting = greeting
def greet(self, name):
print(f"{self.greeting}, {name}!")
You'll find this trick prefaces every greeting with the calling function or method. This might be very useful for logging or debugging; especially as it lets you "hijack" print statements in third party code.

How do I perform some operation on the output of a function in Python?

I am trying to make a function's output behave as if it's my input. The goal is to make a new output from the old output.
I have some code that looks like this:
def func():
BLOCK OF CODE
func()
There is no return statement in the function and no parameters within the parenthesis.
When I type func() to call my function as shown above, I get the desired output, which is a bunch of printed statements. Now I want to do something with that output to get another output.
All I'm trying to do is effectively "pipe" the output of one function into the input of another function (or, if possible, not even worry about creating another function at all, and instead doing something more direct). I looked into Python 3 writing to a pipe
but it did not help me. I also tried defining another function and using the preceding function as a parameter, which did not work either:
def another_func(func):
print another_statement
another_func(func)
I also tried making a closure (which "kind" of worked because at least it printed the same thing that func() would print, but still not very encouraging):
def func():
def another_func():
print another_statement
BLOCK OF CODE
another_func()
Finally, I tried designing both a decorator and a nested function to accomplish this, but I have no parameters in my function, which really threw off my code (didn't print anything at all).
Any advice on how to manipulate a function's output like as if it is your input so that it's possible to create a new output?
You could achieve this by redirecting stdout using a decorator:
from StringIO import StringIO
import sys
def pipe(f):
def decorated(*args, **kwargs):
old,sys.stdout = sys.stdout,StringIO()
try:
result = f(*args, **kwargs)
output = sys.stdout.getvalue()
finally:
sys.stdout = old
return result, output
return decorated
You could then get the result, output pair from any decorated function, eg:
#pipe
def test(x):
print x
return 0
test(3) -> (0, '3\n')
However, I can't think of a good reason why you'd want to do this.
(Actually, that's not quite true; it is handy when writing unit tests for user IO, such as when testing student assignments in a software engineering course. I seriously doubt that that's what the OP is trying to do, though.)
Return the desired value(s) from the function - instead of printing the values on the console, return them as strings, numbers, lists or any other type that makes sense. Otherwise, how do you expect to "connect" the output of a function as the input to another, if there is no output to begin with?
Of course, printing on the console doesn't count as output unless you're planning to eventually use OS pipes or a similar mechanism to connect two programs on the console, but keep things simple! just use the function's return values and worry about pipes later if and only if that's necessary for your problem in particular.
After reading the comments: "connecting" two functions by printing on the console from one and reading from the console from the other would be a really bad idea in this case, first you have to grasp the way functions return values to each other, trust me on this one: you have to rethink your program! even though other answers (strictly speaking) answer your original question, that's absolutely not what you should do.
just for fun ... because OP asked for it
import StringIO
import sys
def func1():
for i in range(1,10):
print "some stuff %d"%i
def func2(func):
old_std = sys.stdout
sys.stdout = StringIO.StringIO()
try:
func()
return sys.stdout.getvalue().splitlines()
finally:
sys.stdout = old_std
print func2(func1)
You need to return a value from your function. This can be used to assign the value into another variable.
Say I define some function doubleThis that will double the input
def doubleThis(x):
print 'this is x :', x
return x * 2 # note the return keyword
Now I can call the function with 3, and it returns 6 as expected
>>> doubleThis(3)
this is x : 3
6
Now I have another function subtractOne that returns the input value, minus 1.
def subtractOne(i):
print 'this is i :', i
return i - 1
Now comes the answer to your question. Note that we can call the first function as the input to the second, due to the fact that it has a return value.
>>> subtractOne(doubleThis(3))
this is x : 3
this is i : 6
5

How to call a function based on list entry?

I'm currently working on an experiment where I'm implementing an interpreter for an old in-game scripting language. It's a forth based language, so I figure it would be fairly easy to just have the instructions (once verified and santized) put into a big list.
Once I've got the code in a list, I am trying to iterate through the entire program in a for loop that processes the instructions one at a time. Certain items, like strings, could be placed onto a variable that holds the current stack, which is easy enough. But where I'm stuck is making commands happen.
I have a big list of functions that are valid and I'd like it to where if any instruction matches them, it calls the associated function.
So, for example, if I had:
"Hello, world!" notify
...the code would check for notify in a list and then execute the notify function. The bottom line is: How do I translate a string into a function name?
You could keep a dictionary of functions the code can call, and then do a look up when you need to:
def notify(s):
print(s)
d = {"notify": notify}
d["notify"]("Hello, world!")
You can do it through locals which is a dictionary with th current local symbol table:
locals()["notify"]()
or though globals which returns a dictionary with the symbol table of globals:
globals()["notify"]()
You can give arguments too e.g.:
locals()["notify"]("Hello, world!")
or
globals()["notify"]("Hello, world!")
If you have a dict called commands that maps names to functions, you can do it like this:
def my_notify_function():
print(stack.pop)
commands = {'notify': my_notify_function, ...}
for item in program:
if item in commands:
commands[item]()
else:
stack.push(item)
Something like:
import re
class LangLib(object):
pattern = re.compile(r'"(.*?)" (.*)')
def run_line(self, line):
arg, command = re.match(LangLib.pattern, line).groups()
return getattr(self, command)(arg)
def notify(self, arg):
print arg
Then your engine code would be:
parser = LangLib()
for line in program_lines:
parser.run_line(line)
Create a dictionary of function names and some tags.
I have tried it several times before, it works really well.

Styling long chains in Python

I've written a Python API that is "chain based" (similar to jQuery). So I can write:
myObject.doStuff().doMoreStuf().goRed().goBlue().die()
The problem is that I haven't found a way to keep the syntax clean with long chains. In JavaScript I could simply do
myOjbect
.doStuf()
.doMoreStuf()
.goRed()
.goBlue()
.die()
but Python has indentation restrictions...
PEP8-compliant solution: formatting the line
Actually PEP8 says:
Long lines can be
broken over multiple lines by wrapping expressions in parentheses. These
should be used in preference to using a backslash for line continuation.
Make sure to indent the continued line appropriately.
So I suppose your code should look like this:
(
myOjbect
.doStuf()
.doMoreStuf()
.goRed()
.goBlue()
.die()
)
Alternative solutions: splitting into separate statements
Judging from the syntax, there are two options possible regarding the values returned by each method call:
Every method (maybe except die(), which is not required, as its result is not being used) returns modified instance (the same instance, on which it was called).
Every method (still, die() is not required to do that) returns copy of the instance on which it was called.
Solution for mutable objects (methods return original instance)
In first case (when returning same instance), the solution to split longer lines into several statements is:
myObject.doStuf()
myObject.doMoreStuf()
myObject.goRed()
myObject.goBlue()
myObject.die()
Real world example involves mutable objects:
my_stuff = []
my_stuff.append('laptop') # my_stuff == ['laptop']
my_stuff.append('jacket') # my_stuff == ['laptop', 'jacket']
my_stuff.append('apple') # my_stuff == ['laptop', 'jacket', 'apple']
(although list.append() does not return anything, just for consistency and for stating explicitly that it is mutable)
Solution for immutable objects (methods return modified copy)
In the second case (when returning copy), the solution to do similar thing is:
myObject = myObject.doStuf()
myObject = myObject.doMoreStuf()
myObject = myObject.goRed()
myObject = myObject.goBlue()
myObject.die()
Real world example involves immutable objects:
name = '-Tadek-'
name = name.strip('-') # name == 'Tadek'
name = name.lower() # name == 'tadek'
name = name.replace('k', 'ck') # name == 'tadeck'
myOjbect \
.doStuf() \
.doMoreStuf() \
.goRed() \
.goBlue() \
.die()
(I feel sorry for myObject. That all sounds quite painful.)
Though I wouldn't call it "clean", it is possible to break inside the parenthesis:
myOjbect.doStuf(
).doMoreStuf(arg1, arg2
).goRed(
).goBlue(
).die()
Tastes vary though, so I'm putting it in an answer for completeness.
I found that a good way to make “chains” look nice is to simply type a bit more:
obj = myObject.doStuff()
obj = obj.doMoreStuf()
obj = obj.goRed()
obj = obj.goBlue()
obj = obj.die()
Or better: use meaningful names, making the code more obvious.
obj = myObject.doStuff().doMoreStuf()
colored_object = obj.goRed().goBlue()
colored_object.die()
I don't think chained methods like this are used very commonly in python (if at all). One option would be to use an API that looks like
myObject.do(
'Stuff',
'MoreStuff',
'Red',
'Blue',
'die',
)
though you might get more relevant advice if you give us a real example - it's hard to guess what kind of operations are being represented here. Maybe some of them should be combined, or placed in __init__ (for example, the colour could be set by passing an optional argument to __init__).

A simple freeze behavior decorator

I'm trying to write a freeze decorator for Python.
The idea is as follows :
(In response to the two comments)
I might be wrong but I think there is two main use of
test case.
One is the test-driven development :
Ideally , developers are writing case before writing the code.
It usually helps defining the architecture because this discipline
forces to define the real interfaces before development.
One may even consider that in some case the person who
dispatches job between dev is writing the test case and
use it to illustrate efficiently the specification he has in mind.
I don't have any experience of the use of test case like that.
The second is the idea that all project with a decent
size and a several programmers is suffering from broken code.
Something that use to work may get broken from a change
that looked like an innocent refactoring.
Though good architecture, loose couple between component may
help to fight against this phenomenon ; you will sleep better
at night if you have written some test case to make sure
that nothing will break your program's behavior.
HOWEVER,
Nobody can deny the overhead of writting test cases. In the
first case one may argue that test case is actually guiding
development and is therefore not to be considered as an overhead.
Frankly speaking, I'm a pretty young programmer and if I were
you, my word on this subject is not really valuable...
Anyway, I think that mosts company/projects are not working
like that, and that unit tests are mainly used in the second
case...
In other words, rather than ensuring that the program is
working correctly, it is aiming at checking that it will
work the same in the future.
This needs can be met without the cost of writing tests,
by using this freezing decorator.
Let's say you have a function
def pow(n,k):
if n == 0: return 1
else: return n * pow(n,k-1)
It is perfectly nice, and you want to rewrite it as an optimized version.
It is part of a big project. You want it to give back the same result
for a few value.
Rather than going through the pain of test cases, one could use some
kind of freeze decorator.
Something such that the first time the decorator is run,
the decorator run the function with the defined args (below 0, and 7)
and saves the result in a map ( f --> args --> result )
#freeze(2,0)
#freeze(1,3)
#freeze(3,5)
#freeze(0,0)
def pow(n,k):
if n == 0: return 1
else: return n * pow(n,k-1)
Next time the program is executed, the decorator will load this map and check
that the result of this function for these args as not changed.
I already wrote quickly the decorator (see below), but hurt a few problems about
which I need your advise...
from __future__ import with_statement
from collections import defaultdict
from types import GeneratorType
import cPickle
def __id_from_function(f):
return ".".join([f.__module__, f.__name__])
def generator_firsts(g, N=100):
try:
if N==0:
return []
else:
return [g.next()] + generator_firsts(g, N-1)
except StopIteration :
return []
def __post_process(v):
specialized_postprocess = [
(GeneratorType, generator_firsts),
(Exception, str),
]
try:
val_mro = v.__class__.mro()
for ( ancestor, specialized ) in specialized_postprocess:
if ancestor in val_mro:
return specialized(v)
raise ""
except:
print "Cannot accept this as a value"
return None
def __eval_function(f):
def aux(args, kargs):
try:
return ( True, __post_process( f(*args, **kargs) ) )
except Exception, e:
return ( False, __post_process(e) )
return aux
def __compare_behavior(f, past_records):
for (args, kargs, result) in past_records:
assert __eval_function(f)(args,kargs) == result
def __record_behavior(f, past_records, args, kargs):
registered_args = [ (a, k) for (a, k, r) in past_records ]
if (args, kargs) not in registered_args:
res = __eval_function(f)(args, kargs)
past_records.append( (args, kargs, res) )
def __open_frz():
try:
with open(".frz", "r") as __open_frz:
return cPickle.load(__open_frz)
except:
return defaultdict(list)
def __save_frz(past_records):
with open(".frz", "w") as __open_frz:
return cPickle.dump(past_records, __open_frz)
def freeze_behavior(*args, **kvargs):
def freeze_decorator(f):
past_records = __open_frz()
f_id = __id_from_function(f)
f_past_records = past_records[f_id]
__compare_behavior(f, f_past_records)
__record_behavior(f, f_past_records, args, kvargs)
__save_frz(past_records)
return f
return freeze_decorator
Dumping and Comparing of results is not trivial for all type. Right now I'm thinking about using a function (I call it postprocess here), to solve this problem.
Basically instead of storing res I store postprocess(res) and I compare postprocess(res1)==postprocess(res2), instead of comparing res1 res2.
It is important to let the user overload the predefined postprocess function.
My first question is :
Do you know a way to check if an object is dumpable or not?
Defining a key for the function decorated is a pain. In the following snippets
I am using the function module and its name.
** Can you think of a smarter way to do that. **
The snippets below is kind of working, but opens and close the file when testing and when recording. This is just a stupid prototype... but do you know a nice way to open the file, process the decorator for all function, close the file...
I intend to add some functionalities to this. For instance, add the possibity to define
an iterable to browse a set of argument, record arguments from real use, etc.
Why would you expect from such a decorator?
In general, would you use such a feature, knowing its limitation... Especially when trying to use it with POO?
"In general, would you use such a feature, knowing its limitation...?"
Frankly speaking -- never.
There are no circumstances under which I would "freeze" results of a function in this way.
The use case appears to be based on two wrong ideas: (1) that unit testing is either hard or complex or expensive; and (2) it could be simpler to write the code, "freeze" the results and somehow use the frozen results for refactoring. This isn't helpful. Indeed, the very real possibility of freezing wrong answers makes this a bad idea.
First, on "consistency vs. correctness". This is easier to preserve with a simple mapping than with a complex set of decorators.
Do this instead of writing a freeze decorator.
print "frozen_f=", dict( (i,f(i)) for i in range(100) )
The dictionary object that's created will work perfectly as a frozen result set. No decorator. No complexity to speak of.
Second, on "unit testing".
The point of a unit test is not to "freeze" some random results. The point of a unit test is to compare real results with results developed another (simpler, more obvious, poorly-performing way). Usually unit tests compare hand-developed results. Other times unit tests use obvious but horribly slow algorithms to produce a few key results.
The point of having test data around is not that it's a "frozen" result. The point of having test data is that it is an independent result. Done differently -- sometimes by different people -- that confirms that the function works.
Sorry. This appears to me to be a bad idea; it looks like it subverts the intent of unit testing.
"HOWEVER, Nobody can deny the overhead of writting test cases"
Actually, many folks would deny the "overhead". It isn't "overhead" in the sense of wasted time and effort. For some of us, unittests are essential. Without them, the code may work, but only by accident. With them, we have ample evidence that it actually works; and the specific cases for which it works.
Are you looking to implement invariants or post conditions?
You should specify the result explicitly, this wil remove most of you problems.

Categories

Resources