I have searched a little bit to try to figure this one out but didn't get a solution that I was exactly looking for.
This is my use case:
I would like to evaluate expressions from a functions/methods doc-string against the f/m's parameters and values, but from outside the function (when being called but outside execution of the function
I can't statically change the source code I am evaluating (cant write in new functionality) but dynamically changing (i.e. wrapping the function or adding attributes at run-time) is acceptable
I would prefer to stick with tools in the standard library but am willing to try external libraries if it will make the task a breeze
Here is a simple example of what I am trying to do:
def f1(a,b):
"""a==b"""
pass
def f2(f):
f_locals = "get f's args and values before f is executed"
return eval(f.__doc__,None,f_locals)
>>> f2(f1(2,2))
While I have no clue why you would want to do this, what you've described can be achieved with the inspect module. This example is as close to your original example that I can come up with.
from inspect import getcallargs
def f1(a,b):
"""a==b"""
pass
def f2(f, *f_args, **f_kwargs):
f_callargs = getcallargs(f, *f_args, **f_kwargs)
return eval(f.__doc__, None, f_callargs)
f2(f1, 2, 2)
This should output True.
Keep in mind that this assumes a great many things about the arguments and docstrings of any such functions passed to f2, not the least of which is that none of the examined functions are malicious or malformed. Why don't you want to call functions normally, and why don't you want to change functions?
Edit: As Pajton pointed out, getcallargs is more appropriate here, and removes the calls to both dict and zip. The above code has been updated to reflect this.
I'm not sure if this is what you are looking for, but here's an alternative without inspect module.
#!/usr/bin/python
# -*- coding: utf-8-unix -*-
"""
This is a sample implementation of Inline.pm (Perl) in Python.
Using #inline decorator, it is now possible to write any code
in any language in docstring, and let it compile down to executable
Python code at runtime.
For this specific example, it simply evals input docstring, so
code in docstring must be in Python as well.
"""
# Language compiler for MyLang
class MyLang:
#classmethod
def compile(self, docstring):
# For this example, this simply generates code that
# evals docstring.
def testfunc(*arg, **kw):
return eval(docstring, None, kw)
return testfunc
# #inline decorator
def inline(lang):
def decorate(func):
parm = func.__code__.co_varnames[0:func.__code__.co_argcount]
fgen = lang.compile(func.__doc__)
def wrap(*arg, **kw):
# turn all args into keyword-args
kw.update(dict(zip(parm, arg)))
return fgen(**kw)
return wrap
return decorate
#inline(MyLang)
def myadd(a, b):
"""a + b"""
print(myadd(1, 9))
print(myadd(b = 8, a = 2))
print(myadd(a = 3, b = 7))
Related
I have written the following decorator:
def partializable(fn):
def arg_partializer(*fixable_parameters):
def partialized_fn(dynamic_arg):
return fn(dynamic_arg, *fixable_parameters)
return partialized_fn
return arg_partializer
The purpose of this decorator is to break the function call into two calls. If I decorate the following:
#partializable
def my_fn(dyn, fix1, fix2):
return dyn + fix1 + fix2
I then can do:
core_accepting_dynamic_argument = my_fn(my_fix_1, my_fix_2)
final_result = core_accepting_dynamic_argument(my_dyn)
My problem is that the now decorated my_fn exhibits the following signature: my_fn(*fixable_parameters)
I want it to be: my_fn(fix1, fix2)
How can I accomplish this? I probably have to use wraps or the decorator module, but I need to preserve only part of the original signature and I don't know if that's possible.
Taking inspiration from https://stackoverflow.com/a/33112180/9204395, it's possible to accomplish this by manually altering the signature of arg_partializer, since only the signature of fn is known in the relevant scope and can be handled with inspect.
from inspect import signature
def partializable(fn):
def arg_partializer(*fixable_parameters):
def partialized_fn(dynamic_arg):
return fn(dynamic_arg, *fixable_parameters)
return partialized_fn
# Override signature
sig = signature(fn)
sig = sig.replace(parameters=tuple(sig.parameters.values())[1:])
arg_partializer.__signature__ = sig
return arg_partializer
This is not particularly elegant, but as I think about the problem I'm starting to suspect that this (or a conceptual equivalent) is the only possible way to pull this stunt. Feel free to contradict me.
Do I have to formally define a function before I can use it as an element of a dictionary?
def my_func():
print 'my_func'
d = {
'function': my_func
}
I would rather define the function inline. I just tried to type out what I want to do, but the whitespace policies of python syntax make it very hard to define an inline func within a dict. Is there any way to do this?
The answer seems to be that there is no way to declare a function inline a dictionary definition in python. Thanks to everyone who took the time to contribute.
Do you really need a dictionary, or just getitem access?
If the latter, then use a class:
>>> class Dispatch(object):
... def funcA(self, *args):
... print('funcA%r' % (args,))
... def funcB(self, *args):
... print('funcB%r' % (args,))
... def __getitem__(self, name):
... return getattr(self, name)
...
>>> d = Dispatch()
>>>
>>> d['funcA'](1, 2, 3)
funcA(1, 2, 3)
You could use a decorator:
func_dict = {}
def register(func):
func_dict[func.__name__] = func
return func
#register
def a_func():
pass
#register
def b_func():
pass
The func_dict will end up mapping using the entire name of the function:
>>> func_dict
{'a_func': <function a_func at 0x000001F6117BC950>, 'b_func': <function b_func at 0x000001F6117BC8C8>}
You can modify the key used by register as desired. The trick is that we use the __name__ attribute of the function to get the appropriate string.
Consider using lambdas, but note that lambdas can only consist of one expression and cannot contain statements (see http://docs.python.org/reference/expressions.html#lambda).
e.g.
d = { 'func': lambda x: x + 1 }
# call d['func'](2) will return 3
Also, note that in Python 2, print is not a function. So you have to do either:
from __future__ import print_function
d = {
'function': print
}
or use sys.stdout.write instead
d = {
'function': sys.stdout.write
}
Some functions can be easily 'inlined' anonymously with lambda expressions, e.g.:
>>> d={'function': lambda x : x**2}
>>> d['function'](5)
25
But for anything semi-complex (or using statements) you probably just should define them beforehand.
There is no good reason to want to write this using a dictionary in Python. It's strange and is not a common way to namespace functions.
The the Python philosophies that apply here are:
There should be one-- and preferably only one --obvious way to do it.
Combined with
Readability counts.
Doing it this way also makes things hard to understand and read for the typical Python user.
The good things the dictionary does in this case is map strings to functions and namespace them within a dictionary, but this functionality is already provided by both modules and classes and it's much easier to understand by those familiar with Python.
Examples:
Module method:
#cool.py
def cool():
print 'cool'
Now use the module like you would be using your dict:
import cool
#cool.__dict__['cool']()
#update - to the more correct idiom vars
vars(cool)['cool']()
Class method:
class Cool():
def cool():
print 'cool'
#Cool.__dict__['cool']()
#update - to the more correct idiom vars
vars(Cool)['cool']()
Edit after comment below:
argparse seems like a good fit for this problem, so you don't have to reinvent the wheel. If you do decide to implement it completely yourself though argparse source should give you some good direction. Anyways the sections below seem to apply to this use case:
15.4.4.5. Beyond sys.argv
Sometimes it may be useful to have an ArgumentParser parse arguments
other than those of sys.argv. This can be accomplished by passing a
list of strings to parse_args(). This is useful for testing at the
interactive prompt:
15.4.5.1. Sub-commands¶
ArgumentParser.add_subparsers()
Many programs split up their functionality into a number of sub-commands, for example, the svn program can invoke sub-commands
like svn checkout, svn update, and svn commit.
15.4.4.6. The Namespace object
It may also be useful to have an ArgumentParser assign attributes to
an already existing object, rather than a new Namespace object. This
can be achieved by specifying the namespace= keyword argument:
Update, here's an example using argparse
strategizer = argparse.ArgumentParser()
strat_subs = strategizer.add_subparsers()
math = strat_subs.add_parser('math')
math_subs = math.add_subparsers()
math_max = math_subs.add_parser('max')
math_sum = math_subs.add_parser('sum')
math_max.set_defaults(strategy=max)
math_sum.set_defaults(strategy=sum)
strategizer.parse_args('math max'.split())
Out[46]: Namespace(strategy=<built-in function max>)
strategizer.parse_args('math sum'.split())
Out[47]: Namespace(strategy=<built-in function sum>)
I would like to note the reasons I would recommend argparse
Mainly the requirement to use strings that represent options and sub options to map to functions.
It's dead simple (after getting past the feature filled argparse module).
Uses a Python Standard Library Module. This let's others familiar with Python grok what your doing without getting into implementation details, and is very well documented for those who aren't.
Many extra features could be taken advantage of out of the box (not the best reason!).
Using argparse and Strategy Pattern together
For the plain and simple implementation of the Strategy Pattern, this has already been answered very well.
How to write Strategy Pattern in Python differently than example in Wikipedia?
#continuing from the above example
class MathStudent():
def do_math(self, numbers):
return self.strategy(numbers)
maximus = strategizer.parse_args('math max'.split(),
namespace=MathStudent())
sumera = strategizer.parse_args('math sum'.split(),
namespace=MathStudent())
maximus.do_math([1, 2, 3])
Out[71]: 3
sumera.do_math([1, 2, 3])
Out[72]: 6
The point of inlining functions is to blur the distinction between dictionaries and class instances. In javascript, for example, this techinque makes it very pleasant to write control classes that have little reusability. Also, and very helpfully the API then conforms to the well-known dictionary protocols, being self explanatory (pun intended).
You can do this in python - it just doesn't look like a dictionary! In fact, you can use the class keyword in ANY scope (i.e. a class def in a function, or a class def inside of a class def), and it's children can be the dictonary you are looking for; just inspect the attributes of a definition as if it was a javascript dictionary.
Example as if it was real:
somedict = {
"foo":5,
"one_function":your method here,
"two_function":your method here,
}
Is actually accomplished as
class somedict:
foo = 5
#classmethod
def one_method(self):
print self.foo
self.foo *= 2;
#classmethod
def two_method(self):
print self.foo
So that you can then say:
somedict.foo #(prints 5)
somedict.one_method() #(prints 5)
somedict.two_method() #(prints 10)
And in this way, you get the same logical groupings as you would with your "inlining".
I'd like to modify the arguments passed to a method in a module, as opposed to replacing its return value.
I've found a way around this, but it seems like something useful and has turned into a lesson in mocking.
module.py
from third_party import ThirdPartyClass
ThirdPartyClass.do_something('foo', 'bar')
ThirdPartyClass.do_something('foo', 'baz')
tests.py
#mock.patch('module.ThirdPartyClass.do_something')
def test(do_something):
# Instead of directly overriding its return value
# I'd like to modify the arguments passed to this function.
# change return value, no matter inputs
do_something.return_value = 'foo'
# change return value, based on inputs, but have no access to the original function
do_something.side_effect = lambda x, y: y, x
# how can I wrap do_something, so that I can modify its inputs and pass it back to the original function?
# much like a decorator?
I've tried something like the following, but not only is it repetitive and ugly, it doesn't work. After some PDB introspection.. I'm wondering if it's simply due to however this third party library works, as I do see the original functions being called successfully when I drop a pdb inside the side_effect.
Either that, or some auto mocking magic I'm just not following that I'd love to learn about.
def test():
from third_party import ThirdPartyClass
original_do_something = ThirdPartyClass.do_something
with mock.patch('module.ThirdPartyClass.do_something' as mocked_do_something:
def side_effect(arg1, arg2):
return original_do_something(arg1, 'overridden')
mocked_do_something.side_effect = side_effect
# execute module.py
Any guidance is appreciated!
You may want to use parameter wraps for the mock call. (Docs for reference.) This way the original function will be called, but it will have everything from Mock interface.
So for changing parameters called to original function you may want to try it like that:
org.py:
def func(x):
print(x)
main.py:
from unittest import mock
import org
of = org.func
def wrapped(a):
of('--{}--'.format(a))
with mock.patch('org.func', wraps=wrapped):
org.func('x')
org.func.assert_called_with('x')
result:
--x--
The trick is to pass the original underlying function that you still want to access as a parameter to the function.
Eg, for race condition testing, have tempfile.mktemp return an existing pathname:
def mock_mktemp(*, orig_mktemp=tempfile.mktemp, **kwargs):
"""Ensure mktemp returns an existing pathname."""
temp = orig_mktemp(**kwargs)
open(temp, 'w').close()
return temp
Above, orig_mktemp is evaluated when the function is declared, not when it is called, so all invocations will have access to the original method of tempfile.mktemp via orig_mktemp.
I used it as follows:
#unittest.mock.patch('tempfile.mktemp', side_effect=mock_mktemp)
def test_retry_on_existing_temp_path(self, mock_mktemp):
# Simulate race condition: creation of temp path after tempfile.mktemp
...
Here I came up with the solution to the other question asked by me on how to remove all costly calling to debug output function scattered over the function code (slowdown was 25 times with using empty function lambda *p: None).
The solution is to edit function code dynamically and prepend all function calls with comment sign #.
from __future__ import print_function
DEBUG = False
def dprint(*args,**kwargs):
'''Debug print'''
print(*args,**kwargs)
def debug(on=False,string='dprint'):
'''Decorator to comment all the lines of the function code starting with string'''
def helper(f):
if not on:
import inspect
source = inspect.getsource(f)
source = source.replace(string, '#'+string) #Beware! Swithces off the whole line after dprint statement
with open('temp_f.py','w') as file:
file.write(source)
from temp_f import f as f_new
return f_new
else:
return f #return f intact
return helper
def f():
dprint('f() started')
print('Important output')
dprint('f() ended')
f = debug(DEBUG,'dprint')(f) #If decorator #debug(True) is used above f(), inspect.getsource somehow includes #debug(True) inside the code.
f()
The problems I see now are these:
# commets all line to the end; but there may be other statements separated by ;. This may be addressed by deleting all pprint calls in f, not commenting, still it may be not that trivial, as there may be nested parantheses.
temp_f.py is created, and then new f code is loaded from it. There should be a better way to do this without writing to hard drive. I found this recipe, but haven't managed to make it work.
if decorator is applied with special syntax used #debug, then inspect.getsource includes the line with decorator to the function code. This line can be manually removed from string, but it may lead to bugs if there are more than one decorator applied to f. I solved it with resorting to old-style decorator application f=decorator(f).
What other problems do you see here?
How can all these problems be solved?
What are upsides and downsides of this approach?
What can be improved here?
Is there any better way to do what I try to achieve with this code?
I think it's a very interesting and contentious technique to preprocess function code before compilation to byte-code. Strange though that nobody got interested in it. I think the code I gave may have a lot of shaky points.
A decorator can return either a wrapper, or the decorated function unaltered. Use it to create a better debugger:
from functools import wraps
def debug(enabled=False):
if not enabled:
return lambda x: x # Noop, returns decorated function unaltered
def debug_decorator(f):
#wraps(f)
def print_start(*args, **kw):
print('{0}() started'.format(f.__name__))
try:
return f(*args, **kw)
finally:
print('{0}() completed'.format(f.__name__))
return print_start
return debug_decorator
The debug function is a decorator factory, when called it produces a decorator function. If debugging is disabled, it simply returns a lambda that returns it argument unchanged, a no-op decorator. When debugging is enabled, it returns a debugging decorator that prints when a decorated function has started and prints again when it returns.
The returned decorator is then applied to the decorated function.
Usage:
DEBUG = True
#debug(DEBUG)
def my_function_to_be_tested():
print('Hello world!')
To reiterate: when DEBUG is set to false, the my_function_to_be_tested remains unaltered, so runtime performance is not affected at all.
Here is the solution I came up with after composing answers from another questions asked by me here on StackOverflow.
This solution don't comment anything and just deletes standalone dprint statements. It uses ast module and works with Abstract Syntax Tree, it lets us avoid parsing source code. This idea was written in the comment here.
Writing to temp_f.py is replaced with execution f in necessary environment. This solution was offered here.
Also, the last solution addresses the problem of decorator recursive application. It's solved by using _blocked global variable.
This code solves the problem asked to be solved in the question. But still, it's suggested not to be used in real projects:
You are correct, you should never resort to this, there are so many
ways it can go wrong. First, Python is not a language designed for
source-level transformations, and it's hard to write it a transformer
such as comment_1 without gratuitously breaking valid code. Second,
this hack would break in all kinds of circumstances - for example,
when defining methods, when defining nested functions, when used in
Cython, when inspect.getsource fails for whatever reason. Python is
dynamic enough that you really don't need this kind of hack to
customize its behavior.
from __future__ import print_function
DEBUG = False
def dprint(*args,**kwargs):
'''Debug print'''
print(*args,**kwargs)
_blocked = False
def nodebug(name='dprint'):
'''Decorator to remove all functions with name 'name' being a separate expressions'''
def helper(f):
global _blocked
if _blocked:
return f
import inspect, ast, sys
source = inspect.getsource(f)
a = ast.parse(source) #get ast tree of f
class Transformer(ast.NodeTransformer):
'''Will delete all expressions containing 'name' functions at the top level'''
def visit_Expr(self, node): #visit all expressions
try:
if node.value.func.id == name: #if expression consists of function with name a
return None #delete it
except(ValueError):
pass
return node #return node unchanged
transformer = Transformer()
a_new = transformer.visit(a)
f_new_compiled = compile(a_new,'<string>','exec')
env = sys.modules[f.__module__].__dict__
_blocked = True
try:
exec(f_new_compiled,env)
finally:
_blocked = False
return env[f.__name__]
return helper
#nodebug('dprint')
def f():
dprint('f() started')
print('Important output')
dprint('f() ended')
print('Important output2')
f()
Other relevant links:
Switching off debug prints
Say,
I have a python function as following:
def ooxx(**kwargs):
doSomething()
for something in cool:
yield something
I would like to provide another function with named arguments for hints as following:
def asdf(arg1, arg2, arg3=1):
frame = inspect.currentframe()
args, _, _, values = inspect.getargvalues(frame)
kwargs = dict((key, values[key]) for key in args) # convert args list into dictionary form
return list(ooxx(**kwargs))
Is it possible to have some sort of methods to generate automatically the function "asdf"? I have lots of dynamic generated ooxx functions and I would like to have corresponding asdf functions with customized named arguments. Not sure if this is the correct requirement or right way to coding :p
Your descriptions doesn't make such sense to me: You wrote a really verbose function that does this:
def asdf(arg1, arg2, arg3=1):
return list(ooxx(**locals()))
but you want to inspect the ooxx and somehow make up appropriate names for asdfs arguments? That is impossible, there is no information about this on ooxx.
If you actually have a signature and want to create a function from it you would have to resort to eval or generate function definitions to a Python file and import it.
There is also the decorator module. You can create a function with it like this:
import decorator
asdf = decorator.FunctionMaker.create(
'asdf(arg1, arg2, arg3)', # signature
'return ooxx(**locals())', # function body
{'ooxx' : ooxx}, # context for the function
('arg3', 1)) # default arguments