Develop Reference

How to pass function as variable with fixed argument

I'm newbie in Python, but the second time I encouter this problem.
Problem:
In some libraries there are functions with arguments. Sometimes there is argument as function, like this:
def somefun(fun):
x = [1,2,3]
z = fun(x)
return z
And I want to pass there some other function like this:
def func(x,y):
return x*y
which have more than one argument. I want to make one argument static, so somefun except func as argument.
Finally I want to make some kind of cycle where I can change static arg.
Something like this:
for i in xrange(1,9):
somefun(func(i,*))
Please do not offer me to change any functions. They are from library and it's not very comfortable to change them.
Thanks a lot!

You can use lambda statement:
somefun(lambda x: func(i, x))

It sure sounds like you are looking for functools.partial. From the docs:
functools.partial(func, *args, **keywords)
Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords.
In your example, you could pass partial(func, 10) as the argument to somefun. Or you could create the partial objects and use them in a loop:
for i in xrange(1,9):
somefun(partial(func, i))

My solution with decorator
from functools import wraps
import numpy as np
def p_decorate(f):
#wraps(f)
def wrapped(*args):
z = f(*args)
return z
return wrapped
#p_decorate
def myfunc(a,b):
"""My new function"""
z = np.dot(a,b)
return z
x = [1,2,3]
y = [4,2,0]
r = myfunc(x,y)
print (r)
print (myfunc.__name__)
print (myfunc.__doc__)
You can change myfunc as you wish.You can also insert more function layers.Without the use of this decorator factory,you would lose the name of myfunc and the docstring.

How to apply class decorator at base of all decorators on methods

I am using this way of decorating all methods
import inspect
def decallmethods(decorator, prefix='test_'):
def dectheclass(cls):
for name, m in inspect.getmembers(cls, inspect.ismethod):
if name.startswith(prefix):
setattr(cls, name, decorator(m))
return cls
return dectheclass
#decallmethods(login_testuser)
class TestCase(object):
def setUp(self):
pass
def test_1(self):
print "test_1()"
def test_2(self):
print "test_2()"
This is working but it applies at the top , if i have other decorators.
I mean
Now the result is
#login_testuser
#other
def test_2(self):
print "test_2()"
But i want
#other
#login_testuser
def test_2(self):
print "test_2()"

This is most certainly a bad idea, but what you want to do can be done in some extent, and this is going to take a lot of time to explain. First off, rather than thinking of decorators as a syntax sugar, think of them as what they really are: a function (that is a closure) with a function that exist inside it. Now this is out of the way, supposed we have a function:
def operation(a, b):
print('doing operation')
return a + b
Simply it will do this
>>> hi = operation('hello', 'world')
doing operation
>>> print(hi)
helloworld
Now define a decorator that prints something before and after calling its inner function (equivalent to the other decorator that you want to decorator later):
def other(f):
def other_inner(*a, **kw):
print('other start')
result = f(*a, **kw)
print('other finish')
return result
return other_inner
With that, build a new function and decorator
#other
def o_operation(a, b):
print('doing operation')
return a + b
Remembering, this is basically equivalent to o_operation = other(operation)
Run this to ensure it works:
>>> r2 = o_operation('some', 'inner')
other start
doing operation
other finish
>>> print(r2)
someinner
Finally, the final decorator you want to call immediately before operation but not d_operation, but with your existing code it results in this:
def inject(f):
def injected(*a, **kw):
print('inject start')
result = f(*a, **kw)
print('inject finish')
return result
return injected
#inject
#other
def i_o_operation(a, b):
print('doing operation')
return a + b
Run the above:
>>> i_o_operation('hello', 'foo')
inject start
other start
doing operation
other finish
inject finish
'hellofoo'
As mentioned decorators are really closures and hence that's why it's possible to have items inside that are effectively instanced inside. You can reach them by going through the __closure__ attribute:
>>> i_o_operation.__closure__
(<cell at 0x7fc0eabd1fd8: function object at 0x7fc0eabce7d0>,)
>>> i_o_operation.__closure__[0].cell_contents
<function other_inner at 0x7fc0eabce7d0>
>>> print(i_o_operation.__closure__[0].cell_contents('a', 'b'))
other start
doing operation
other finish
ab
See how this effectively calls the function inside the injected closure directly, as if that got unwrapped. What if that closure can be replaced with the one that did the injection? For all of our protection, __closure__ and cell.cell_contents are read-only. What needs to be done is to construct completely new functions with the intended closures by making use of the FunctionType function constructor (found in the types module)
Back to the problem. Since what we have now is:
i_o_operation = inject(other(operation))
And what we want is
o_i_operation = other(inject(operation))
We effectively have to somehow strip the call to other from i_o_operation and somehow wrap it around with inject to produce o_i_operation. (Dragons follows after the break)
First, construct a function that effectively calls inject(operation) by taking the closure to level deep (so that f will contain just the original operation call) but mix it with the code produced by inject(f):
i_operation = FunctionType(
i_o_operation.__code__,
globals=globals(),
closure=i_o_operation.__closure__[0].cell_contents.__closure__,
)
Since i_o_operation is the result of inject(f) we can take that code to produce a new function. The globals is a formality that's required, and finally take the closure of the nested level, and the first part of the function is produced. Verify that the other is not called.
>>> i_operation('test', 'strip')
inject start
doing operation
inject finish
'teststrip'
Neat. However we still want the other to be wrapped outside of this to finally produce o_i_operation. We do need to somehow put this new function we produced in a closure, and a way to do this is to create a surrogate function that produce one
def closure(f):
def surrogate(*a, **kw):
return f(*a, **kw)
return surrogate
And simply use it to construct and extract our closure
o_i_operation = FunctionType(
i_o_operation.__closure__[0].cell_contents.__code__,
globals=globals(),
closure=closure(i_operation).__closure__,
)
Call this:
>>> o_i_operation('job', 'complete')
other start
inject start
doing operation
inject finish
other finish
'jobcomplete'
Looks like we finally got what we need. While this doesn't exactly answer your exact problem, this started down the right track but is already pretty hairy.
Now for the actual problem: a function that will ensure a decorator function be the most inner (final) callable before a given original, undecorated function - i.e. for a given target and a f(g(...(callable)), we want to emulate a result that gives f(g(...(target(callable)))). This is the code:
from types import FunctionType
def strip_decorators(f):
"""
Strip all decorators from f. Assumes each are functions with a
closure with a first cell being the target function.
"""
# list of not the actual decorator, but the returned functions
decorators = []
while f.__closure__:
# Assume first item is the target method
decorators.append(f)
f = f.__closure__[0].cell_contents
return decorators, f
def inject_decorator(decorator, f):
"""
Inject a decorator to the most inner function within the stack of
closures in `f`.
"""
def closure(f):
def surrogate(*a, **kw):
return f(*a, **kw)
return surrogate
decorators, target_f = strip_decorators(f)
result = decorator(target_f)
while decorators:
# pop out the last one in
decorator = decorators.pop()
result = FunctionType(
decorator.__code__,
globals=globals(),
closure=closure(result).__closure__,
)
return result
To test this, we use a typical example use-case - html tags.
def italics(f):
def i(s):
return '<i>' + f(s) + '</i>'
return i
def bold(f):
def b(s):
return '<b>' + f(s) + '</b>'
return b
def underline(f):
def u(s):
return '<u>' + f(s) + '</u>'
return u
#italics
#bold
def hi(s):
return s
Running the test.
>>> hi('hello')
'<i><b>hello</b></i>'
Our target is to inject the underline decorator (specifically the u(hi) callable) into the most inner closure. This can be done like so, with the function we have defined above:
>>> hi_u = inject_decorator(underline, hi)
>>> hi_u('hello')
'<i><b><u>hello</u></b></i>'
Works with undecorated functions:
>>> def pp(s):
... return s
...
>>> pp_b = inject_decorator(bold, pp)
>>> pp_b('hello')
'<b>hello</b>'
A major assumption was made for this first-cut version of the rewriter, which is that all decorators in the chain only have a closure length of one, that one element being the function being decorated with. Take this decorator for instance:
def prefix(p):
def decorator(f):
def inner(*args, **kwargs):
new_args = [p + a for a in args]
return f(*new_args, **kwargs)
return inner
return decorator
Example usage:
>>> #prefix('++')
... def prefix_hi(s):
... return s
...
>>> prefix_hi('test')
'++test'
Now try to inject a bold decorator like so:
>>> prefix_hi_bold = inject_decorator(bold, prefix_hi)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 18, in inject_decorator
ValueError: inner requires closure of length 2, not 1
This is simply because the closure formed by decorator within prefix has two elements, one being the prefix string p and the second being the actual function, and inner being nested inside that expects both those to be present inside its closure. Resolving that will require more code to analyse and reconstruct the details.
Anyway, this explanation took quite a bit of time and words, so I hope you understand this and maybe get you started on the actual right track.
If you want to turn inject_decorator into a decorator, and/or mix it into your class decorator, best of luck, most of the hard work is already done.

A function that calls X function

I want to basically turn a list element into a function with
the do function. This way any pre-written funcction i can call by just use a
do(list[x]).
What im trying to do is a function that takes away the quotes of a list element and then executes the function that is in that list element.
def func():
print "python"
def func1():
print "is"
def func2():
print "awesome"
def do(fun):
fun()
#I think the problem is here
funs = ['func()','func1()','func2()']
print ''.join(funs[0])
do(''.join(funs[0]))
Edit:
What im trying to do is a function that takes away the quotes of a
list element and then executes the function that is in that list
element

You don't need the extra functions, and you don't need to turn them into a string either:
def func():
print "python"
def func1():
print "is"
def func2():
print "awesome"
funcs = [func, func1, func2]
for function in funcs:
function()

Well, it basically works like this. Note that the list contains the functions themselves, not a string.
def func():
print "python"
def func1():
print "is"
def func2():
print "awesome"
def do(fun):
fun()
funcs = [func, func1, func2]
for function in funcs:
do(function)
Output:
python
is
awesome
EDIT: If you do want the list to contain the functions' names as strings, use eval():
funcs = ['func', 'func1', 'func2']
for function in funcs:
do(eval(function))

If you really want to execute arbitrarily named functions from a list of names in the current global/module scope then this will do:
NB: This does NOT use the potentially unsafe and dangerous eval():
Example:
def func():
return "python"
def func1():
return "is"
def func2():
return "awesome"
def do(func_name, *args, **kwargs):
f = globals().get(func_name, lambda : None)
if callable(f):
return f(*args, **kwargs)
funs = ["func", "func1", "func2"]
print "".join(funs[0])
print "".join(map(do, funs))
Output:
$ python foo.py
func
pythonisawesome
You can also individually call "named" functions:
>>> do(funs[0])
python
Note the implementation of do(). This could also be applied more generically on objects and other modules too swapping out globals() lookups.

As the people above have said the best way is to define a dictionary of functions, as functions are objects in python this is possible
def One():
pass
def Two():
pass
functions = {"ONE":One, "TWO":Two}
You can then call it like this:
functions[input]()
If you want to give true control to the user (And I DO NOT recommend doing this) you could use the eval function.
eval(input+"()")

How can a decorator pass variables into a function without changing its signature?

Let me first acknowledge that what I want to do may be considered anything from silly to evil, but I want to find out if I can do it in Python anyway.
Let's say I have a function decorator that takes keyword arguments defining variables, and I want to access those variables in the wrapped function. I might do something like this:
def more_vars(**extras):
def wrapper(f):
#wraps(f)
def wrapped(*args, **kwargs):
return f(extras, *args, **kwargs)
return wrapped
return wrapper
Now I can do something like:
#more_vars(a='hello', b='world')
def test(deco_vars, x, y):
print(deco_vars['a'], deco_vars['b'])
print(x, y)
test(1, 2)
# Output:
# hello world
# 1 2
The thing I don't like about this is that when you use this decorator, you have to change the call signature of the function, adding the extra variable in addition to slapping on the decorator. Also, if you look at the help for the function, you see an extra variable that you're not expected to use when calling the function:
help(test)
# Output:
# Help on function test in module __main__:
#
# test(deco_vars, x, y)
This makes it look like the user is expected to call the function with 3 parameters, but obviously that won't work. So you'd have to also add a message to the docstring indicating that the first parameter isn't part of the interface, it's just an implementation detail and should be ignored. That's kind of crappy, though. Is there any way to do this without hanging these variables on something in the global scope? Ideally, I'd like it to look like the following:
#more_vars(a='hello', b='world')
def test(x, y):
print(a, b)
print(x, y)
test(1, 2)
# Output:
# hello world
# 1 2
help(test)
# Output:
# Help on function test in module __main__:
#
# test(x, y)
I am content with a Python 3 only solution if one exists.

You could do this with some trickery that inserts the variables passed to the decorator into the function's local variables:
import sys
from functools import wraps
from types import FunctionType
def is_python3():
return sys.version_info >= (3, 0)
def more_vars(**extras):
def wrapper(f):
#wraps(f)
def wrapped(*args, **kwargs):
fn_globals = {}
fn_globals.update(globals())
fn_globals.update(extras)
if is_python3():
func_code = '__code__'
else:
func_code = 'func_code'
call_fn = FunctionType(getattr(f, func_code), fn_globals)
return call_fn(*args, **kwargs)
return wrapped
return wrapper
#more_vars(a="hello", b="world")
def test(x, y):
print("locals: {}".format(locals()))
print("x: {}".format(x))
print("y: {}".format(y))
print("a: {}".format(a))
print("b: {}".format(b))
if __name__ == "__main__":
test(1, 2)
Can you do this? Sure! Should you do this? Probably not!
(Code available here.)

EDIT: answer edited for readability. Latest answer is on top, original follows.
If I understand well
you want the new arguments to be defined as keywords in the #more_vars decorator
you want to use them in the decorated function
and you want them to be hidden to the normal users (the exposed signature should still be the normal signature)
Have a look at the #with_partial decorator in my library makefun. It provides this functionality out of the box:
from makefun import with_partial
#with_partial(a='hello', b='world')
def test(a, b, x, y):
"""Here is a doc"""
print(a, b)
print(x, y)
It yields the expected output and the docstring is modified accordingly:
test(1, 2)
help(test)
yields
hello world
1 2
Help on function test in module <...>:
test(x, y)
<This function is equivalent to 'test(x, y, a=hello, b=world)', see original 'test' doc below.>
Here is a doc
To answer the question in your comment, the function creation strategy in makefun is exactly the same than the one in the famous decorator library: compile + exec. No magic here, but decorator has been using this trick for years in real-world applications so it is quite solid. See def _make in the source code.
Note that the makefun library also provides a partial(f, *args, **kwargs) function if you want to create the decorator yourself for some reason (see below for inspiration).
If you wish to do this manually, this is a solution that should work as you expect, it relies on the wraps function provided by makefun, to modify the exposed signature.
from makefun import wraps, remove_signature_parameters
def more_vars(**extras):
def wrapper(f):
# (1) capture the signature of the function to wrap and remove the invisible
func_sig = signature(f)
new_sig = remove_signature_parameters(func_sig, 'invisible_args')
# (2) create a wrapper with the new signature
#wraps(f, new_sig=new_sig)
def wrapped(*args, **kwargs):
# inject the invisible args again
kwargs['invisible_args'] = extras
return f(*args, **kwargs)
return wrapped
return wrapper
You can test that it works:
#more_vars(a='hello', b='world')
def test(x, y, invisible_args):
a = invisible_args['a']
b = invisible_args['b']
print(a, b)
print(x, y)
test(1, 2)
help(test)
You can even make the decorator definition more compact if you use decopatch to remove the useless level of nesting:
from decopatch import DECORATED
from makefun import wraps, remove_signature_parameters
#function_decorator
def more_vars(f=DECORATED, **extras):
# (1) capture the signature of the function to wrap and remove the invisible
func_sig = signature(f)
new_sig = remove_signature_parameters(func_sig, 'invisible_args')
# (2) create a wrapper with the new signature
#wraps(f, new_sig=new_sig)
def wrapped(*args, **kwargs):
kwargs['invisible_args'] = extras
return f(*args, **kwargs)
return wrapped
Finally, if you rather do not want to depend on any external library, the most pythonic way to do it is to create a function factory (but then you cannot have this as a decorator):
def make_test(a, b, name=None):
def test(x, y):
print(a, b)
print(x, y)
if name is not None:
test.__name__ = name
return test
test = make_test(a='hello', b='world')
test2 = make_test(a='hello', b='there', name='test2')
I'm the author of makefun and decopatch by the way ;)

It sounds like your only problem is that help is showing the signature of the raw test as the signature of the wrapped function, and you don't want it to.
The only reason that's happening is that wraps (or, rather, update_wrapper, which wraps calls) explicitly copies this from the wrappee to the wrapper.
You can decide exactly what you do and don't want to copy. If what you want to do differently is simple enough, it's just a matter of filtering stuff out of the default WRAPPER_ASSIGNMENTS and WRAPPER_UPDATES. If you want to change other stuff, you may need to fork update_wrapper and use your own version—but functools is one of those modules that has a link to the source right at the top of the docs, because it's meant to be used as readable sample code.
In your case, it may just be a matter of wraps(f, updated=[]), or you may want to do something fancy, like use inspect.signature to get the signature of f, and modify it to remove the first parameter, and build a wrapper explicitly around that to fool even the inspect module.

I've found a solution to this problem, although the solution is by most standards almost certainly worse than the problem itself. With some clever rewriting of the decorated function's bytecode, you can redirect all references to variables of a given name to a new closure you can dynamically create for the function. This solution only works for the standard CPython, and I have only tested it with 3.7.
import inspect
from dis import opmap, Bytecode
from types import FunctionType, CodeType
def more_vars(**vars):
'''Decorator to inject more variables into a function.'''
def wrapper(f):
code = f.__code__
new_freevars = code.co_freevars + tuple(vars.keys())
new_globals = [var for var in code.co_names if var not in vars.keys()]
new_locals = [var for var in code.co_varnames if var not in vars.keys()]
payload = b''.join(
filtered_bytecode(f, new_freevars, new_globals, new_locals))
new_code = CodeType(code.co_argcount,
code.co_kwonlyargcount,
len(new_locals),
code.co_stacksize,
code.co_flags & ~inspect.CO_NOFREE,
payload,
code.co_consts,
tuple(new_globals),
tuple(new_locals),
code.co_filename,
code.co_name,
code.co_firstlineno,
code.co_lnotab,
code.co_freevars + tuple(vars.keys()),
code.co_cellvars)
closure = tuple(get_cell(v) for (k, v) in vars.items())
return FunctionType(new_code, f.__globals__, f.__name__, f.__defaults__,
(f.__closure__ or ()) + closure)
return wrapper
def get_cell(val=None):
'''Create a closure cell object with initial value.'''
# If you know a better way to do this, I'd like to hear it.
x = val
def closure():
return x # pragma: no cover
return closure.__closure__[0]
def filtered_bytecode(func, freevars, globals, locals):
'''Get the bytecode for a function with adjusted closed variables
Any references to globlas or locals in the bytecode which exist in the
freevars are modified to reference the freevars instead.
'''
opcode_map = {
opmap['LOAD_FAST']: opmap['LOAD_DEREF'],
opmap['STORE_FAST']: opmap['STORE_DEREF'],
opmap['LOAD_GLOBAL']: opmap['LOAD_DEREF'],
opmap['STORE_GLOBAL']: opmap['STORE_DEREF']
}
freevars_map = {var: idx for (idx, var) in enumerate(freevars)}
globals_map = {var: idx for (idx, var) in enumerate(globals)}
locals_map = {var: idx for (idx, var) in enumerate(locals)}
for instruction in Bytecode(func):
if instruction.opcode not in opcode_map:
yield bytes([instruction.opcode, instruction.arg or 0])
elif instruction.argval in freevars_map:
yield bytes([opcode_map[instruction.opcode],
freevars_map[instruction.argval]])
elif 'GLOBAL' in instruction.opname:
yield bytes([instruction.opcode,
globals_map[instruction.argval]])
elif 'FAST' in instruction.opname:
yield bytes([instruction.opcode,
locals_map[instruction.argval]])
This behaves exactly as I wanted:
In [1]: #more_vars(a='hello', b='world')
...: def test(x, y):
...: print(a, b)
...: print(x, y)
...:
In [2]: test(1, 2)
hello world
1 2
In [3]: help(test)
Help on function test in module __main__:
test(x, y)
This is almost certainly not ready for production use. I would be surprised if there weren't edge cases that behave unexpectedly, and possibly even segfault. I'd probably file this under the "educational curiosity" heading.

Is there a way to get the function a decorator has wrapped?

Suppose I have
#someDecorator
def func():
'''this function does something'''
print 1
Now, the object func is an instance of someDecorator. Is there some way I can access the function it holds, i.e something like func.getInnerFunction().
For instance, if I need to retrieve the doc string of func().

See functools.wraps: http://docs.python.org/library/functools.html. The decorator gets the name and doc string of the original function. You use it like this:
def decorator(f):
#functools.wraps(f)
def wrapper():
....

SilentGhost and sri have partial answers for how to deal with this. But the general answer is no: there is no way to get the "wrapped" function out of a decorated function because there is no requirement that the decorator wrap the function in the first place. It may very well have returned an entirely unrelated function, and any references to your original may have already been garbage collected.

Are you looking for something along these lines?
>>> def dec(f):
def inner():
print(f.__doc__)
return inner
>>> #dec
def test():
"""abc"""
print(1)
>>> test()
abc
You're passing function explicitly to the decorator, of course you can access it.

You can attach the wrapped function to the inner function
In [1]: def wrapper(f):
...: def inner():
...: print "inner"
...: inner._orig = f
...: return inner
...:
In [2]: #wrapper
...: def foo():
...: print "foo"
...:
...:
In [3]: foo()
inner
In [4]: foo._orig()
foo

You can try using the undecorated library:
With your example func, you could simply do this to return the original function:
undecorated(func)