Suppose I have a generic function f. I want to programmatically create a function f2 that behaves the same as f, but has a customized signature.
More detail
Given a list l and and dictionary d I want to be able to:
Set the non-keyword arguments of f2 to the strings in l
Set the keyword arguments of f2 to the keys in d and the default values to the values of d
ie. Suppose we have
l = ["x", "y"]
d = {"opt": None}
def f(*args, **kwargs):
# My code
Then I would want a function with signature:
def f2(x, y, opt=None):
# My code
A specific use case
This is just a simplified version of my specific use case. I am giving this as an example only.
My actual use case (simplified) is as follows. We have a generic initiation function:
def generic_init(self, *args, **kwargs):
"""Function to initiate a generic object"""
for name, arg in zip(self.__init_args__, args):
setattr(self, name, arg)
for name, default in self.__init_kw_args__.items():
if name in kwargs:
setattr(self, name, kwargs[name])
else:
setattr(self, name, default)
We want to use this function in a number of classes. In particular, we want to create a function __init__ that behaves like generic_init, but has the signature defined by some class variables at creation time:
class my_class:
__init_args__ = ["x", "y"]
__kw_init_args__ = {"my_opt": None}
__init__ = create_initiation_function(my_class, generic_init)
setattr(myclass, "__init__", __init__)
We want create_initiation_function to create a new function with the signature defined using __init_args__ and __kw_init_args__. Is it possible to write create_initiation_function?
Please note:
If I just wanted to improve the help, I could set __doc__.
We want to set the function signature on creation. After that, it doesn't need to be changed.
Instead of creating a function like generic_init, but with a different signature we could create a new function with the desired signature that just calls generic_init
We want to define create_initiation_function. We don't want to manually specify the new function!
Related
Preserving signatures of decorated functions: This is how to preserve a signature when decorating a function. We need to be able to set the signature to an arbitrary value
From PEP-0362, there actually does appear to be a way to set the signature in py3.3+, using the fn.__signature__ attribute:
from inspect import signature
from functools import wraps
def shared_vars(*shared_args):
"""Decorator factory that defines shared variables that are
passed to every invocation of the function"""
def decorator(f):
#wraps(f)
def wrapper(*args, **kwargs):
full_args = shared_args + args
return f(*full_args, **kwargs)
# Override signature
sig = signature(f)
sig = sig.replace(parameters=tuple(sig.parameters.values())[1:])
wrapper.__signature__ = sig
return wrapper
return decorator
Then:
>>> #shared_vars({"myvar": "myval"})
>>> def example(_state, a, b, c):
>>> return _state, a, b, c
>>> example(1,2,3)
({'myvar': 'myval'}, 1, 2, 3)
>>> str(signature(example))
'(a, b, c)'
Note: the PEP is not exactly right; Signature.replace moved the params from a positional arg to a kw-only arg.
For your usecase, having a docstring in the class/function should work -- that will show up in help() okay, and can be set programmatically (func.__doc__ = "stuff").
I can't see any way of setting the actual signature. I would have thought the functools module would have done it if it was doable, but it doesn't, at least in py2.5 and py2.6.
You can also raise a TypeError exception if you get bad input.
Hmm, if you don't mind being truly vile, you can use compile()/eval() to do it. If your desired signature is specified by arglist=["foo","bar","baz"], and your actual function is f(*args, **kwargs), you can manage:
argstr = ", ".join(arglist)
fakefunc = "def func(%s):\n return real_func(%s)\n" % (argstr, argstr)
fakefunc_code = compile(fakefunc, "fakesource", "exec")
fakeglobals = {}
eval(fakefunc_code, {"real_func": f}, fakeglobals)
f_with_good_sig = fakeglobals["func"]
help(f) # f(*args, **kwargs)
help(f_with_good_sig) # func(foo, bar, baz)
Changing the docstring and func_name should get you a complete solution. But, uh, eww...
I wrote a package named forge that solves this exact problem for Python 3.5+:
With your current code looking like this:
l=["x", "y"]
d={"opt":None}
def f(*args, **kwargs):
#My code
And your desired code looking like this:
def f2(x, y, opt=None):
#My code
Here is how you would solve that using forge:
f2 = forge.sign(
forge.arg('x'),
forge.arg('y'),
forge.arg('opt', default=None),
)(f)
As forge.sign is a wrapper, you could also use it directly:
#forge.sign(
forge.arg('x'),
forge.arg('y'),
forge.arg('opt', default=None),
)
def func(*args, **kwargs):
# signature becomes: func(x, y, opt=None)
return (args, kwargs)
assert func(1, 2) == ((), {'x': 1, 'y': 2, 'opt': None})
Have a look at makefun, it was made for that (exposing variants of functions with more or less parameters and accurate signature), and works in python 2 and 3.
Your example would be written like this:
try: # python 3.3+
from inspect import signature, Signature, Parameter
except ImportError:
from funcsigs import signature, Signature, Parameter
from makefun import create_function
def create_initiation_function(cls, gen_init):
# (1) check which signature we want to create
params = [Parameter('self', kind=Parameter.POSITIONAL_OR_KEYWORD)]
for mandatory_arg_name in cls.__init_args__:
params.append(Parameter(mandatory_arg_name, kind=Parameter.POSITIONAL_OR_KEYWORD))
for default_arg_name, default_arg_val in cls.__opt_init_args__.items():
params.append(Parameter(default_arg_name, kind=Parameter.POSITIONAL_OR_KEYWORD, default=default_arg_val))
sig = Signature(params)
# (2) create the init function dynamically
return create_function(sig, generic_init)
# ----- let's use it
def generic_init(self, *args, **kwargs):
"""Function to initiate a generic object"""
assert len(args) == 0
for name, val in kwargs.items():
setattr(self, name, val)
class my_class:
__init_args__ = ["x", "y"]
__opt_init_args__ = {"my_opt": None}
my_class.__init__ = create_initiation_function(my_class, generic_init)
and works as expected:
# check
o1 = my_class(1, 2)
assert vars(o1) == {'y': 2, 'x': 1, 'my_opt': None}
o2 = my_class(1, 2, 3)
assert vars(o2) == {'y': 2, 'x': 1, 'my_opt': 3}
o3 = my_class(my_opt='hello', y=3, x=2)
assert vars(o3) == {'y': 3, 'x': 2, 'my_opt': 'hello'}
You can't do this with live code.
That is, you seem to be wanting to take an actual, live function that looks like this:
def f(*args, **kwargs):
print args[0]
and change it to one like this:
def f(a):
print a
The reason this can't be done--at least without modifying actual Python bytecode--is because these compile differently.
The former results in a function that receives two parameters: a list and a dict, and the code you're writing operates on that list and dict. The second results in a function that receives one parameter, and which is accessed as a local variable directly. If you changed the function "signature", so to speak, it'd result in a function like this:
def f(a):
print a[0]
which obviously wouldn't work.
If you want more detail (though it doesn't really help you), a function that takes an *args or *kwargs has one or two bits set in f.func_code.co_flags; you can examine this yourself. The function that takes a regular parameter has f.func_code.co_argcount set to 1; the *args version is 0. This is what Python uses to figure out how to set up the function's stack frame when it's called, to check parameters, etc.
If you want to play around with modifying the function directly--if only to convince yourself that it won't work--see this answer for how to create a code object and live function from an existing one to modify bits of it. (This stuff is documented somewhere, but I can't find it; it's nowhere in the types module docs...)
That said, you can dynamically change the docstring of a function. Just assign to func.__doc__. Be sure to only do this at load time (from the global context or--most likely--a decorator); if you do it later on, tools that load the module to examine docstrings will never see it.
Maybe I didn't understand the problem well, but if it's about keeping the same behavior while changing the function signature, then you can do something like :
# define a function
def my_func(name, age) :
print "I am %s and I am %s" % (name, age)
# label the function with a backup name
save_func = my_func
# rewrite the function with a different signature
def my_func(age, name) :
# use the backup name to use the old function and keep the old behavior
save_func(name, age)
# you can use the new signature
my_func(35, "Bob")
This outputs :
I am Bob and I am 35
We want create_initiation_function to change the signature
Please don't do this.
We want to use this function in a number of classes
Please use ordinary inheritance.
There's no value in having the signature "changed" at run time.
You're creating a maintenance nightmare. No one else will ever bother to figure out what you're doing. They'll simply rip it out and replace it with inheritance.
Do this instead. It's simple and obvious and makes your generic init available in all subclasses in an obvious, simple, Pythonic way.
class Super( object ):
def __init__( self, *args, **kwargs ):
# the generic __init__ that we want every subclass to use
class SomeSubClass( Super ):
def __init__( self, this, that, **kwdefaults ):
super( SomeSubClass, self ).__init__( this, that, **kwdefaults )
class AnotherSubClass( Super ):
def __init__( self, x, y, **kwdefaults ):
super( AnotherSubClass, self ).__init__( x, y, **kwdefaults )
Edit 1: Answering new question:
You ask how you can create a function with this signature:
def fun(a, b, opt=None):
pass
The correct way to do that in Python is thus:
def fun(a, b, opt=None):
pass
Edit 2: Answering explanation:
"Suppose I have a generic function f. I want to programmatically create a function f2 that behaves the same as f, but has a customised signature."
def f(*args, **kw):
pass
OK, then f2 looks like so:
def f2(a, b, opt=None):
f(a, b, opt=opt)
Again, the answer to your question is so trivial, that you obviously want to know something different that what you are asking. You really do need to stop asking abstract questions, and explain your concrete problem.
Related
I'm creating a wrapper for a function with functools.wraps. My wrapper has the effect of overriding a default parameter (and it doesn't do anything else):
def add(*, a=1, b=2):
"Add numbers"
return a + b
#functools.wraps(add)
def my_add(**kwargs):
kwargs.setdefault('b', 3)
return add(**kwargs)
This my_add definition behaves the same as
#functools.wraps(add)
def my_add(*, a=1, b=3):
return add(a=a, b=b)
except that I didn't have to manually type out the parameter list.
However, when I run help(my_add), I see the help string for add, which has the wrong function name and the wrong default argument for the parameter b:
add(*, a=1, b=2)
Add numbers
How can I override the function name and the default argument in this help() output?
(Or, is there a different way to define my_add, using for example some magic function my_add = magic(add, func_name='my_add', kwarg_defaults={'b': 3}) that will do what I want?)
Let me try and explain what happens.
When you call the help functions, this is going to request information about your function using the inspect module. Therefore you have to change the function signature, in order to change the default argument.
Now this is not something that is advised, or often preferred, but who cares about that right? The provided solution is considered hacky and probably won't work for all versions of Python. Therefore you might want to reconsider how important the help function is... Any way let's start with some explanation on how it was done, followed by the code and test case.
Copying functions
Now the first thing we will do is copy the entire function, this is because I only want to change the signature of the new function and not the original function. This decouples the new my_add signature (and default values) from the original add function.
See:
How to create a copy of a python function
How can I make a deepcopy of a function in Python?
For ideas of how to do this (I will show my version in a bit).
Copying / updating signature
The next step is to get a copy of the function signature, for that this post was very useful. Except for the part where we have to adjust the signature parameters to match the new keyword default arguments.
For that we have to change the value of a mappingproxy, which we can see when running the debugger on the return value of inspect.signature(g). Now so far this can only be done by changing the private variables (the values with leading underscores _private). Therefore this solution will be considered hacky and is not guaranteed to withstand possible updates. That said, let's see the solution!
Full code
import inspect
import types
import functools
def update_func(f, func_name='', update_kwargs: dict = None):
"""Based on http://stackoverflow.com/a/6528148/190597 (Glenn Maynard)"""
g = types.FunctionType(
code=f.__code__,
globals=f.__globals__.copy(),
name=f.__name__,
argdefs=f.__defaults__,
closure=f.__closure__
)
g = functools.update_wrapper(g, f)
g.__signature__ = inspect.signature(g)
g.__kwdefaults__ = f.__kwdefaults__.copy()
# Adjust your arguments
for key, value in (update_kwargs or {}).items():
g.__kwdefaults__[key] = value
g.__signature__.parameters[key]._default = value
g.__name__ = func_name or g.__name__
return g
def add(*, a=1, b=2):
"Add numbers"
return a + b
my_add = update_func(add, func_name="my_add", update_kwargs=dict(b=3))
Example
if __name__ == '__main__':
a = 2
print("*" * 50, f"\nMy add\n", )
help(my_add)
print("*" * 50, f"\nOriginal add\n", )
help(add)
print("*" * 50, f"\nResults:"
f"\n\tMy add : a = {a}, return = {my_add(a=a)}"
f"\n\tOriginal add: a = {a}, return = {add(a=a)}")
Output
**************************************************
My add
Help on function my_add in module __main__:
my_add(*, a=1, b=3)
Add numbers
**************************************************
Original add
Help on function add in module __main__:
add(*, a=1, b=2)
Add numbers
**************************************************
Results:
My add : a = 2, return = 5
Original add: a = 2, return = 4
Usages
f: is the function that you want to update
func_name: is optionally the new name of the function (if empty, keeps the old name)
update_kwargs: is a dictionary containing the key and value of the default arguments that you want to update.
Notes
The solution is using copy variables to make full copies of dictionaries, such that there is no impact on the original add function.
The _default value is a private variable, and can be changed in future releases of python.
I have the follow init function who receives a lot of args to run the class (the args are default values if the user don't input anything or can be a value inputed by the user). What's the most elegant way to reduce the number of variables (not show a lot of args in the init) without lose readability? Use the *args function (like def__init__(self, *args))?
class World(object):
def __init__(self, grid_size=(GRID_WIDTH, GRID_HEIGHT),
cell_size=(CELL_WIDTH, CELL_HEIGHT),
obstacles_position= OBSTACLES,
recharge_position= RECHARGE_ZONE,
treadmill_position= TREADMILL_ZONE,
workers_positions= WORKERS_POS,
delivery_positions= DELIVERY_ZONE):
# some code bellow
def main():
# init some libraries
world = worldGrid()
# Do a while loop with the input variables from the world class
if __name__ = '__main__':
main()
Obs: I'm using Python 3+
In my opinion, you should probably stick with all of the function parameters in the function header (as you currently have it). This makes your code more readable, allows Python to tell you which arguments you may have omitted, plays nicely with Python's built-in help() method, allows third-party IDE code hinting, etc., etc...
If you really want to shorten the function header, you could use *args and **kwargs which will take any variadic arguments, e.g.:
def func(self, *args, **kwargs):
print("args:", args)
print("kwargs:", kwargs)
Usage would look like this:
>>> func(1, 2, 3, one="one", two="two")
args: (2, 3)
kwargs: {'one': 'one', 'two': 'two'}
Therefore, you could theoretically refactor your class to look something like below. This code doesn't handle default values or any error checking at all -- it just sets any keyword-arguments as attributes on the class itself:
class World(object):
def __init__(self, **kwargs):
for key, value in kwargs.items():
setattr(self, key, value)
And usage:
>>> w = World(one=1, two=2, three=3)
>>> w.one
1
>>> w.two
2
>>> w.three
3
I'm learning to use python decorator.
def my_dcrtr(fun):
def new_fun():
return fun()
return new_fun
I realize the decorated function 'fun' acts like a black box inside the decorator. I can choose to use fun() or not at all inside new_fun. However, I don't know whether I can break into 'fun' and interact with fun's local scope inside the new_fun?
e.g. I'm trying to make a toy Remote Procedural Call (RPC) with python.
def return_locals_rpc_decorator(fun):
def decorated_fun(*args, **kw):
local_args = fun(*args, **kw)
# pickle the local_args and send it to server
# server unpickle and doing the RPC
# fetch back server results and unpickle to results
return rpc_results
return decorated_fun
#return_locals_rpc_decorator
def rpc_fun(a, b, c=3):
return locals() # This looks weird. how can I make this part of the decorator?
print(rpc_fun(2, 1, 6))
In this example, I try to get rpc_fun's argument list at runtime with the 'locals()' command. Then send it to server to execute. Instead of letting rpc_fun returns its locals(), is it possible to use the decorator to retrieve decorated function's argument space?
You can use function annotations for Python3:
def return_locals_rpc_decorator(fun):
def decorated_fun(*args, **kw):
local_args = fun(*args, **kw)
print(local_args)
fun_parameters = fun.__annotations__
final_parameters = {a:list(args)[int(b[-1])-1] for a, b in fun_parameters.items() if a != 'return'}
return final_parameters
return decorated_fun
#return_locals_rpc_decorator
def my_funct(a:"val1", b:"val2", c:"val3") -> int:
return a + b + c
print(my_funct(10, 20, 30))
Output:
60
{'a': 10, 'b': 20, 'c': 30}
In this way, you are using the wrapper function decorated_fun to access the decorated function's parameters and further information specified by the annotation. I changed the parameter descriptions in the annotations so that each string value would end in a digit that could be used to index args. However, if you do not want to change the parameter descriptions in the annotations, you can sort via ending character.
Edit: the code in the body of my_funct is executed when called in the wrapper function (decorated_fun), since the args, declared in the scope of decorated_fun is passed to and unpacked in local_args.
I have one weird problem. I want to pass/change an argument of a function which itself is passed as an argument in some other function. See the below code for more details
def generic_method(selector_type='CSS', selector=None, parent_element=None, postfunc=None):
# Do your stuff and get value of attr_value
print "Doing Stuff"
attr_value = '$123.70'
print "Post-Processing Step"
if postfunc:
attr_value = postfunc(attrval=attr_value)
return attr_value
# The 2 methods below are in separate file
from functools import partial
def method_in_bot():
p, q, r = 11, 12, 13
postfunc = partial(post_processing, 12, p, q, r, post=23)
value = generic_method('XPATH', '.class-name', 'parent_element', postfunc)
return value
def post_processing(y=None, *args, **kwargs):
attr_value = kwargs.get('attrval', 'None')
if attr_value:
return attr_value.split('$')
return []
So I passed my post_processing method along with all its parameters to my generic_method by using functools's partial and also passed a new variable attrval to my post_processing method. But what is more desirable is to pass or assign attr_value directly to variable y to post_processing.
I was looking for ways to modify functions parameters during run-time. I was searching over the net and found out that their is a inspect library in python which tells you about the arguments passed to the function. Can it be used in this case.
In Python 3, you could do def post_processing(*args, y=None, **kwargs): and that would be it. With Python 2 you have to find a different technique than partial. Or maybe subclass it as in implementing functools.partial that prepends additional arguments .
The Question
I want to be able to initialize an object with a function that references the instance's attributes. What I want I tried to capture in this snippet, which produces a NameError: "global name 'self' is not defined":
class Test(object):
def __init__(self, function = None):
self.dicty = {1:{'height': 4, 'width': 2}, 2:{'height': 1, 'width': 2} }
if function == None:
self.function = lambda x : self.dicty[x]['height']
else:
self.function = function
if __name__ == '__main__':
def func1(x):
return self.dicty[x]['width']
def func2(x):
return self.dicty[x]['width']**2
G = Test(function = func1)
H = Test(function = func2)
I could solve the problem by creating a bunch of subclasses to Test, but that doesn't seem readable.
The Motivation
I am using NetworkX to do Python modeling and experiments. I was looking at the classic Albert-Barabasi Model and creating subclasses of the DiGraph class that included a Preference(self, node), Attachment(self, parent, child), and then a Grow(self, max_allowable_nodes). Instead of creating a whole bunch of subclasses like I mentioned before, I would love to be able to create an instance that modifies preference(). This would allow me to run numerical experiments without my code looking too much like Frankenstein. Looking forward to learning something new.
Edit:
Didn't know about the types class or the general idea of reflection. Obviously, still pretty new here. Really appreciate everyone answering my questions and pointing me in the right direction!
Given that the lambda you create in your __init__ refers to the instance (self), it looks like you want to attach a method to your instance, whereas here you're attaching a function. You need to create a method from the function and attach it to the instance:
import types
class Test(object):
def __init__(self, function = None):
self.dicty = {1:{'height': 4, 'width': 2}, 2:{'height': 1, 'width': 2} }
if function == None:
function = lambda self, x: self.dicty[x]['height']
self.function = types.MethodType(function, self)
A method is basically a function that is always passed the instance as the first argument, so you need to ensure any function you pass into your initialiser has self as the initial argument.
>>> t1 = Test()
>>> t1.function(1)
4
>>> t2 = Test(lambda self, x: self.dicty[x]['width'])
>>> t2.function(1)
2
When you define func1, there is no such thing as self. It's not an argument to the function, and it's not in any higher scope.
You could, instead, define a function that takes the dict you use as an argument and operates on that. In the Test class, you can then call the function on self.dicty. This would require you to change your lambda to also take dicty and x instead of just x.
def func1(dicty, x):
return dicty[x]['width']
...and in Test...
class Test(object):
# ... current code but with lambda tweak:
# lambda dicty, x: dicty[x]['height']
def do_something(self, x):
self.function(self.dicty, x)
Without seeing the rest of your code, it's hard to know what further simplifications you could make. But since all the functions seem to be using dicty[x] anyway, you could just write them to take that directly.