I would like to create functions in my Python3 code, making use of data supplied during run-time. I'm stuck on how I can write a function along the lines of
def foo(varname1, varname2):
return varname1 + varname2
where the strings varname1 and varname2 that give the parameter names are specified as arguments to some constructor function, e.g.:
def makeNewFooFunc(varname1, varname2):
# do magic
return foo
fooFunc = makeNewFooFunc('first', 'second')
print(fooFunc(first=1, second=2))
# should return 3
what would be the #do magic step? Is this a job for eval, or is there an alternative?
You don't need to write a function like that.
just use **kwargs
def foo_func(**kwargs):
return sum(kwargs.values())
foo_func(any_name=1, any_name_2=2)
but if you still need to do what you want, you can try
def make_new_func(var_name_1, var_name_2):
def foo(**kwargs):
# make sure in kwargs only expected parameters
assert set(kwargs) == {var_name_1, var_name_2}
return kwargs[var_name_1] + kwargs[var_name_2]
return foo
foo_func = make_new_func('a', 'b')
foo_func(a=1, b=2)
Say I have a class given by:
class MyClass:
def __init__(self, **kwargs):
self.kwargs = kwargs
def __repr__(self):
return "<%s: %r>" % (self.__class__.__name__, self.kwargs)
__repr__ isn't really important for core functionality, but occasionally gets called for logging, shows up in stack traces, and such, so I'd like to unit-test it.
The problem is similar to that faced when using doctest, but I'd rather keep any complexity (like sorting) in the test function and not in __repr__.
Right now I'm using eval and re to extract the dictionary from the repr() call, but I wanted to check if there were non-eval alternatives out there that people used.
def test_repr():
retval = repr(MyClass(a=1, b=2, c=3))
match = re.match("^<MyClass: ({.*})>\Z", retval)
assert match
assert eval(match.group(1)) == dict(a=1, b=2, c=3)
You only need to check that the kwargs dictionary is correctly being represented in the output, so just pass zero or one keyword arguments:
>>> repr(MyClass(foo='bar')) == "<MyClass: {'foo': 'bar'}>"
True
>>> repr(MyClass()) == '<MyClass: {}>'
True
Then ordering doesn't matter at all.
If you decide to stick with evaluating the extracted dictionary, use ast.literal_eval instead of vanilla eval. I would also use a slice rather than re, as you know the expected format:
>>> '<MyClass: {}>'[10:-1]
'{}'
I have created 5 instances of a class and need to call another class with arguments from the instances like so:
class Class2:
...
def __init__(self, arg1, arg2):
self.time = 0
...
def run_loop(self, arg, *args):
...
while True:
fn_with_args(*args)
...
if self.time == 35:
fn_print_exit()
break
def fn_with_args(self, *args):
for i in args:
if i != "none":
fn_other()
clas1_ob1 = Clas1(arg1, arg2)
clas1_ob2 = Clas1(arg1, arg2)
clas1_ob3 = Clas1(arg1, arg2)
clas1_ob4 = Clas1(arg1, arg2)
clas1_ob5 = Clas1(arg1, arg2)
clas2_ob1 = Clas2(arg)
clas2_ob1.run_loop(clas1_ob1, "none")
clas2_ob1.run_loop(clas1_ob2, clas1_ob1)
clas2_ob1.run_loop(clas1_ob3, clas1_ob1, clas1_ob2)
clas2_ob1.run_loop(clas1_ob4, clas1_ob1, clas1_ob2, clas1_ob3)
clas2_ob1.run_loop(clas1_ob5, clas1_ob1, clas1_ob2, clas1_ob3, clas1_ob4)
Its obviously pretty ugly, but it does work at the moment. However, I would of course prefer to not have to write out each instance and then write out each call. I would prefer to run for loops in both cases.
I could of course, use lists, but when I tried it did not work because I could not iterate through lists of objects in the function fn_with_args(). *args however, can be iterated through. So my question presently is 'how can I pass *args into the run_loop() call so I can simply call it once?'
Or if there is a way I can iterate over the list of objects I suppose that is an option too however I would prefer to not do this because it requires many more lines of code and a fair bit of re-structuring.
Any and all input appreciated, let me know if I need explain more,
Thanks
PS: I realize also that I could simply pass the elements of the object into a list but this presents different problems for the program and seems counter to oop structure.
EDIT--->
Example:
number = 5
clas2_ob1 = Clas2(arg)
for i in range(number):
clas1_ob1 = Clas1(arg1, arg2)
clas2_ob1.run_loop(clas1_ob1, "none")
#This is the structure I want but this will obviously just overwrite 5 times
#The arguments in this case also become incorrect
It's not hard to do what you want using a slice of a list of potential arguments:
class1_objs = [Clas1(arg1, arg2) for _ in range(5)] # create Class1 objects in a list
class2_obj = Clas2(arg)
for i, obj in enumerate(class1_objs): # loop over objects and indexes with enumerate
class2_obj.runloop(obj, *class1_objs[:i] or ["none"]) # use index to slice the list
I'd suggest however that you might want to redesign your method signatures to simply accept two arguments, an object and a sequence. The code you've shown packs and unpacks the arguments over and over, which is somewhat wasteful if you don't need it. Just get rid of the *s in all of the function definitions and their calls and you'll have simpler, more efficient code.
How can I keep help strings in functions to be visible after applying a decorator?
Right now the doc string is (partially) replaced with that of the inner function of the decorator.
def deco(fn):
def x(*args, **kwargs):
return fn(*args, **kwargs)
x.func_doc = fn.func_doc
x.func_name = fn.func_name
return x
#deco
def y(a, b):
"""This is Y"""
pass
def z(c, d):
"""This is Z"""
pass
help(y) # 1
help(z) # 2
In the Y function, required arguments aren't shown in the help. The user may assume it takes any arguments, while actually it doesn't.
y(*args, **kwargs) <= y(a, b) is desired
This is Y
z(c, d)
This is Z
I use help() and dir() a lot, since it's faster than pdf manuals, and want to make reliable document strings for my library and tools, but this is an obstacle.
give the decorator module a peek. i believe it does exactly what you want.
In [1]: from decorator import decorator
In [2]: #decorator
...: def say_hello(f, *args, **kwargs):
...: print "Hello!"
...: return f(*args, **kwargs)
...:
In [3]: #say_hello
...: def double(x):
...: return 2*x
...:
and info says "double(x)" in it.
What you're requesting is very hard to do "properly", because help gets the function signature from inspect.getargspec which in turn gets it from introspection which cannot directly be fooled -- to do it "properly" would mean generating a new function object on the fly (instead of a simple wrapper function) with the right argument names and numbers (and default values). Extremely hard, advanced, black-magic bytecode hacking required, in other words.
I think it may be easier to do it by monkeypatching (never a pleasant prospect, but sometimes the only way to perform customization tasks that are otherwise so difficult as to prove almost impossible, like the one you require) -- replace the real inspect.getargspec with your own lookalike function which uses a look-aside table (mapping the wrapper functions you generate to the wrapped functions' argspecs and otherwise delegating to the real thing).
import functools
import inspect
realgas = inspect.getargspec
lookaside = dict()
def fakegas(f):
if f in lookaside:
return lookaside[f]
return realgas(f)
inspect.getargspec = fakegas
def deco(fn):
#functools.wraps(fn)
def x(*args, **kwargs):
return fn(*args, **kwargs)
lookaside[x] = realgas(fn)
return x
#deco
def x(a, b=23):
"""Some doc for x."""
return a + b
help(x)
This prints, as required:
Help on function x in module __main__:
x(a, b=23)
Some doc for x.
(END)
Suppose I have a generic function f. I want to programmatically create a function f2 that behaves the same as f, but has a customized signature.
More detail
Given a list l and and dictionary d I want to be able to:
Set the non-keyword arguments of f2 to the strings in l
Set the keyword arguments of f2 to the keys in d and the default values to the values of d
ie. Suppose we have
l = ["x", "y"]
d = {"opt": None}
def f(*args, **kwargs):
# My code
Then I would want a function with signature:
def f2(x, y, opt=None):
# My code
A specific use case
This is just a simplified version of my specific use case. I am giving this as an example only.
My actual use case (simplified) is as follows. We have a generic initiation function:
def generic_init(self, *args, **kwargs):
"""Function to initiate a generic object"""
for name, arg in zip(self.__init_args__, args):
setattr(self, name, arg)
for name, default in self.__init_kw_args__.items():
if name in kwargs:
setattr(self, name, kwargs[name])
else:
setattr(self, name, default)
We want to use this function in a number of classes. In particular, we want to create a function __init__ that behaves like generic_init, but has the signature defined by some class variables at creation time:
class my_class:
__init_args__ = ["x", "y"]
__kw_init_args__ = {"my_opt": None}
__init__ = create_initiation_function(my_class, generic_init)
setattr(myclass, "__init__", __init__)
We want create_initiation_function to create a new function with the signature defined using __init_args__ and __kw_init_args__. Is it possible to write create_initiation_function?
Please note:
If I just wanted to improve the help, I could set __doc__.
We want to set the function signature on creation. After that, it doesn't need to be changed.
Instead of creating a function like generic_init, but with a different signature we could create a new function with the desired signature that just calls generic_init
We want to define create_initiation_function. We don't want to manually specify the new function!
Related
Preserving signatures of decorated functions: This is how to preserve a signature when decorating a function. We need to be able to set the signature to an arbitrary value
From PEP-0362, there actually does appear to be a way to set the signature in py3.3+, using the fn.__signature__ attribute:
from inspect import signature
from functools import wraps
def shared_vars(*shared_args):
"""Decorator factory that defines shared variables that are
passed to every invocation of the function"""
def decorator(f):
#wraps(f)
def wrapper(*args, **kwargs):
full_args = shared_args + args
return f(*full_args, **kwargs)
# Override signature
sig = signature(f)
sig = sig.replace(parameters=tuple(sig.parameters.values())[1:])
wrapper.__signature__ = sig
return wrapper
return decorator
Then:
>>> #shared_vars({"myvar": "myval"})
>>> def example(_state, a, b, c):
>>> return _state, a, b, c
>>> example(1,2,3)
({'myvar': 'myval'}, 1, 2, 3)
>>> str(signature(example))
'(a, b, c)'
Note: the PEP is not exactly right; Signature.replace moved the params from a positional arg to a kw-only arg.
For your usecase, having a docstring in the class/function should work -- that will show up in help() okay, and can be set programmatically (func.__doc__ = "stuff").
I can't see any way of setting the actual signature. I would have thought the functools module would have done it if it was doable, but it doesn't, at least in py2.5 and py2.6.
You can also raise a TypeError exception if you get bad input.
Hmm, if you don't mind being truly vile, you can use compile()/eval() to do it. If your desired signature is specified by arglist=["foo","bar","baz"], and your actual function is f(*args, **kwargs), you can manage:
argstr = ", ".join(arglist)
fakefunc = "def func(%s):\n return real_func(%s)\n" % (argstr, argstr)
fakefunc_code = compile(fakefunc, "fakesource", "exec")
fakeglobals = {}
eval(fakefunc_code, {"real_func": f}, fakeglobals)
f_with_good_sig = fakeglobals["func"]
help(f) # f(*args, **kwargs)
help(f_with_good_sig) # func(foo, bar, baz)
Changing the docstring and func_name should get you a complete solution. But, uh, eww...
I wrote a package named forge that solves this exact problem for Python 3.5+:
With your current code looking like this:
l=["x", "y"]
d={"opt":None}
def f(*args, **kwargs):
#My code
And your desired code looking like this:
def f2(x, y, opt=None):
#My code
Here is how you would solve that using forge:
f2 = forge.sign(
forge.arg('x'),
forge.arg('y'),
forge.arg('opt', default=None),
)(f)
As forge.sign is a wrapper, you could also use it directly:
#forge.sign(
forge.arg('x'),
forge.arg('y'),
forge.arg('opt', default=None),
)
def func(*args, **kwargs):
# signature becomes: func(x, y, opt=None)
return (args, kwargs)
assert func(1, 2) == ((), {'x': 1, 'y': 2, 'opt': None})
Have a look at makefun, it was made for that (exposing variants of functions with more or less parameters and accurate signature), and works in python 2 and 3.
Your example would be written like this:
try: # python 3.3+
from inspect import signature, Signature, Parameter
except ImportError:
from funcsigs import signature, Signature, Parameter
from makefun import create_function
def create_initiation_function(cls, gen_init):
# (1) check which signature we want to create
params = [Parameter('self', kind=Parameter.POSITIONAL_OR_KEYWORD)]
for mandatory_arg_name in cls.__init_args__:
params.append(Parameter(mandatory_arg_name, kind=Parameter.POSITIONAL_OR_KEYWORD))
for default_arg_name, default_arg_val in cls.__opt_init_args__.items():
params.append(Parameter(default_arg_name, kind=Parameter.POSITIONAL_OR_KEYWORD, default=default_arg_val))
sig = Signature(params)
# (2) create the init function dynamically
return create_function(sig, generic_init)
# ----- let's use it
def generic_init(self, *args, **kwargs):
"""Function to initiate a generic object"""
assert len(args) == 0
for name, val in kwargs.items():
setattr(self, name, val)
class my_class:
__init_args__ = ["x", "y"]
__opt_init_args__ = {"my_opt": None}
my_class.__init__ = create_initiation_function(my_class, generic_init)
and works as expected:
# check
o1 = my_class(1, 2)
assert vars(o1) == {'y': 2, 'x': 1, 'my_opt': None}
o2 = my_class(1, 2, 3)
assert vars(o2) == {'y': 2, 'x': 1, 'my_opt': 3}
o3 = my_class(my_opt='hello', y=3, x=2)
assert vars(o3) == {'y': 3, 'x': 2, 'my_opt': 'hello'}
You can't do this with live code.
That is, you seem to be wanting to take an actual, live function that looks like this:
def f(*args, **kwargs):
print args[0]
and change it to one like this:
def f(a):
print a
The reason this can't be done--at least without modifying actual Python bytecode--is because these compile differently.
The former results in a function that receives two parameters: a list and a dict, and the code you're writing operates on that list and dict. The second results in a function that receives one parameter, and which is accessed as a local variable directly. If you changed the function "signature", so to speak, it'd result in a function like this:
def f(a):
print a[0]
which obviously wouldn't work.
If you want more detail (though it doesn't really help you), a function that takes an *args or *kwargs has one or two bits set in f.func_code.co_flags; you can examine this yourself. The function that takes a regular parameter has f.func_code.co_argcount set to 1; the *args version is 0. This is what Python uses to figure out how to set up the function's stack frame when it's called, to check parameters, etc.
If you want to play around with modifying the function directly--if only to convince yourself that it won't work--see this answer for how to create a code object and live function from an existing one to modify bits of it. (This stuff is documented somewhere, but I can't find it; it's nowhere in the types module docs...)
That said, you can dynamically change the docstring of a function. Just assign to func.__doc__. Be sure to only do this at load time (from the global context or--most likely--a decorator); if you do it later on, tools that load the module to examine docstrings will never see it.
Maybe I didn't understand the problem well, but if it's about keeping the same behavior while changing the function signature, then you can do something like :
# define a function
def my_func(name, age) :
print "I am %s and I am %s" % (name, age)
# label the function with a backup name
save_func = my_func
# rewrite the function with a different signature
def my_func(age, name) :
# use the backup name to use the old function and keep the old behavior
save_func(name, age)
# you can use the new signature
my_func(35, "Bob")
This outputs :
I am Bob and I am 35
We want create_initiation_function to change the signature
Please don't do this.
We want to use this function in a number of classes
Please use ordinary inheritance.
There's no value in having the signature "changed" at run time.
You're creating a maintenance nightmare. No one else will ever bother to figure out what you're doing. They'll simply rip it out and replace it with inheritance.
Do this instead. It's simple and obvious and makes your generic init available in all subclasses in an obvious, simple, Pythonic way.
class Super( object ):
def __init__( self, *args, **kwargs ):
# the generic __init__ that we want every subclass to use
class SomeSubClass( Super ):
def __init__( self, this, that, **kwdefaults ):
super( SomeSubClass, self ).__init__( this, that, **kwdefaults )
class AnotherSubClass( Super ):
def __init__( self, x, y, **kwdefaults ):
super( AnotherSubClass, self ).__init__( x, y, **kwdefaults )
Edit 1: Answering new question:
You ask how you can create a function with this signature:
def fun(a, b, opt=None):
pass
The correct way to do that in Python is thus:
def fun(a, b, opt=None):
pass
Edit 2: Answering explanation:
"Suppose I have a generic function f. I want to programmatically create a function f2 that behaves the same as f, but has a customised signature."
def f(*args, **kw):
pass
OK, then f2 looks like so:
def f2(a, b, opt=None):
f(a, b, opt=opt)
Again, the answer to your question is so trivial, that you obviously want to know something different that what you are asking. You really do need to stop asking abstract questions, and explain your concrete problem.