I'm looking at the (presumably) built-in module called "typing.py", and I'm trying to understand what's going on in there. I'm specifically looking at the code below, where we see an input argument called "typed=False". What does that even mean?
As I understand it, "func=None" appears to mean "no function is allowed in the inputs" (correct me if I'm wrong), where "func" refers to the object-type "function" (which presumably most programmers are familiar with because it's a basic concept). But what about "typed=False"?
def _tp_cache(func=None, /, *, typed=False):
"""Internal wrapper caching __getitem__ of generic types with a fallback to
original function for non-hashable arguments.
"""
def decorator(func):
cached = functools.lru_cache(typed=typed)(func)
_cleanups.append(cached.cache_clear)
#functools.wraps(func)
def inner(*args, **kwds):
try:
return cached(*args, **kwds)
except TypeError:
pass # All real errors (not unhashable args) are raised below.
return func(*args, **kwds)
return inner
if func is not None:
return decorator(func)
return decorator
As I understand it, "func=None" appears to mean "no function is allowed in the inputs" (correct me if I'm wrong), where "func" refers to the object-type "function" (which presumably most programmers are familiar with because it's a basic concept). But what about "typed=False"?
None of that is correct.
func is a parameter name, not a type. =None means that it defaults to None if no value is provided. typed is another parameter name; =False means it defaults to False. The * and / in the parameters list indicate that func is a positional parameter (i.e. it's always just the first argument, the caller doesn't need to say func=) and typed is a keyword argument (i.e. it must always be specified as typed=SOMETHING by the caller).
Trying to understand Python code without having a basic understanding of Python syntax is going to be extremely difficult; it's not something that you can guess at as you've attempted to do here, especially if you're diving straight into decorator (higher-order) functions, which are a relatively advanced feature and require that you already have a very firm grasp of how basic functions work.
I recommend making your way through a Python tutorial, e.g. https://docs.python.org/3/tutorial/.
Related
In python, it is possible to define a function taking an arbitrary number of positional arguments like so:
def f(*args):
print(args)
f(1, 2, 3) # (1, 2, 3)
When called as f(a, b, c), all positional arguments are put together into a tuple.
This behavior is described in python 2 and 3 documentation, but I haven't found a PEP to it.
PEP 3132, introducing extended iterable unpacking (first, *middle, last = seqence) states under "Acceptance" that
Make the starred target a tuple instead of a list. This would be consistent with a function's *args, but make further processing of the result harder.
was discussed. If I write a wrapper, I may also want to further process arguments like so:
def force_type(position, type):
def wrapper(f):
def new(*args, **kwargs):
args = list(args) # Why?
args[position] = type(args[position])
return f(*args, **kwargs)
return new
return wrapper
#force_type(1, int)
def func(a, b, c):
assert isinstance(b, int)
This further processing is made harder by the fact args is a tuple. Were wrappers just not used at the early stages this was introduced? If so, why wasn't this changed in python3 with other compatibility breaking changes (PEP3132 favours ease of processing over consistency (which seems at least similar to compatibility in a compatibility- breaking change).
Why are a functions *args (still) a tuple even though a list allows easier further processing?
I don't know if this was the thinking behind it, but that ease of processing (even though instantiate a list with the tuple data is not that hard) would come at possible confusing behavior.
def fce1(*args):
fce2(args)
# some more code using args
def fce2(args):
args.insert(0, 'other_val')
fce1(1, 2, 3)
Could surprise people writing fce1 code not realizing that args they deal with later on are not what the function was called with.
I would also presume immutable types are easier to deal with internally and come with less overhead.
Why not? The thing about tuple is, that you can not change it after creation. This allows to increase speed of executing your script, and you do not really need a list for your function arguments, because you do not really need to modify the given arguments of a function.
Would you need append or remove methods for your arguments? At most cases it would be no. Do you want your program run faster. That would be yes. And that's the way the most people would prefer to have things. The *args thing returns tuple because of that, and if you really need a list, you can transform it with one line of code!
args = list(args)
So in general:
It speeds up your program execution. You do not it to change the arguments. It is not that hard to change it's type.
My best guess would be that if *args generates a list(mutable), it can lead to very surprising results for a multitude of situations. #Ondrej K. has given a great example. As an analogy, when having a list as a default argument, every function call might have different default arguments. This is the result of default arguments being evaluated only once, and this situation is not the most intuitive. Even the official python docs have a specific workaround for this exact situation.
Default parameter values are evaluated from left to right when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. This is generally not what was intended. A way around this is to use None as the default, and explicitly test for it in the body of the function, e.g.:
def whats_on_the_telly(penguin=None):
if penguin is None:
penguin = []
penguin.append("property of the zoo")
return penguin
Source documentation
To summarize, I believe that *args is a tuple because having it as a list would cause all the problems associated with a mutable type (like slower speed) and the bigger issue would be that most do not expect function arguments to change.
Although I do agree that this implementation is very inconsistent with PEP-3132 and will cause confusion for most learners. I am very new to Python and it took me a while to understand what might be the reason for *args to be a tuple and not a list for the sake of consistency with PEP-3132's acceptance.
I have some code that looks like this:
from functools import lru_cache
#lru_cache()
def get_cheese(type):
print('{}? We\'re all out.'.format(type))
return None
get_cheese(type='cheddar')
get_cheese('cheddar')
print(get_cheese.cache_info())
cache_info() reports that there were two misses - yet I called the function with the same argument.
It actually took some doing, but I figured out that it's because in one instance I was using the keyword arg, and the other I was using a positional argument.
But why?
The wrapper created by functools.lru_cache makes no attempt to inspect or copy the signature of the wrapped function. The Python version is defined as
def wrapper(*args, **kwargs):
...
key = make_key(args, kwds, typed)
...
and as you can see, it builds the key used in the cache based on args and kwargs, without knowing whether any positional or keyword arguments are equivalent. The C version similarly builds the key without caring about the original signature.
As for why it's designed that way? I'm not aware of the original rationale. It could have been deliberate, either for simplicity of implementation or to avoid the overhead of matching keyword arguments to positionals, or it could have been an oversight. If you think changing it would be worthwhile, you could bring it up on the issue tracker.
def lowercasewrapper(func):
def wrapper(*args, **kwargs):
return [item.lower() for item in func(*args, **kwargs)]
return wrapper
I understand what decorators do, I have implemented the decorator above into my code and it works, but I'm little unsure about a few things.
Why can't (func) be replaced by (*args, **kwargs) and in the process remove the def wrapper line? I'm guessing the first 2 lines don't do the same thing, but to me that's what it seems like. It seems like:
def lowercasewrapper(accept function)
def wrapper(accept function)
What is the significance of the word 'func' here? I noticed I can replace that word with anything and my code still works. Does the function I put below #lowercasewrapper just feed into the decorator regardless of whats in the '( )'?
Also, a little off topic but the word item also has no significance right? I can replace that with any word as well and it still works.
I would appreciate if anyone would try to help and explain and answer in detail instead of redirecting me to a "what's a decorator" thread.
The short version is that a decorator actually turns this:
#decorated
def f(*args):
# function body
Into this:
def f(*args):
# function body
f = decorated(f)
So the reason you need the inner function is that the decorator must return a function, or the above makes no sense. With that in mind:
Point 1: notice that the last line returns wrapper, as in the function wrapper itself. This is why you can't remove that part; the function is actually building an altered function to return.
Points 2 and 3: You're right, it's just an arbitrary variable name and has no meaning outside this function.
So! With that in mind, here's what's going on in the decorator:
lowercasewrapper(f) is called (where f is apparently assumed to return an iterable of strings)
lowercasewrapper defines another function that takes some arbitrary arguments, then calls f on those arguments, then returns the result but with the items converted to lowercase
lowercasewrapper then returns the altered function
The biggest obstacle here is likely to be the idea of returning a function as opposed to returning the result of calling a function. Read up on first-class functions (or see Leon Young's link) if this makes no sense to you.
I have a function that is taking an arbitrary set of arguments and then choosing the correct function to process them based on the types of the arguments.
My current approach is using a decorator on all of the processing functions that checks the types of the arguments, and then iterating through all of the functions until one accepts the arguments.
Something about this seems a little hacky to me, and as a relatively new python programmer, I was wondering if there was a more 'pythonic' method to doing this.
so, currently, I have something like this:
def function_router(*args):
for func in functions: #functions is a list of functions
try:
return func(*args)
except TypeError:
pass
#probably raise an exception if no function works
and each function in 'functions' would have a decorator like this:
def accepts(*types) :
def my_decorator(func):
def wrapped(*args, **kwargs):
for i in range(len(types)):
if not isinstance(args[i], types[i]):
raise TypeError('Type error, %s not instance of %s, it is %s' %(args[i],types[i], type(args[i])))
return func(*args, **kwargs)
return wrapped
return my_decorator
EDIT: Oh man, I actually really liked reading all of the solutions. The answer I chose was most effective for what I'm currently doing, but I learned something from all of the answers, so thank you all for your time.
Perhaps the right way to do this is to use keyword arguments, instead of relying on the types of the arguments. That way, you don't have to decorate the little functions, just name the arguments correctly. It will also let you take advantage of Python's duck typing.
It sounds like you're trying to describe multimethods, for which GvR has provided a nice recipe in the form of a decorator.
this is from the source code of csv2rec in matplotlib
how can this function work, if its only parameters are 'func, default'?
def with_default_value(func, default):
def newfunc(name, val):
if ismissing(name, val):
return default
else:
return func(val)
return newfunc
ismissing takes a name and a value and determines if the row should be masked in a numpy array.
func will either be str, int, float, or dateparser...it converts data. Maybe not important. I'm just wondering how it can get a 'name' and a 'value'
I'm a beginner. Thanks for any 2cents! I hope to get good enough to help others!
This with_default_value function is what's often referred to (imprecisely) as "a closure" (technically, the closure is rather the inner function that gets returned, here newfunc -- see e.g. here). More generically, with_default_value is a higher-order function ("HOF"): it takes a function (func) as an argument, it also returns a function (newfunc) as the result.
I've seen answers confusing this with the decorator concept and construct in Python, which is definitely not the case -- especially since you mention func as often being a built-in such as int. Decorators are also higher-order functions, but rather specific ones: ones which return a decorated, i.e. "enriched", version of their function argument (which must be the only argument -- "decorators with arguments" are obtained through one more level of function/closure nesting, not by giving the decorator HOF more than one argument), which gets reassigned to exactly the same name as that function argument (and so typically has the same signature -- using a decorator otherwise would be extremely peculiar, un-idiomatic, unreadable, etc).
So forget decorators, which have absolutely nothing to do with the case, and focus on the newfunc closure. A lexically nested function can refer to (though not rebind) all local variable names (including argument names, since arguments are local variables) of the enclosing function(s) -- that's why it's known as a closure: it's "closed over" these "free variables". Here, newfunc can refer to func and default -- and does.
Higher-order functions are a very natural thing in Python, especially since functions are first-class objects (so there's nothing special you need to do to pass them as arguments, return them as function values, or even storing them in lists or other containers, etc), and there's no namespace distinction between functions and other kinds of objects, no automatic calling of functions just because they're mentioned, etc, etc. (It's harder - a bit harder, or MUCH harder, depending - in other languages that do draw lots of distinctions of this sort). In Python, mentioning a function is just that -- a mention; the CALL only happens if and when the function object (referred to by name, or otherwise) is followed by parentheses.
That's about all there is to this example -- please do feel free to edit your question, comment here, etc, if there's some other specific aspect that you remain in doubt about!
Edit: so the OP commented courteously asking for more examples of "closure factories". Here's one -- imagine some abstract kind of GUI toolkit, and you're trying to do:
for i in range(len(buttons)):
buttons[i].onclick(lambda: mainwin.settitle("button %d click!" % i))
but this doesn't work right -- i within the lambda is late-bound, so by the time one button is clicked i's value is always going to be the index of the last button, no matter which one was clicked. There are various feasible solutions, but a closure factory's an elegant possibility:
def makeOnclick(message):
return lambda: mainwin.settitle(message)
for i in range(len(buttons)):
buttons[i].onclick(makeOnClick("button %d click!" % i))
Here, we're using the closure factory to tweak the binding time of variables!-) In one specific form or another, this is a pretty common use case for closure factories.
This is a Python decorator -- basically a function wrapper. (Read all about decorators in PEP 318 -- http://www.python.org/dev/peps/pep-0318/)
If you look through the code, you will probably find something like this:
def some_func(name, val):
# ...
some_func = with_default_value(some_func, 'the_default_value')
The intention of this decorator seems to supply a default value if either the name or val arguments are missing (presumably, if they are set to None).
As for why it works:
with_default_value returns a function object, which is basically going to be a copy of that nested newfunc, with the 'func' call and default value substited with whatever was passed to with_default_value.
If someone does 'foo = with_default_value(bar, 3)', the return value is basically going to be a new function:
def foo(name, val):
ifismissing(name, val):
return 3
else:
return bar(val)
so you can then take that return value, and call it.
This is a function that returns another function. name and value are the parameters of the returned function.