Related
When declaring a function, should we use return annotations?
def generate_nine() -> int:
return 9
Or can we simply put something like this:
def generate_nine():
return 9
Yes, You should write return type annotation for writing cleaner code. It's been said that return type annotation enhanced code readability. Consider an example
For example:
def get_data(a):
return a ** 2
It is not easier for anyone, what is the expected return type.If the incoming data is float, the function will return float.
If the function receive int value, it will return int. So even it is not easier to read.
So for better understanding it is being recommended to write both arguments type and return type while writing. See the below example:
def greeting(name: str) -> str:
return 'Hello ' + name
In the function greeting, the argument name is expected to be of type str and the return type str. Subtypes are accepted as arguments. The compiler will raise an error if it is not expecting a string value.
The code is much clear and understandable compare to the previous one. So to summarize, for writing cleaner code we should add return type annotation. Please see this docs for better understanding
In more detail, Python 2.x has docstrings, which allow you to attach a metadata string to various types of object. This is amazingly handy, so Python 3 extends the feature by allowing you to attach metadata to functions describing their parameters and return values.
There's no preconceived use case, but the PEP suggests several. One very handy one is to allow you to annotate parameters with their expected types; it would then be easy to write a decorator that verifies the annotations or coerces the arguments to the right type. Another is to allow parameter-specific documentation instead of encoding it into the docstring.
DOCS
Function Annotations: PEP-3107
I ran across a snippet of code demonstrating Python3's function annotations. The concept is simple but I can't think of why these were implemented in Python3 or any good uses for them. Perhaps SO can enlighten me?
How it works:
def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
... function body ...
Everything following the colon after an argument is an 'annotation', and the information following the -> is an annotation for the function's return value.
foo.func_annotations would return a dictionary:
{'a': 'x',
'b': 11,
'c': list,
'return': 9}
What's the significance of having this available?
Function annotations are what you make of them.
They can be used for documentation:
def kinetic_energy(mass: 'in kilograms', velocity: 'in meters per second'):
...
They can be used for pre-condition checking:
def validate(func, locals):
for var, test in func.__annotations__.items():
value = locals[var]
msg = 'Var: {0}\tValue: {1}\tTest: {2.__name__}'.format(var, value, test)
assert test(value), msg
def is_int(x):
return isinstance(x, int)
def between(lo, hi):
def _between(x):
return lo <= x <= hi
return _between
def f(x: between(3, 10), y: is_int):
validate(f, locals())
print(x, y)
>>> f(0, 31.1)
Traceback (most recent call last):
...
AssertionError: Var: y Value: 31.1 Test: is_int
Also see http://www.python.org/dev/peps/pep-0362/ for a way to implement type checking.
I think this is actually great.
Coming from an academic background, I can tell you that annotations have proved themselves invaluable for enabling smart static analyzers for languages like Java. For instance, you could define semantics like state restrictions, threads that are allowed to access, architecture limitations, etc., and there are quite a few tools that can then read these and process them to provide assurances beyond what you get from the compilers. You could even write things that check preconditions/postconditions.
I feel something like this is especially needed in Python because of its weaker typing, but there were really no constructs that made this straightforward and part of the official syntax.
There are other uses for annotations beyond assurance. I can see how I could apply my Java-based tools to Python. For instance, I have a tool that lets you assign special warnings to methods, and gives you indications when you call them that you should read their documentation (E.g., imagine you have a method that must not be invoked with a negative value, but it's not intuitive from the name). With annotations, I could technically write something like this for Python. Similarly, a tool that organizes methods in a large class based on tags can be written if there is an official syntax.
This is a way late answer, but AFAICT, the best current use of function annotations is PEP-0484 and MyPy. There's also PyRight from Microsoft which is used by VSCode and also available via CLI.
Mypy is an optional static type checker for Python. You can add type hints to your Python programs using the upcoming standard for type annotations introduced in Python 3.5 beta 1 (PEP 484), and use mypy to type check them statically.
Used like so:
from typing import Iterator
def fib(n: int) -> Iterator[int]:
a, b = 0, 1
while a < n:
yield a
a, b = b, a + b
Just to add a specific example of a good use from my answer here, coupled with decorators a simple mechanism for multimethods can be done.
# This is in the 'mm' module
registry = {}
import inspect
class MultiMethod(object):
def __init__(self, name):
self.name = name
self.typemap = {}
def __call__(self, *args):
types = tuple(arg.__class__ for arg in args) # a generator expression!
function = self.typemap.get(types)
if function is None:
raise TypeError("no match")
return function(*args)
def register(self, types, function):
if types in self.typemap:
raise TypeError("duplicate registration")
self.typemap[types] = function
def multimethod(function):
name = function.__name__
mm = registry.get(name)
if mm is None:
mm = registry[name] = MultiMethod(name)
spec = inspect.getfullargspec(function)
types = tuple(spec.annotations[x] for x in spec.args)
mm.register(types, function)
return mm
and an example of use:
from mm import multimethod
#multimethod
def foo(a: int):
return "an int"
#multimethod
def foo(a: int, b: str):
return "an int and a string"
if __name__ == '__main__':
print("foo(1,'a') = {}".format(foo(1,'a')))
print("foo(7) = {}".format(foo(7)))
This can be done by adding the types to the decorator as Guido's original post shows, but annotating the parameters themselves is better as it avoids the possibility of wrong matching of parameters and types.
Note: In Python you can access the annotations as function.__annotations__ rather than function.func_annotations as the func_* style was removed on Python 3.
Uri has already given a proper answer, so here's a less serious one: So you can make your docstrings shorter.
The first time I saw annotations, I thought "great! Finally I can opt in to some type checking!" Of course, I hadn't noticed that annotations are not actually enforced.
So I decided to write a simple function decorator to enforce them:
def ensure_annotations(f):
from functools import wraps
from inspect import getcallargs
#wraps(f)
def wrapper(*args, **kwargs):
for arg, val in getcallargs(f, *args, **kwargs).items():
if arg in f.__annotations__:
templ = f.__annotations__[arg]
msg = "Argument {arg} to {f} does not match annotation type {t}"
Check(val).is_a(templ).or_raise(EnsureError, msg.format(arg=arg, f=f, t=templ))
return_val = f(*args, **kwargs)
if 'return' in f.__annotations__:
templ = f.__annotations__['return']
msg = "Return value of {f} does not match annotation type {t}"
Check(return_val).is_a(templ).or_raise(EnsureError, msg.format(f=f, t=templ))
return return_val
return wrapper
#ensure_annotations
def f(x: int, y: float) -> float:
return x+y
print(f(1, y=2.2))
>>> 3.2
print(f(1, y=2))
>>> ensure.EnsureError: Argument y to <function f at 0x109b7c710> does not match annotation type <class 'float'>
I added it to the Ensure library.
It a long time since this was asked but the example snippet given in the question is (as stated there as well) from PEP 3107 and at the end of thas PEP example Use cases are also given which might answer the question from the PEPs point of view ;)
The following is quoted from PEP3107
Use Cases
In the course of discussing annotations, a number of use-cases have been raised. Some of these are presented here, grouped by what kind of information they convey. Also included are examples of existing products and packages that could make use of annotations.
Providing typing information
Type checking ([3], [4])
Let IDEs show what types a function expects and returns ([17])
Function overloading / generic functions ([22])
Foreign-language bridges ([18], [19])
Adaptation ([21], [20])
Predicate logic functions
Database query mapping
RPC parameter marshaling ([23])
Other information
Documentation for parameters and return values ([24])
See the PEP for more information on specific points (as well as their references)
Python 3.X (only) also generalizes function definition to allow
arguments and return values to be annotated with object values
for use in extensions.
Its META-data to explain, to be more explicit about the function values.
Annotations are coded as :value after the
argument name and before a default, and as ->value after the
argument list.
They are collected into an __annotations__ attribute of the function, but are not otherwise treated as special by Python itself:
>>> def f(a:99, b:'spam'=None) -> float:
... print(a, b)
...
>>> f(88)
88 None
>>> f.__annotations__
{'a': 99, 'b': 'spam', 'return': <class 'float'>}
Source: Python Pocket Reference, Fifth Edition
EXAMPLE:
The typeannotations module provides a set of tools for type checking and type inference of Python code. It also a provides a set of types useful for annotating functions and objects.
These tools are mainly designed to be used by static analyzers such as linters, code completion libraries and IDEs. Additionally, decorators for making run-time checks are provided. Run-time type checking is not always a good idea in Python, but in some cases it can be very useful.
https://github.com/ceronman/typeannotations
How Typing Helps to Write Better Code
Typing can help you do static code analysis to catch type errors
before you send your code to production and prevent you from some
obvious bugs. There are tools like mypy, which you can add to your
toolbox as part of your software life cycle. mypy can check for
correct types by running against your codebase partially or fully.
mypy also helps you to detect bugs such as checking for the None type
when the value is returned from a function. Typing helps to make your
code cleaner. Instead of documenting your code using comments, where
you specify types in a docstring, you can use types without any
performance cost.
Clean Python: Elegant Coding in Python
ISBN: ISBN-13 (pbk): 978-1-4842-4877-5
PEP 526 -- Syntax for Variable Annotations
https://www.python.org/dev/peps/pep-0526/
https://www.attrs.org/en/stable/types.html
Despite all uses described here, the one enforceable and, most likely, enforced use of annotations will be for type hints.
This is currently not enforced in any way but, judging from PEP 484, future versions of Python will only allow types as the value for annotations.
Quoting What about existing uses of annotations?:
We do hope that type hints will eventually become the sole use for annotations, but this will require additional discussion and a deprecation period after the initial roll-out of the typing module with Python 3.5. The current PEP will have provisional status (see PEP 411 ) until Python 3.6 is released. The fastest conceivable scheme would introduce silent deprecation of non-type-hint annotations in 3.6, full deprecation in 3.7, and declare type hints as the only allowed use of annotations in Python 3.8.
Though I haven't seen any silent deprecations in 3.6 yet, this could very well be bumped to 3.7, instead.
So, even though there might be some other good use-cases, it is best to keep them solely for type hinting if you don't want to go around changing everything in a future where this restriction is in place.
As a bit of a delayed answer, several of my packages (marrow.script, WebCore, etc.) use annotations where available to declare typecasting (i.e. transforming incoming values from the web, detecting which arguments are boolean switches, etc.) as well as to perform additional markup of arguments.
Marrow Script builds a complete command-line interface to arbitrary functions and classes and allows for defining documentation, casting, and callback-derived default values via annotations, with a decorator to support older runtimes. All of my libraries that use annotations support the forms:
any_string # documentation
any_callable # typecast / callback, not called if defaulting
(any_callable, any_string) # combination
AnnotationClass() # package-specific rich annotation object
[AnnotationClass(), AnnotationClass(), …] # cooperative annotation
"Bare" support for docstrings or typecasting functions allows for easier mixing with other libraries that are annotation-aware. (I.e. have a web controller using typecasting that also happens to be exposed as a command-line script.)
Edited to add: I've also begun making use of the TypeGuard package using development-time assertions for validation. Benefit: when run with "optimizations" enabled (-O / PYTHONOPTIMIZE env var) the checks, which may be expensive (e.g. recursive) are omitted, with the idea that you've properly tested your app in development so the checks should be unnecessary in production.
Annotations can be used for easily modularizing code. E.g. a module for a program which I'm maintaining could just define a method like:
def run(param1: int):
"""
Does things.
:param param1: Needed for counting.
"""
pass
and we could ask the user for a thing named "param1" which is "Needed for counting" and should be an "int". In the end we can even convert the string given by the user to the desired type to get the most hassle free experience.
See our function metadata object for an open source class which helps with this and can automatically retrieve needed values and convert them to any desired type (because the annotation is a conversion method). Even IDEs show autocompletions right and assume that types are according to annotations - a perfect fit.
If you look at the list of benefits of Cython, a major one is the ability to tell the compiler which type a Python object is.
I can envision a future where Cython (or similar tools that compile some of your Python code) will use the annotation syntax to do their magic.
class foo(object):
def __init__(self,f):
self.f = f
def __call__(self,args_list):
def wrapped_f(args_list):
return [self.f(*args) for args in args_list]
return wrapped_f(args_list)
if __name__=='__main__':
class abc(object):
#foo
def f(a,b,c):
return a+b+c
a = range(5)
b = range(5)
c = range(5)
data = list(zip(a,b,c))
print(abc.f(data))
I wrote this a few years back. When you decorate any function f(X) with #foo it becomes
f(list of Xs).
What is this process called? What is it? What is its functional programming name?
Its not currying. I know simple map9(f,list of Xs) could have done it.
What are decorators/operation of decorating called mathematically?
There are two transformations performed on your original function:
it is converted from a function of three arguments to a function that takes a 3-tuple
conversion from a function of a 3-tuple to a function that takes a list of 3-tuples
First transformation
In Haskell, there is a function called uncurry, documented here. (This is a two-argument version; 3-, 4-, ... versions could be easily created, too).
Second transformation
Also in Haskell, there are sets of functions with lift in their names. Here's a page on the Haskell wiki about lifting. I think that page explains it better than I could:
Lifting is a concept which allows you to transform a function into a
corresponding function within another (usually more general) setting.
So in your case, you're lifting a function from operating on tuples to operating on a list of tuples.
Notes:
the OP asked for the mathematical name for decorators. I don't know what that would be, but I've heard that Haskell is supposed to be like executable mathematics, so I think Haskell's terminology is a good starting point. YMMV.
the OP asked for the FP name of these processes. Again, I don't know, but I assume that Haskell's terminology is acceptable.
Decorators just have special syntax, but there are no rules what decorators can return and no mathematical description. They can be any callable after all.
Your function is just a partially applied starmap:
from functools import partial
from itertools import starmap
def foo(f):
return partial(starmap, f)
In a functional language like Haskell, you would do this by partially applying the map function to a function which takes a tuple of arguments, resulting in a function which takes a list of argument tuples. As Jochen Ritzel pointed out in another answer, even in Python you can implement this pretty trivially using functools.partial.
Therefore I suppose this process is called "partial application of map", or some such thing. I'm not aware of any particular name for this special case.
They are simply called Decorators. What it does can be called function chaining or function annotation, but I looked around quite a bit and found no special functional/mathmatical name for this process besides those 2 (chaining/annotation).
PEP Index > PEP 318 -- Decorators for Functions and Methods
On the name 'Decorator'
There's been a number of complaints about the choice of the name
'decorator' for this feature. The major one is that the name is not
consistent with its use in the GoF book [11]. The name 'decorator'
probably owes more to its use in the compiler area -- a syntax tree is
walked and annotated. It's quite possible that a better name may turn
up.
While I am aware of the duck-typing concept of Python, I sometimes struggle with the type of arguments of functions, or the type of the return value of the function.
Now, if I wrote the function myself, I DO know the types. But what if somebody wants to use and call my functions, how is he/she expected to know the types?
I usually put type information in the function's docstring (like: "...the id argument should be an integer..." and "... the function will return a (string, [integer]) tuple.")
But is looking up the information in the docstring (and putting it there, as a coder) really the way it is supposed to be done?
Edit: While the majority of answers seem to direct towards "yes, document!" I feel this is not always very easy for 'complex' types. For example: how to describe concisely in a docstring that a function returns a list of tuples, with each tuple of the form (node_id, node_name, uptime_minutes) and that the elements are respectively a string, string and integer?
The docstring PEP documentation doesn't give any guidelines on that.
I guess the counterargument will be that in that case classes should be used, but I find python very flexible because it allows passing around these things using lists and tuples, i.e. without classes.
Well things have changed a little bit since 2011! Now there's type hints in Python 3.5 which you can use to annotate arguments and return the type of your function. For example this:
def greeting(name):
return 'Hello, {}'.format(name)
can now be written as this:
def greeting(name: str) -> str:
return 'Hello, {}'.format(name)
As you can now see types, there's some sort of optional static type checking which will help you and your type checker to investigate your code.
for more explanation I suggest to take a look at the blog post on type hints in PyCharm blog.
This is how dynamic languages work. It is not always a good thing though, especially if the documentation is poor - anyone tried to use a poorly documented python framework? Sometimes you have to revert to reading the source.
Here are some strategies to avoid problems with duck typing:
create a language for your problem domain
this will help you to name stuff properly
use types to represent concepts in your domain language
name function parameters using the domain language vocabulary
Also, one of the most important points:
keep data as local as possible!
There should only be a few well-defined and documented types being passed around. Anything else should be obvious by looking at the code: Don't have weird parameter types coming from far away that you can't figure out by looking in the vicinity of the code...
Related, (and also related to docstrings), there is a technique in python called doctests. Use that to document how your methods are expected to be used - and have nice unit test coverage at the same time!
Actually there is no need as python is a dynamic language, BUT if you want to specify a return value then do this
def foo(a) -> int: #after arrow type the return type
return 1 + a
But it won't really help you much. It doesn't raise exceptions in the same way like in staticly-typed language like java, c, c++. Even if you returned a string, it won't raise any exceptions.
and then for argument type do this
def foo(a: int) -> int:
return a+ 1
after the colon (:)you can specify the argument type.
This won't help either, to prove this here is an example:
def printer(a: int) -> int: print(a)
printer("hello")
The function above actually just returns None, because we didn't return anything, but we did tell it we would return int, but as I said it doesn't help. Maybe it could help in IDEs (Not all but few like pycharm or something, but not on vscode)
I attended a coursera course, there was lesson in which, we were taught about design recipe.
Below docstring format I found preety useful.
def area(base, height):
'''(number, number ) -> number #**TypeContract**
Return the area of a tring with dimensions base #**Description**
and height
>>>area(10,5) #**Example **
25.0
>>area(2.5,3)
3.75
'''
return (base * height) /2
I think if docstrings are written in this way, it might help a lot to developers.
Link to video [Do watch the video] : https://www.youtube.com/watch?v=QAPg6Vb_LgI
Yes, you should use docstrings to make your classes and functions more friendly to other programmers:
More: http://www.python.org/dev/peps/pep-0257/#what-is-a-docstring
Some editors allow you to see docstrings while typing, so it really makes work easier.
Yes it is.
In Python a function doesn't always have to return a variable of the same type (although your code will be more readable if your functions do always return the same type). That means that you can't specify a single return type for the function.
In the same way, the parameters don't always have to be the same type too.
For example: how to describe concisely in a docstring that a function returns a list of tuples, with each tuple of the form (node_id, node_name, uptime_minutes) and that the elements are respectively a string, string and integer?
Um... There is no "concise" description of this. It's complex. You've designed it to be complex. And it requires complex documentation in the docstring.
Sorry, but complexity is -- well -- complex.
Answering my own question >10 years later, there are now 2 things I use to manage this:
type hints (as already mentioned in other answers)
dataclasses, when parameter or return type hints become unwieldy/hard to read
As an example of the latter, say I have a function
def do_something(param:int) -> list[tuple[list, int|None]]:
...
return result
I would now rewrite using a dataclass, e.g. along the lines of:
from dataclasses import dataclass
#dataclass
class Stat:
entries: list
value: int | None = None
def do_something(param:int) -> list[Stat]:
...
return result
Yes, since it's a dynamically type language ;)
Read this for reference: PEP 257
Docstrings (and documentation in general). Python 3 introduces (optional) function annotations, as described in PEP 3107 (but don't leave out docstrings)
A generic function is dispatched based on the type of all its arguments. The programmer defines several implementations of a function. The correct one is chosen at call time based on the types of its arguments. This is useful for object adaptation among other things. Python has a few generic functions including len().
These packages tend to allow code that looks like this:
#when(int)
def dumbexample(a):
return a * 2
#when(list)
def dumbexample(a):
return [("%s" % i) for i in a]
dumbexample(1) # calls first implementation
dumbexample([1,2,3]) # calls second implementation
A less dumb example I've been thinking about lately would be a web component that requires a User. Instead of requiring a particular web framework, the integrator would just need to write something like:
class WebComponentUserAdapter(object):
def __init__(self, guest):
self.guest = guest
def canDoSomething(self):
return guest.member_of("something_group")
#when(my.webframework.User)
componentNeedsAUser(user):
return WebComponentUserAdapter(user)
Python has a few generic functions implementations. Why would I chose one over the others? How is that implementation being used in applications?
I'm familiar with Zope's zope.component.queryAdapter(object, ISomething). The programmer registers a callable adapter that takes a particular class of object as its argument and returns something compatible with the interface. It's a useful way to allow plugins. Unlike monkey patching, it works even if an object needs to adapt to multiple interfaces with the same method names.
I'd recommend the PEAK-Rules library by P. Eby. By the same author (deprecated though) is the RuleDispatch package (the predecessor of PEAK-Rules). The latter being no longer maintained IIRC.
PEAK-Rules has a lot of nice features, one being, that it is (well, not easily, but) extensible. Besides "classic" dispatch on types ony, it features dispatch on arbitrary expressions as "guardians".
The len() function is not a true generic function (at least in the sense of the packages mentioned above, and also in the sense, this term is used in languages like Common Lisp, Dylan or Cecil), as it is simply a convenient syntax for a call to specially named (but otherwise regular) method:
len(s) == s.__len__()
Also note, that this is single-dispatch only, that is, the actual receiver (s in the code above) determines the method implementation called. And even a hypothetical
def call_special(receiver, *args, **keys):
return receiver.__call_special__(*args, **keys)
is still a single-dispatch function, as only the receiver is used when the method to be called is resolved. The remaining arguments are simply passed on, but they don't affect the method selection.
This is different from multiple-dispatch, where there is no dedicated receiver, and all arguments are used in order to find the actual method implementation to call. This is, what actually makes the whole thing worthwhile. If it were only some odd kind of syntactic sugar, nobody would bother with using it, IMHO.
from peak.rules import abstract, when
#abstract
def serialize_object(object, target):
pass
#when(serialize_object, (MyStuff, BinaryStream))
def serialize_object(object, target):
target.writeUInt32(object.identifier)
target.writeString(object.payload)
#when(serialize_object, (MyStuff, XMLStream))
def serialize_object(object, target):
target.openElement("my-stuff")
target.writeAttribute("id", str(object.identifier))
target.writeText(object.payload)
target.closeElement()
In this example, a call like
serialize_object(MyStuff(10, "hello world"), XMLStream())
considers both arguments in order to decide, which method must actually be called.
For a nice usage scenario of generic functions in Python I'd recommend reading the refactored code of the peak.security which gives a very elegant solution to access permission checking using generic functions (using RuleDispatch).
You can use a construction like this:
def my_func(*args, **kwargs):
pass
In this case args will be a list of any unnamed arguments, and kwargs will be a dictionary of the named ones. From here you can detect their types and act as appropriate.
I'm unable to see the point in these "generic" functions. It sounds like simple polymorphism.
Your "generic" features can be implemented like this without resorting to any run-time type identification.
class intWithDumbExample( int ):
def dumbexample( self ):
return self*2
class listWithDumbExmaple( list ):
def dumpexample( self ):
return [("%s" % i) for i in self]
def dumbexample( a ):
return a.dumbexample()