I recently ran into Ellipsis (...) that is used in function parameters in aiohttp code and then in that function's body:
def make_mocked_request(method, path, headers=None, *,
match_info=sentinel,
version=HttpVersion(1, 1), closing=False,
app=None,
writer=sentinel,
protocol=sentinel,
transport=sentinel,
payload=sentinel,
sslcontext=None,
client_max_size=1024**2,
loop=...):
"""Creates mocked web.Request testing purposes.
Useful in unit tests, when spinning full web server is overkill or
specific conditions and errors are hard to trigger.
"""
task = mock.Mock()
if loop is ...:
loop = mock.Mock()
loop.create_future.return_value = ()
Can you explain this new python 3 feature?
Ellipsis is a Built-in constant in Python. In Python 3 it has a literal syntax ... so it can be used like any other literal. This was accepted by Guido for Python 3 because some folks thought it would be cute.
The code you have found (use as function argument default) is apparently one such "cute" usage. Later in that code you'll see:
if loop is ...:
loop = mock.Mock()
loop.create_future.return_value = ()
It's just being used as a sentinel, here, and may as well be object() or anything else - there is nothing specific to Ellipsis. Perhaps the usual sentinel, None, has some other specific meaning in this context, although I can not see any evidence of that in the commit (it seems like None will have worked just as well).
Another use-case for an ellipsis literal sometimes seen in the wild is a placeholder for code not written yet, similar to pass statement:
class Todo:
...
For the more typical use-cases, which are involving the extended slice syntax, see What does the Python Ellipsis object do?
Building on https://stackoverflow.com/a/54406084/1988486, using ... vs. None has some validity when dealing with type annotations. Consider the following:
def foo(map: dict = None):
if not map:
map = {}
return map
Granted, this doesn't do much, but this is causing type hints to conflict in a modern editor such as VS Code. This is due to the fact that you're claiming that map is a dict and yet the default is a NoneType (not a dict).
The Ellipsis is its own type, so VS Code will not complain, but you can interact with it in similar ways because it is a sentinel value.
def food(map: dict = ...):
if map is ...:
map = {}
return map
VS Code will not raise any concerns about the type hinting and the various types this variable can become in the course of running the code.
I can't say for sure why the aiohttp code uses it since they don't appear to be using type annotations, but it's at least a viable reason to do so.
Related
I'm starting to get into type hints (aka annotations) in python 3.6, and I can't figure some of the dynamic aspects of this feature.
I wrote the following piece of code, and I want to add annotation and not sure how, even after looking through the docs on type hinting.
This is the function:
def validate_expression(expression: ?):
try:
assert expression
except AssertionError as e:
...
expression needs to be anything that an assert works on (assuming any expression for which bool(expression) is valid).
What should I write instead of the question mark?
UPDATE:
I know that most python expressions can be cast as a Boolean, but the context in which I write this code is one where it is reasonable to expect an expression to not be a assertable.
The relevant example in my case is pandas.DataFrame. Running
bool(pandas.DataFrame()) raises an error, and I have good reason to expect that someone might try to pass a dataframe to the validation function.
UPDATE 2:
Following Chepner's comments and answer, I understand now that:
1. In the vast majority of cases, any python expression will have a valid casting to Boolean, and this is either covered by typing.Any or by not adding annotation at all.
2. In the edge case I was interested in, which is bool(pandas.DataFrame()) # --> ValueError, annotations won't help since this is a runtime error.
3. If there is another edge case that is relevant for static type hinting, I am not aware of it.
4. Given the rarity/non-existence of a relevant example, there's no out of the box type that generically describes just the quality of the ability to be casted to boolean (similar to typing.Iterable), and as far as I'm concerned it is not worth bending over backwards to address such an edge case (although it would be interesting to hear of relevant example and a bend-y solution!)
Any value whatsoever can be used in a boolean context. An instance of object is considered to be a truthy value unless a descendent class provides an alternate definition; anything that is considered false (like an empty list, an empty str, an empty dict, False itself, etc) does so because it has been specially defined to be so.
As such, the only type hint you could use is typing.Any:
from typing import Any
def validate_expression(expression: Any):
try:
assert expression
except AssertionError as e:
...
which, really, is barely worth stating explicitly.
Regarding your update 2: here's a somewhat janky way of accomplishing what you want to do:
Create a custom Protocol that matches any type that defines a __bool__ method. (And perhaps also the __nonzero__ method, if you also want to support Python 2.)
Find or create stubs for the pandas library. Ensure that the type hints for DataFrame do not contain the __bool__ method. That is, will not match the protocol.
Create a function that uses your custom protocol as the type hint.
For example:
# If you're using Python 3.8+
from typing import Protocol
# If you're not, run 'pip install typing_extensions' and do the below instead
from typing_extensions import Protocol
class SupportsBool(Protocol):
def __bool__(self) -> bool: ...
class MyFakeDataFrame:
# ...snip...
pass
class MyFakeBoolableThing:
def __bool__(self) -> bool:
return True
def validate_expression(x: SupportsBool) -> None:
bool(x)
# These all type-check!
validate_expression(True)
validate_expression(0)
validate_expression(MyFakeBoolableThing())
# This will *not* typecheck
validate_expression(MyFakeDataFrame())
# Perhaps surprisingly, these will also not typecheck:
validate_expression("foobar")
validate_expression([1, 2, 3])
The reason why the latter two expressions will not type-check is because neither strings nor lists actually define a custom __bool__ method (or __nonzero__ method in Python 2): instead, they define a __len__ method, and the bool(...) function will fall back to checking __len__ if __bool__/__nonzero__ doesn't exist.
If you do want your validate function to accept such expressions, you'd need to use a type like Union[SupportsBool, SupportsInt] -- but unfortunately, I believe the pandas DataFrame class does implement a functional __len__ method, so you're back to square zero if you take that approach.
So basically, in this case you're either (1) forced into rejecting certain types like str or list that do meaningful things when bool'd, or (2) forced into accepting pandas.DataFrame as an acceptable boolable thing.
Function Annotations: PEP-3107
I ran across a snippet of code demonstrating Python3's function annotations. The concept is simple but I can't think of why these were implemented in Python3 or any good uses for them. Perhaps SO can enlighten me?
How it works:
def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
... function body ...
Everything following the colon after an argument is an 'annotation', and the information following the -> is an annotation for the function's return value.
foo.func_annotations would return a dictionary:
{'a': 'x',
'b': 11,
'c': list,
'return': 9}
What's the significance of having this available?
Function annotations are what you make of them.
They can be used for documentation:
def kinetic_energy(mass: 'in kilograms', velocity: 'in meters per second'):
...
They can be used for pre-condition checking:
def validate(func, locals):
for var, test in func.__annotations__.items():
value = locals[var]
msg = 'Var: {0}\tValue: {1}\tTest: {2.__name__}'.format(var, value, test)
assert test(value), msg
def is_int(x):
return isinstance(x, int)
def between(lo, hi):
def _between(x):
return lo <= x <= hi
return _between
def f(x: between(3, 10), y: is_int):
validate(f, locals())
print(x, y)
>>> f(0, 31.1)
Traceback (most recent call last):
...
AssertionError: Var: y Value: 31.1 Test: is_int
Also see http://www.python.org/dev/peps/pep-0362/ for a way to implement type checking.
I think this is actually great.
Coming from an academic background, I can tell you that annotations have proved themselves invaluable for enabling smart static analyzers for languages like Java. For instance, you could define semantics like state restrictions, threads that are allowed to access, architecture limitations, etc., and there are quite a few tools that can then read these and process them to provide assurances beyond what you get from the compilers. You could even write things that check preconditions/postconditions.
I feel something like this is especially needed in Python because of its weaker typing, but there were really no constructs that made this straightforward and part of the official syntax.
There are other uses for annotations beyond assurance. I can see how I could apply my Java-based tools to Python. For instance, I have a tool that lets you assign special warnings to methods, and gives you indications when you call them that you should read their documentation (E.g., imagine you have a method that must not be invoked with a negative value, but it's not intuitive from the name). With annotations, I could technically write something like this for Python. Similarly, a tool that organizes methods in a large class based on tags can be written if there is an official syntax.
This is a way late answer, but AFAICT, the best current use of function annotations is PEP-0484 and MyPy. There's also PyRight from Microsoft which is used by VSCode and also available via CLI.
Mypy is an optional static type checker for Python. You can add type hints to your Python programs using the upcoming standard for type annotations introduced in Python 3.5 beta 1 (PEP 484), and use mypy to type check them statically.
Used like so:
from typing import Iterator
def fib(n: int) -> Iterator[int]:
a, b = 0, 1
while a < n:
yield a
a, b = b, a + b
Just to add a specific example of a good use from my answer here, coupled with decorators a simple mechanism for multimethods can be done.
# This is in the 'mm' module
registry = {}
import inspect
class MultiMethod(object):
def __init__(self, name):
self.name = name
self.typemap = {}
def __call__(self, *args):
types = tuple(arg.__class__ for arg in args) # a generator expression!
function = self.typemap.get(types)
if function is None:
raise TypeError("no match")
return function(*args)
def register(self, types, function):
if types in self.typemap:
raise TypeError("duplicate registration")
self.typemap[types] = function
def multimethod(function):
name = function.__name__
mm = registry.get(name)
if mm is None:
mm = registry[name] = MultiMethod(name)
spec = inspect.getfullargspec(function)
types = tuple(spec.annotations[x] for x in spec.args)
mm.register(types, function)
return mm
and an example of use:
from mm import multimethod
#multimethod
def foo(a: int):
return "an int"
#multimethod
def foo(a: int, b: str):
return "an int and a string"
if __name__ == '__main__':
print("foo(1,'a') = {}".format(foo(1,'a')))
print("foo(7) = {}".format(foo(7)))
This can be done by adding the types to the decorator as Guido's original post shows, but annotating the parameters themselves is better as it avoids the possibility of wrong matching of parameters and types.
Note: In Python you can access the annotations as function.__annotations__ rather than function.func_annotations as the func_* style was removed on Python 3.
Uri has already given a proper answer, so here's a less serious one: So you can make your docstrings shorter.
The first time I saw annotations, I thought "great! Finally I can opt in to some type checking!" Of course, I hadn't noticed that annotations are not actually enforced.
So I decided to write a simple function decorator to enforce them:
def ensure_annotations(f):
from functools import wraps
from inspect import getcallargs
#wraps(f)
def wrapper(*args, **kwargs):
for arg, val in getcallargs(f, *args, **kwargs).items():
if arg in f.__annotations__:
templ = f.__annotations__[arg]
msg = "Argument {arg} to {f} does not match annotation type {t}"
Check(val).is_a(templ).or_raise(EnsureError, msg.format(arg=arg, f=f, t=templ))
return_val = f(*args, **kwargs)
if 'return' in f.__annotations__:
templ = f.__annotations__['return']
msg = "Return value of {f} does not match annotation type {t}"
Check(return_val).is_a(templ).or_raise(EnsureError, msg.format(f=f, t=templ))
return return_val
return wrapper
#ensure_annotations
def f(x: int, y: float) -> float:
return x+y
print(f(1, y=2.2))
>>> 3.2
print(f(1, y=2))
>>> ensure.EnsureError: Argument y to <function f at 0x109b7c710> does not match annotation type <class 'float'>
I added it to the Ensure library.
It a long time since this was asked but the example snippet given in the question is (as stated there as well) from PEP 3107 and at the end of thas PEP example Use cases are also given which might answer the question from the PEPs point of view ;)
The following is quoted from PEP3107
Use Cases
In the course of discussing annotations, a number of use-cases have been raised. Some of these are presented here, grouped by what kind of information they convey. Also included are examples of existing products and packages that could make use of annotations.
Providing typing information
Type checking ([3], [4])
Let IDEs show what types a function expects and returns ([17])
Function overloading / generic functions ([22])
Foreign-language bridges ([18], [19])
Adaptation ([21], [20])
Predicate logic functions
Database query mapping
RPC parameter marshaling ([23])
Other information
Documentation for parameters and return values ([24])
See the PEP for more information on specific points (as well as their references)
Python 3.X (only) also generalizes function definition to allow
arguments and return values to be annotated with object values
for use in extensions.
Its META-data to explain, to be more explicit about the function values.
Annotations are coded as :value after the
argument name and before a default, and as ->value after the
argument list.
They are collected into an __annotations__ attribute of the function, but are not otherwise treated as special by Python itself:
>>> def f(a:99, b:'spam'=None) -> float:
... print(a, b)
...
>>> f(88)
88 None
>>> f.__annotations__
{'a': 99, 'b': 'spam', 'return': <class 'float'>}
Source: Python Pocket Reference, Fifth Edition
EXAMPLE:
The typeannotations module provides a set of tools for type checking and type inference of Python code. It also a provides a set of types useful for annotating functions and objects.
These tools are mainly designed to be used by static analyzers such as linters, code completion libraries and IDEs. Additionally, decorators for making run-time checks are provided. Run-time type checking is not always a good idea in Python, but in some cases it can be very useful.
https://github.com/ceronman/typeannotations
How Typing Helps to Write Better Code
Typing can help you do static code analysis to catch type errors
before you send your code to production and prevent you from some
obvious bugs. There are tools like mypy, which you can add to your
toolbox as part of your software life cycle. mypy can check for
correct types by running against your codebase partially or fully.
mypy also helps you to detect bugs such as checking for the None type
when the value is returned from a function. Typing helps to make your
code cleaner. Instead of documenting your code using comments, where
you specify types in a docstring, you can use types without any
performance cost.
Clean Python: Elegant Coding in Python
ISBN: ISBN-13 (pbk): 978-1-4842-4877-5
PEP 526 -- Syntax for Variable Annotations
https://www.python.org/dev/peps/pep-0526/
https://www.attrs.org/en/stable/types.html
Despite all uses described here, the one enforceable and, most likely, enforced use of annotations will be for type hints.
This is currently not enforced in any way but, judging from PEP 484, future versions of Python will only allow types as the value for annotations.
Quoting What about existing uses of annotations?:
We do hope that type hints will eventually become the sole use for annotations, but this will require additional discussion and a deprecation period after the initial roll-out of the typing module with Python 3.5. The current PEP will have provisional status (see PEP 411 ) until Python 3.6 is released. The fastest conceivable scheme would introduce silent deprecation of non-type-hint annotations in 3.6, full deprecation in 3.7, and declare type hints as the only allowed use of annotations in Python 3.8.
Though I haven't seen any silent deprecations in 3.6 yet, this could very well be bumped to 3.7, instead.
So, even though there might be some other good use-cases, it is best to keep them solely for type hinting if you don't want to go around changing everything in a future where this restriction is in place.
As a bit of a delayed answer, several of my packages (marrow.script, WebCore, etc.) use annotations where available to declare typecasting (i.e. transforming incoming values from the web, detecting which arguments are boolean switches, etc.) as well as to perform additional markup of arguments.
Marrow Script builds a complete command-line interface to arbitrary functions and classes and allows for defining documentation, casting, and callback-derived default values via annotations, with a decorator to support older runtimes. All of my libraries that use annotations support the forms:
any_string # documentation
any_callable # typecast / callback, not called if defaulting
(any_callable, any_string) # combination
AnnotationClass() # package-specific rich annotation object
[AnnotationClass(), AnnotationClass(), …] # cooperative annotation
"Bare" support for docstrings or typecasting functions allows for easier mixing with other libraries that are annotation-aware. (I.e. have a web controller using typecasting that also happens to be exposed as a command-line script.)
Edited to add: I've also begun making use of the TypeGuard package using development-time assertions for validation. Benefit: when run with "optimizations" enabled (-O / PYTHONOPTIMIZE env var) the checks, which may be expensive (e.g. recursive) are omitted, with the idea that you've properly tested your app in development so the checks should be unnecessary in production.
Annotations can be used for easily modularizing code. E.g. a module for a program which I'm maintaining could just define a method like:
def run(param1: int):
"""
Does things.
:param param1: Needed for counting.
"""
pass
and we could ask the user for a thing named "param1" which is "Needed for counting" and should be an "int". In the end we can even convert the string given by the user to the desired type to get the most hassle free experience.
See our function metadata object for an open source class which helps with this and can automatically retrieve needed values and convert them to any desired type (because the annotation is a conversion method). Even IDEs show autocompletions right and assume that types are according to annotations - a perfect fit.
If you look at the list of benefits of Cython, a major one is the ability to tell the compiler which type a Python object is.
I can envision a future where Cython (or similar tools that compile some of your Python code) will use the annotation syntax to do their magic.
Specifically the ":int" part...
I assumed it somehow checked the type of the parameter at the time the function is called and perhaps raised an exception in the case of a violation. But the following run without problems:
def some_method(param:str):
print("blah")
some_method(1)
def some_method(param:int):
print("blah")
some_method("asdfaslkj")
In both cases "blah" is printed - no exception raised.
I'm not sure what the name of the feature is so I wasn't sure what to google.
EDIT: OK, so it's http://www.python.org/dev/peps/pep-3107/. I can see how it'd be useful in frameworks that utilize metadata. It's not what I assumed it was. Thanks for the responses!
FOLLOW-UP QUESTION - Any thoughts on whether it's a good idea or bad idea to define my functions as def some_method(param:int) if I really only can handle int inputs - even if, as pep 3107 explains, it's just metadata - no enforcement as I originally assumed? At least the consumers of the methods will see clearly what I intended. It's an alternative to documentation. Think this is good/bad/waste of time? Granted, good parameter naming (unlike my contrived example) usually makes it clear what types are meant to be passed in.
it's not used for anything much - it's just there for experimentation (you can read them from within python if you want, for example). they are called "function annotations" and are described in pep 3107.
i wrote a library that builds on it to do things like type checking (and more - for example you can map more easily from JSON to python objects) called pytyp (more info), but it's not very popular... (i should also add that the type checking part of pytyp is not at all efficient - it can be useful for tracking down a bug, but you wouldn't want to use it across an entire program).
[update: i would not recommend using function annotations in general (ie with no particular use in mind, just as docs) because (1) they might eventually get used in a way that you didn't expect and (2) the exact type of things is often not that important in python (more exactly, it's not always clear how best to specify the type of something in a useful way - objects can be quite complex, and often only "parts" are used by any one function, with multiple classes implementing those parts in different ways...). this is a consequence of duck typing - see the "more info" link for related discussion on how python's abstract base classes could be used to tackle this...]
Function annotations are what you make of them.
They can be used for documentation:
def kinetic_energy(mass: 'in kilograms', velocity: 'in meters per second'):
...
They can be used for pre-condition checking:
def validate(func, locals):
for var, test in func.__annotations__.items():
value = locals[var]
msg = 'Var: {0}\tValue: {1}\tTest: {2.__name__}'.format(var, value, test)
assert test(value), msg
def is_int(x):
return isinstance(x, int)
def between(lo, hi):
def _between(x):
return lo <= x <= hi
return _between
def f(x: between(3, 10), y: is_int):
validate(f, locals())
print(x, y)
>>> f(0, 31.1)
Traceback (most recent call last):
...
AssertionError: Var: y Value: 31.1 Test: is_int
Also see http://www.python.org/dev/peps/pep-0362/ for a way to implement type checking.
Not experienced in python, but I assume the point is to annotate/declare the parameter type that the method expects. Whether or not the expected type is rigidly enforced at runtime is beside the point.
For instance, consider:
intToHexString(param:int)
Although the language may technically allow you to call intToHexString("Hello"), it's not semantically meaningful to do so. Having the :int as part of the method declaration helps to reinforce that.
It's basically just used for documentation. When some examines the method signature, they'll see that param is labelled as an int, which will tell them the author of the method expected them to pass an int.
Because Python programmers use duck typing, this doesn't mean you have to pass an int, but it tells you the code is expecting something "int-like". So you'll probably have to pass something basically "numeric" in nature, that supports arithmetic operations. Depending on the method it may have to be usable as an index, or it may not.
However, because it's syntax and not just a comment, the annotation is visible to any code that wants to introspect it. This opens up the possibility of writing a typecheck decorator that can enforce strict type checking on arbitrary functions; this allows you to put the type checking logic in one place, and have each method declare which parameters it wants strictly type checked (by attaching a type annotation) with a minimum on syntax, in a way that is visible to client programmers who are browsing method definitions to find out the interface.
Or you could do other things with those annotations. No standardized meaning has yet been developed. Maybe if someone comes up with a killer feature that uses them and has huge adoption, then it'll one day become part of the Python language, but I suspect the flexibility of using them however you want will be too useful to ever do that.
You might also use the "-> returnValue" notation to indicate what type the function might return.
def mul(a:int, b:int) -> None:
print(a*b)
While I am aware of the duck-typing concept of Python, I sometimes struggle with the type of arguments of functions, or the type of the return value of the function.
Now, if I wrote the function myself, I DO know the types. But what if somebody wants to use and call my functions, how is he/she expected to know the types?
I usually put type information in the function's docstring (like: "...the id argument should be an integer..." and "... the function will return a (string, [integer]) tuple.")
But is looking up the information in the docstring (and putting it there, as a coder) really the way it is supposed to be done?
Edit: While the majority of answers seem to direct towards "yes, document!" I feel this is not always very easy for 'complex' types. For example: how to describe concisely in a docstring that a function returns a list of tuples, with each tuple of the form (node_id, node_name, uptime_minutes) and that the elements are respectively a string, string and integer?
The docstring PEP documentation doesn't give any guidelines on that.
I guess the counterargument will be that in that case classes should be used, but I find python very flexible because it allows passing around these things using lists and tuples, i.e. without classes.
Well things have changed a little bit since 2011! Now there's type hints in Python 3.5 which you can use to annotate arguments and return the type of your function. For example this:
def greeting(name):
return 'Hello, {}'.format(name)
can now be written as this:
def greeting(name: str) -> str:
return 'Hello, {}'.format(name)
As you can now see types, there's some sort of optional static type checking which will help you and your type checker to investigate your code.
for more explanation I suggest to take a look at the blog post on type hints in PyCharm blog.
This is how dynamic languages work. It is not always a good thing though, especially if the documentation is poor - anyone tried to use a poorly documented python framework? Sometimes you have to revert to reading the source.
Here are some strategies to avoid problems with duck typing:
create a language for your problem domain
this will help you to name stuff properly
use types to represent concepts in your domain language
name function parameters using the domain language vocabulary
Also, one of the most important points:
keep data as local as possible!
There should only be a few well-defined and documented types being passed around. Anything else should be obvious by looking at the code: Don't have weird parameter types coming from far away that you can't figure out by looking in the vicinity of the code...
Related, (and also related to docstrings), there is a technique in python called doctests. Use that to document how your methods are expected to be used - and have nice unit test coverage at the same time!
Actually there is no need as python is a dynamic language, BUT if you want to specify a return value then do this
def foo(a) -> int: #after arrow type the return type
return 1 + a
But it won't really help you much. It doesn't raise exceptions in the same way like in staticly-typed language like java, c, c++. Even if you returned a string, it won't raise any exceptions.
and then for argument type do this
def foo(a: int) -> int:
return a+ 1
after the colon (:)you can specify the argument type.
This won't help either, to prove this here is an example:
def printer(a: int) -> int: print(a)
printer("hello")
The function above actually just returns None, because we didn't return anything, but we did tell it we would return int, but as I said it doesn't help. Maybe it could help in IDEs (Not all but few like pycharm or something, but not on vscode)
I attended a coursera course, there was lesson in which, we were taught about design recipe.
Below docstring format I found preety useful.
def area(base, height):
'''(number, number ) -> number #**TypeContract**
Return the area of a tring with dimensions base #**Description**
and height
>>>area(10,5) #**Example **
25.0
>>area(2.5,3)
3.75
'''
return (base * height) /2
I think if docstrings are written in this way, it might help a lot to developers.
Link to video [Do watch the video] : https://www.youtube.com/watch?v=QAPg6Vb_LgI
Yes, you should use docstrings to make your classes and functions more friendly to other programmers:
More: http://www.python.org/dev/peps/pep-0257/#what-is-a-docstring
Some editors allow you to see docstrings while typing, so it really makes work easier.
Yes it is.
In Python a function doesn't always have to return a variable of the same type (although your code will be more readable if your functions do always return the same type). That means that you can't specify a single return type for the function.
In the same way, the parameters don't always have to be the same type too.
For example: how to describe concisely in a docstring that a function returns a list of tuples, with each tuple of the form (node_id, node_name, uptime_minutes) and that the elements are respectively a string, string and integer?
Um... There is no "concise" description of this. It's complex. You've designed it to be complex. And it requires complex documentation in the docstring.
Sorry, but complexity is -- well -- complex.
Answering my own question >10 years later, there are now 2 things I use to manage this:
type hints (as already mentioned in other answers)
dataclasses, when parameter or return type hints become unwieldy/hard to read
As an example of the latter, say I have a function
def do_something(param:int) -> list[tuple[list, int|None]]:
...
return result
I would now rewrite using a dataclass, e.g. along the lines of:
from dataclasses import dataclass
#dataclass
class Stat:
entries: list
value: int | None = None
def do_something(param:int) -> list[Stat]:
...
return result
Yes, since it's a dynamically type language ;)
Read this for reference: PEP 257
Docstrings (and documentation in general). Python 3 introduces (optional) function annotations, as described in PEP 3107 (but don't leave out docstrings)
Suppose I need to create my own small DSL that would use Python to describe a certain data structure. E.g. I'd like to be able to write something like
f(x) = some_stuff(a,b,c)
and have Python, instead of complaining about undeclared identifiers or attempting to invoke the function some_stuff, convert it to a literal expression for my further convenience.
It is possible to get a reasonable approximation to this by creating a class with properly redefined __getattr__ and __setattr__ methods and use it as follows:
e = Expression()
e.f[e.x] = e.some_stuff(e.a, e.b, e.c)
It would be cool though, if it were possible to get rid of the annoying "e." prefixes and maybe even avoid the use of []. So I was wondering, is it possible to somehow temporarily "redefine" global name lookups and assignments? On a related note, maybe there are good packages for easily achieving such "quoting" functionality for Python expressions?
I'm not sure it's a good idea, but I thought I'd give it a try. To summarize:
class PermissiveDict(dict):
default = None
def __getitem__(self, item):
try:
return dict.__getitem__(self, item)
except KeyError:
return self.default
def exec_with_default(code, default=None):
ns = PermissiveDict()
ns.default = default
exec code in ns
return ns
You might want to take a look at the ast or parser modules included with Python to parse, access and transform the abstract syntax tree (or parse tree, respectively) of the input code. As far as I know, the Sage mathematical system, written in Python, has a similar sort of precompiler.
In response to Wai's comment, here's one fun solution that I've found. First of all, to explain once more what it does, suppose that you have the following code:
definitions = Structure()
definitions.add_definition('f[x]', 'x*2')
definitions.add_definition('f[z]', 'some_function(z)')
definitions.add_definition('g.i', 'some_object[i].method(param=value)')
where adding definitions implies parsing the left hand sides and the right hand sides and doing other ugly stuff. Now one (not necessarily good, but certainly fun) approach here would allow to write the above code as follows:
#my_dsl
def definitions():
f[x] = x*2
f[z] = some_function(z)
g.i = some_object[i].method(param=value)
and have Python do most of the parsing under the hood.
The idea is based on the simple exec <code> in <environment> statement, mentioned by Ian, with one hackish addition. Namely, the bytecode of the function must be slightly tweaked and all local variable access operations (LOAD_FAST) switched to variable access from the environment (LOAD_NAME).
It is easier shown than explained: http://fouryears.eu/wp-content/uploads/pydsl/
There are various tricks you may want to do to make it practical. For example, in the code presented at the link above you can't use builtin functions and language constructions like for loops and if statements within a #my_dsl function. You can make those work, however, by adding more behaviour to the Env class.
Update. Here is a slightly more verbose explanation of the same thing.