I know it's not Pythonic to write functions that care about the type of the arguments, but there are cases when it's simply impossible to ignore types because they are handled differently.
Having a bunch of isinstance checks in your function is just ugly; is there any function decorator available that enables function overloads? Something like this:
#overload(str)
def func(val):
print('This is a string')
#overload(int)
def func(val):
print('This is an int')
Update:
Here's some comments I left on David Zaslavsky's answer:
With a few modification[s], this will suit my purposes pretty well. One other limitation I noticed in your implementation, since you use func.__name__ as the dictionary key, you are prone to name collisions between modules, which is not always desirable. [cont'd]
[cont.] For example, if I have one module that overloads func, and another completely unrelated module that also overloads func, these overloads will collide because the function dispatch dict is global. That dict should be made local to the module, somehow. And not only that, it should also support some kind of 'inheritance'. [cont'd]
[cont.] By 'inheritance' I mean this: say I have a module first with some overloads. Then two more modules that are unrelated but each import first; both of these modules add new overloads to the already existing ones that they just imported. These two modules should be able to use the overloads in first, but the new ones that they just added should not collide with each other between modules. (This is actually pretty hard to do right, now that I think about it.)
Some of these problems could possibly be solved by changing the decorator syntax a little bit:
first.py
#overload(str, str)
def concatenate(a, b):
return a + b
#concatenate.overload(int, int)
def concatenate(a, b):
return str(a) + str(b)
second.py
from first import concatenate
#concatenate.overload(float, str)
def concatenate(a, b):
return str(a) + b
Since Python 3.4 the functools module supports a #singledispatch decorator. It works like this:
from functools import singledispatch
#singledispatch
def func(val):
raise NotImplementedError
#func.register
def _(val: str):
print('This is a string')
#func.register
def _(val: int):
print('This is an int')
Usage
func("test") --> "This is a string"
func(1) --> "This is an int"
func(None) --> NotImplementedError
Quick answer: there is an overload package on PyPI which implements this more robustly than what I describe below, although using a slightly different syntax. It's declared to work only with Python 3 but it looks like only slight modifications (if any, I haven't tried) would be needed to make it work with Python 2.
Long answer: In languages where you can overload functions, the name of a function is (either literally or effectively) augmented by information about its type signature, both when the function is defined and when it is called. When a compiler or interpreter looks up the function definition, then, it uses both the declared name and the types of the parameters to resolve which function to access. So the logical way to implement overloading in Python is to implement a wrapper that uses both the declared name and the parameter types to resolve the function.
Here's a simple implementation:
from collections import defaultdict
def determine_types(args, kwargs):
return tuple([type(a) for a in args]), \
tuple([(k, type(v)) for k,v in kwargs.iteritems()])
function_table = defaultdict(dict)
def overload(arg_types=(), kwarg_types=()):
def wrap(func):
named_func = function_table[func.__name__]
named_func[arg_types, kwarg_types] = func
def call_function_by_signature(*args, **kwargs):
return named_func[determine_types(args, kwargs)](*args, **kwargs)
return call_function_by_signature
return wrap
overload should be called with two optional arguments, a tuple representing the types of all positional arguments and a tuple of tuples representing the name-type mappings of all keyword arguments. Here's a usage example:
>>> #overload((str, int))
... def f(a, b):
... return a * b
>>> #overload((int, int))
... def f(a, b):
... return a + b
>>> print f('a', 2)
aa
>>> print f(4, 2)
6
>>> #overload((str,), (('foo', int), ('bar', float)))
... def g(a, foo, bar):
... return foo*a + str(bar)
>>> #overload((str,), (('foo', float), ('bar', float)))
... def g(a, foo, bar):
... return a + str(foo*bar)
>>> print g('a', foo=7, bar=4.4)
aaaaaaa4.4
>>> print g('b', foo=7., bar=4.4)
b30.8
Shortcomings of this include
It doesn't actually check that the function the decorator is applied to is even compatible with the arguments given to the decorator. You could write
#overload((str, int))
def h():
return 0
and you'd get an error when the function was called.
It doesn't gracefully handle the case where no overloaded version exists corresponding to the types of the arguments passed (it would help to raise a more descriptive error)
It distinguishes between named and positional arguments, so something like
g('a', 7, bar=4.4)
doesn't work.
There are a lot of nested parentheses involved in using this, as in the definitions for g.
As mentioned in the comments, this doesn't deal with functions having the same name in different modules.
All of these could be remedied with enough fiddling, I think. In particular, the issue of name collisions is easily resolved by storing the dispatch table as an attribute of the function returned from the decorator. But as I said, this is just a simple example to demonstrate the basics of how to do it.
This doesn't directly answer your question, but if you really want to have something that behaves like an overloaded function for different types and (quite rightly) don't want to use isinstance then I'd suggest something like:
def func(int_val=None, str_val=None):
if sum(x != None for x in (int_val, str_val)) != 1:
#raise exception - exactly one value should be passed in
if int_val is not None:
print('This is an int')
if str_val is not None:
print('This is a string')
In use the intent is obvious, and it doesn't even require the different options to have different types:
func(int_val=3)
func(str_val="squirrel")
Yes, there is an overload decorator in the typing library that can be used to help make complex type hints easier.
from collections.abc import Sequence
from typing import overload
#overload
def double(input_: int) -> int:
...
#overload
def double(input_: Sequence[int]) -> list[int]:
...
def double(input_: int | Sequence[int]) -> int | list[int]:
if isinstance(input_, Sequence):
return [i * 2 for i in input_]
return input_ * 2
Check this link for more details.
Just noticed it is a 11 years old question, sorry to bring it up again. It was by mistake.
Related
Consider having a function that returns a complex value:
def my_fn():
return (create_this(), create_that(), someotherstuff)
Assyming pylance knows what create_this() returns as well as what the other values are, it will implicitly tell you my_fn returns a Tuple[Type1, Type2, Type3].
Now let's say you have a function that expects to receive an argument that contains whatever this function returned, but you want to still get type hints. You can do this:
def process_fn_value(data: Tuple[Type1, Type2, Type3]):
...
But that's rather verbose, isn't it. It would be better to just write:
def process_fn_value(data: ReturnOf[my_fn]):
...
I have tried the following, hoping to extract the type from a function by making a generic type and then calling type() on it. But it doesn't even properly decode the type of the generic value:
T = TypeVar('T')
def RetVal(cb: Callable[[Any], T]):
return type(cb())
def test_fn():
return "test"
def test_consumer(arg: RetVal[test_fn]):
return arg
Another thing I tried, mostly after looking how Generic[T] is implemented:
class ReturnValue(Type[T], _root=True):
def __new__(func, cb: Callable[[], Generic[T]]) -> T:
return type(cb())
def test_fn():
return [1,2,3]
def test_consumer(arg: ReturnValue[test_fn]):
return arg
testtype = ReturnValue(test_fn)
None of these work.
Is there any such type hint in Python?
Note: If you think that this is a problem I shouldn't be facing if I wrote the code in such and such way, maybe you're right. But please consider sometimes one cannot change EVERYTHING and yet might be able to create at least partial improvement in the codebase.
There is such a function for currying. The problem is that I don’t know how to make this function return a decorated function with the correct types. Help, I have not found a solution anywhere.
import functools
import typing as ty
from typing import TypeVar, Callable, Any, Optional
F = TypeVar("F", bound=Callable[..., Any])
def curry(func: F, max_argc: Optional[int] = None):
if max_argc is None:
max_argc = func.__code__.co_argcount
#functools.wraps(func)
def wrapped(*args):
argc = len(args)
if argc < max_argc:
return curry(functools.partial(func, *args), max_argc - argc)
else:
return func(*args)
return ty.cast(F, wrapped)
#curry
def foo(x: int, y: int) -> int:
return x + y
foo("df")(5) # mypy error: Too few arguments for "foo"
# mypy error: "int" not callable
# mypy error: Argument 1 to "foo" has incompatible type "str"; expected "int" # True
How to fix 1, 2 mypy errors?
Interestingly enough, i tried the exact same thing, writing a decorator, which would return a curried version of any function, including generic higher-order ones.
I tried to build a curry that would allow any partitioning of the input arguments.
Denial
However, AFAIK it is not possible due to some constraints of the python type system.
I'm struggling to find a generic typesafe approach right now. I mean Haskell lives it, C++ adopted it, Typescript manages to do it as well, so why should python not be able to?
I reached out to mypy alternatives such as pyright, which has AWESOME maintainers, BUT is stull bound by PEPs, which state that some things are just not possible.
Anger
When submitting an issue with pyright for the last missing piece in my curry-chain, i boiled the issue down to the following (as can be seen in this issue: https://github.com/microsoft/pyright/issues/1720)
from typing import TypeVar
from typing_extensions import Protocol
R = TypeVar("R", covariant=True)
S = TypeVar("S", contravariant=True)
class ArityOne(Protocol[S, R]):
def __call__(self, __s: S) -> R:
...
def id_f(x: ArityOne[S, R]) -> ArityOne[S, R]:
return x
X = TypeVar("X")
def identity(x: X) -> X:
return x
i: int = id_f(identity)(4) # Does not type check, expected type `X`, got type `Literal[4]`
Mind the gap, this is a minimal reproducible example of the missing link.
What i initially attempted to do was the following (skipping the actual curry implementation, which, in comparison, resembles a walk in the park):
Write a curry decorator (without types)
Define Unary, Binary and Ternary (etc.) Protocols, which is the more modern version of the function type Callable. Coincidentally, Protocols can specify type #overloads for their __call__ methods, which brings me to the next point.
Define CurriedBinary, CurriedTernary (etc.) using Protocols with type #overloaded __call__ methods.
Define type #overloads for the curry function, e.g. Binary -> CurriedBinary, Ternary -> CurriedTernary
With this, everything was in place, and it works awesome for fixed-type functions i.e. int -> int or str -> int -> bool.
I don't have my attempted implementation on hand right now tho'.
However, when currying functions such as map, or filter it fails to match the curried, generic version of the function with the actual types.
Bargaining
This happens due to how scoping of type variables works. For more details you can take a look at the GitHub issue. It is explained in greater detail there.
Essentially what happens is that the type variable of the generic function-to-be-curried cannot be influenced by the actual type of the data-to-be-passed partially, because there is a class Protocol in between, which defines its own scope of type variables.
Trying to wrap or reorganize stuff did not yield fruitful results.
Depression
I used Protocols there to be able to represent the type of a curried function,
which is not possible with e.g. Callable, and although pyright displays the type of an overloaded function as Overload[contract1, contract2, ...] there is no such symbol, only #overload.
So either way there's something that prevents you from expressing the type you want.
It is currently not possible to represent a fully generic typesafe curry function due to limitations of the python type system.
Acceptance
However, it is possible to compromise on certain features, like generics, or arbitrary partitioning of input arguments.
The following implementation of curry works in pyright 1.1.128.
from typing import TypeVar, Callable, List, Optional, Union
R = TypeVar("R", covariant=True)
S = TypeVar("S", contravariant=True)
T = TypeVar("T", contravariant=True)
def curry(f: Callable[[T, S], R]) -> Callable[[T], Callable[[S], R]]:
raise Exception()
X = TypeVar("X")
Y = TypeVar("Y")
def function(x: X, y: X) -> X:
raise Exception()
def to_optional(x: X) -> Optional[X]:
raise Exception()
def map(f: Callable[[X], Y], xs: List[X]) -> List[Y]:
raise Exception()
i: int = curry(function)(4)(5)
s: List[Optional[Union[str, int]]] = curry(map)(to_optional)(["dennis", 4])
First things first, I wouldn't make it a decorator, I'd wrap it as curry(foo). I find it confusing to look at an API where the decorated function signature is different to its initial definition.
On the subject of types, I would be very impressed if the general case is possible with Python type hints. I'm not sure how I'd even do it in Scala. You can do a limited number of cases, using overload for functions of two parameters as
T1 = TypeVar("T1")
T2 = TypeVar("T2")
U = TypeVar("U")
#overload
def curry(
func: Callable[[T1, T2], U],
max_argc: Optional[int]
) -> Callable[[T1], Callable[[T2], U]]:
...
adding versions for one, three, four parameters etc. Functions with lots of parameters are code smells anyway, with the exception of varargs, which I'm not sure if it even makes sense to curry.
I'm trying to create a function that chains results from multiple arguments.
def hi(string):
print(string)<p>
return hi
Calling hi("Hello")("World") works and becomes Hello \n World as expected.
the problem is when I want to append the result as a single string, but
return string + hi produces an error since hi is a function.
I've tried using __str__ and __repr__ to change how hi behaves when it has not input. But this only creates a different problem elsewhere.
hi("Hello")("World") = "Hello"("World") -> Naturally produces an error.
I understand why the program cannot solve it, but I cannot find a solution to it.
You're running into difficulty here because the result of each call to the function must itself be callable (so you can chain another function call), while at the same time also being a legitimate string (in case you don't chain another function call and just use the return value as-is).
Fortunately Python has you covered: any type can be made to be callable like a function by defining a __call__ method on it. Built-in types like str don't have such a method, but you can define a subclass of str that does.
class hi(str):
def __call__(self, string):
return hi(self + '\n' + string)
This isn't very pretty and is sorta fragile (i.e. you will end up with regular str objects when you do almost any operation with your special string, unless you override all methods of str to return hi instances instead) and so isn't considered very Pythonic.
In this particular case it wouldn't much matter if you end up with regular str instances when you start using the result, because at that point you're done chaining function calls, or should be in any sane world. However, this is often an issue in the general case where you're adding functionality to a built-in type via subclassing.
To a first approximation, the question in your title can be answered similarly:
class add(int): # could also subclass float
def __call__(self, value):
return add(self + value)
To really do add() right, though, you want to be able to return a callable subclass of the result type, whatever type it may be; it could be something besides int or float. Rather than trying to catalog these types and manually write the necessary subclasses, we can dynamically create them based on the result type. Here's a quick-and-dirty version:
class AddMixIn(object):
def __call__(self, value):
return add(self + value)
def add(value, _classes={}):
t = type(value)
if t not in _classes:
_classes[t] = type("add_" + t.__name__, (t, AddMixIn), {})
return _classes[t](value)
Happily, this implementation works fine for strings, since they can be concatenated using +.
Once you've started down this path, you'll probably want to do this for other operations too. It's a drag copying and pasting basically the same code for every operation, so let's write a function that writes the functions for you! Just specify a function that actually does the work, i.e., takes two values and does something to them, and it gives you back a function that does all the class munging for you. You can specify the operation with a lambda (anonymous function) or a predefined function, such as one from the operator module. Since it's a function that takes a function and returns a function (well, a callable object), it can also be used as a decorator!
def chainable(operation):
class CallMixIn(object):
def __call__(self, value):
return do(operation(self, value))
def do(value, _classes={}):
t = type(value)
if t not in _classes:
_classes[t] = type(t.__name__, (t, CallMixIn), {})
return _classes[t](value)
return do
add = chainable(lambda a, b: a + b)
# or...
import operator
add = chainable(operator.add)
# or as a decorator...
#chainable
def add(a, b): return a + b
In the end it's still not very pretty and is still sorta fragile and still wouldn't be considered very Pythonic.
If you're willing to use an additional (empty) call to signal the end of the chain, things get a lot simpler, because you just need to return functions until you're called with no argument:
def add(x):
return lambda y=None: x if y is None else add(x+y)
You call it like this:
add(3)(4)(5)() # 12
You are getting into some deep, Haskell-style, type-theoretical issues by having hi return a reference to itself. Instead, just accept multiple arguments and concatenate them in the function.
def hi(*args):
return "\n".join(args)
Some example usages:
print(hi("Hello", "World"))
print("Hello\n" + hi("World"))
Is there a "standard" way to add simple dynamic type checking in Python, doing something like:
def add(a, b):
# Argument type check
check(a, int)
check(b, int)
# Calculate
res = a + b
# Result type check and return
check(res, int)
return res
An exception could then be raised by check in case of a type mismatch.
I could of course cook something myself, doing isinstance(..., ...) or type(...) == ..., but I wonder if there is some "standard" module for this kind of type checking.
It would be nice if more complex type checking could also be done, like checking if an argument is either str or int, or for example a list of str.
I am aware that it somehow defies Pythons principle of duck typing, but I just spent several hours debugging due to an argument with wrong type, and it was a large program, so the cause shown up many nested calls from the reason.
You could use a decorator function. Something like this:
def typecheck(*types):
def __f(f):
def _f(*args):
for a, t in zip(args, types):
if not isinstance(a, t):
print "WARNING: Expected", t, "got", a
return f(*args)
return _f
return __f
#typecheck(int, str, int)
def foo(a, b, c):
pass
foo(1, "bar", 5)
foo(4, None, "string")
Output (for the second call) is
WARNING: Expected <type 'str'>, got None
WARNING: Expected <type 'int'>, got 'string'
As it stands, this does not work for keyword parameters, though.
Edit: After some googling, I found some much more complex type checking decorators (1) (2) also supporting keyword parameters and return types.
There is mypy which is being considered for entry into Python proper but in general, there isn't any way to do what you want.
Your code should not depend on concrete types but on general interfaces (e.g. not whether two things are integers but whether they are "addable"). This allows you to take advantage of dynamic typing and write generic functions. If a type does not handle the interface you want, it will throw an exception which you can catch. So, your add would be better done like so.
def add(a, b):
try:
return a + b
except TypeError as t:
# some logging code here for your debugging ease
raise t
If you are on Python 3, there is optional type annotation for functions. This means that the following code is valid Python 3.
def add(a:int, b:int):
return a + b
I don't know if there any tools that take advantage of the hints to give you actual compile time checking though.
I'm trying to make a function that does different things when called on different argument types. Specifically, one of the functions should have the signature
def myFunc(string, string):
and the other should have the signature
def myFunc(list):
How can I do this, given that I'm not allowed to specify whether the arguments are strings or lists?
Python does not support overloading, even by the argument count. You need to do:
def foo(string_or_list, string = None):
if isinstance(string_or_list, list):
...
else:
...
which is pretty silly, or just rethink your design to not have to overload.
There is a recipe at http://code.activestate.com/recipes/577065-type-checking-function-overloading-decorator/ which does what you want;
basically, you wrap each version of your function with #takes and #returns type declarations; when you call the function, it tries each version until it finds one that does not throw a type error.
Edit: here is a cut-down version; it's probably not a good thing to do, but if you gotta, here's how:
from collections import defaultdict
def overloaded_function(overloads):
"""
Accepts a sequence of ((arg_types,), fn) pairs
Creates a dispatcher function
"""
dispatch_table = defaultdict(list)
for arg_types,fn in overloads:
dispatch_table[len(arg_types)].append([list(arg_types),fn])
def dispatch(*args):
for arg_types,fn in dispatch_table[len(args)]:
if all(isinstance(arg, arg_type) for arg,arg_type in zip(args,arg_types)):
return fn(*args)
raise TypeError("could not find an overloaded function to match this argument list")
return dispatch
and here's how it works:
def myfn_string_string(s1, s2):
print("Got the strings {} and {}".format(s1, s2))
def myfn_list(lst):
print("Got the list {}".format(lst))
myfn = overloaded_function([
((basestring, basestring), myfn_string_string),
((list,), myfn_list)
])
myfn("abcd", "efg") # prints "Got the strings abcd and efg"
myfn(["abc", "def"]) # prints "Got the list ['abc', 'def']"
myfn(123) # raises TypeError
*args is probably the better way, but you could do something like:
def myFunc(arg1, arg2=None):
if arg2 is not None:
#do this
else:
#do that
But that's probably a terrible way of doing it.
Not a perfect solution, but if the second string argument will never legitimately be None, you could try:
def myFunc( firstArg, secondArg = None ):
if secondArg is None:
# only one arg provided, try treating firstArg as a list
else:
# two args provided, try treating them both as strings
Define it as taking variable arguments:
def myFunc(*args):
Then you can check the amount and type of the arguments via len and isinstance, and route the call to the appropriate case-specific function.
It may make for clearer code, however, if you used optional named arguments. It would be better still if you didn't use overloading at all, it's kinda not python's way.
You can't - for instance a class instance method can be inserted in run-time.
If you had multiple __init__ for a class for instance, you'd be better off with multiple #classmethod's such as from_strings or from_sequence