Related
I am confused with following difference. Say I have this class with some use case:
class C:
def f(self, a, b, c=None):
print(f"Real f called with {a=}, {b=} and {c=}.")
my_c = C()
my_c.f(1, 2, c=3) # Output: Real f called with a=1, b=2 and c=3.
I can monkey patch it for purpose of testing like this:
class C:
def f(self, a, b, c=None):
print(f"Real f called with {a=}, {b=} and {c=}.")
def f_monkey_patched(self, *args, **kwargs):
print(f"Patched f called with {args=} and {kwargs=}.")
C.f = f_monkey_patched
my_c = C()
my_c.f(1, 2, c=3) # Output: Patched f called with args=(1, 2) and kwargs={'c': 3}.
So far so good. But I would like to patch only one single instance and it somehow consumes first argument:
class C:
def f(self, a, b, c=None):
print(f"Real f called with {a=}, {b=} and {c=}.")
def f_monkey_patched(self, *args, **kwargs):
print(f"Patched f called with {args=} and {kwargs=}.")
my_c = C()
my_c.f = f_monkey_patched
my_c.f(1, 2, c=3) # Output: Patched f called with args=(2,) and kwargs={'c': 3}.
Why has been first argument consumed as self instead of the instance itself?
Functions in Python are descriptors; when they're attached to a class, but looked up on an instance of the class, the descriptor protocol gets invoked, producing a bound method on your behalf (so my_c.f, where f is defined on the class, is distinct from the actual function f you originally defined, and implicitly passes my_c as self).
If you want to make a replacement that shadows the class f only for a specific instance, but still passes along the instance as self like you expect, you need to manually bind the instance to the function to create the bound method using the (admittedly terribly documented) types.MethodType:
from types import MethodType # The class implementing bound methods in Python 3
# ... Definition of C and f_monkey_patched unchanged
my_c = C()
my_c.f = MethodType(f_monkey_patched, my_c) # Creates a pre-bound method from the function and
# the instance to bind to
Being bound, my_c.f will now behave as a function that does not accept self from the caller, but when called self will be received as the instance bound to my_c at the time the MethodType was constructed.
Update with performance comparisons:
Looks like, performance-wise, all the solutions are similar enough as to be irrelevant performance-wise (Kedar's explicit use of the descriptor protocol and my use of MethodType are equivalent, and the fastest, but the percentage difference over functools.partial is so small that it won't matter under the weight of any useful work you're doing):
>>> # ... define C as per OP
>>> def f_monkey_patched(self, a): # Reduce argument count to reduce unrelated overhead
... pass
>>> from types import MethodType
>>> from functools import partial
>>> partial_c, mtype_c, desc_c = C(), C(), C()
>>> partial_c.f = partial(f_monkey_patched, partial_c)
>>> mtype_c.f = MethodType(f_monkey_patched, mtype_c)
>>> desc_c.f = f_monkey_patched.__get__(desc_c, C)
>>> %%timeit x = partial_c # Swapping in partial_c, mtype_c or desc_c
... x.f(1)
...
I'm not even going to give exact timing outputs for the IPython %%timeit magic, as it varied across runs, even on a desktop without CPU throttling involved. All I could say for sure is that partial was reliably a little slower, but only by a matter of ~1 ns (the other two typically ran in 56-56.5 ns, the partial solution typically took 56.5-57.5), and it took quite a lot of paring of extraneous stuff (e.g. switching from %timeit reading the names from global scope causing dict lookups to caching to a local name in %%timeit to use simple array lookups) to even get the differences that predictable.
Point is, any of them work, performance-wise. I'd personally recommend either my MethodType or Kedar's explicit use of descriptor protocol approach (they are identical in end result AFAICT; both produce the same bound method class), whichever one looks prettier to you, as it means the bound method is actually a bound method (so you can extract .__self__ and .__func__ like you would on any bound method constructed the normal way, where partial requires you to switch to .args[0] and .func to get the same info).
You can convert the function to bound method by calling its __get__ method (since all function as descriptors as well, thus have this method)
def t(*args, **kwargs):
print(args)
print(kwargs)
class Test():
pass
Test.t = t.__get__(Test(), Test) # binding to the instance of Test
For example
Test().t(1,2, x=1, y=2)
(<__main__.Test object at 0x7fd7f6d845f8>, 1, 2)
{'y': 2, 'x': 1}
Note that the instance is also passed as an positional argument. That is if you want you function to be instance method, the function should have been written in such a way that first argument behaves as instance of the class. Else, you can bind the function to None instance and the class, which will be like staticmethod.
Test.tt = t.__get__(None, Test)
Test.tt(1,2,x=1, y=2)
(1, 2)
{'y': 2, 'x': 1}
Furthermore, to make it a classmethod (first argument is class):
Test.ttt = t.__get__(Test, None) # bind to class
Test.ttt(1,2, x=1, y=2)
(<class '__main__.Test'>, 1, 2)
{'y': 2, 'x': 1}
When you do C.f = f_monkey_patched, and later instantiate an object of C, the function is bound to that object, effectively doing something like
obj.f = functools.partial(C.f, obj)
When you call obj.f(...), you are actually calling the partially bound function, i.e. f_monkey_patched(obj, ...)
On the other hand, doing my_c.f = f_monkey_patched, you assign the function as-is to the attribute my_c.f. When you call my_c.f(...), those arguments are passed to the function as-is, so self is the first argument you passed, i.e. 1, and the remaining arguments go to *args
I am trying to implement method overloading in Python:
class A:
def stackoverflow(self):
print 'first method'
def stackoverflow(self, i):
print 'second method', i
ob=A()
ob.stackoverflow(2)
but the output is second method 2; similarly:
class A:
def stackoverflow(self):
print 'first method'
def stackoverflow(self, i):
print 'second method', i
ob=A()
ob.stackoverflow()
gives
Traceback (most recent call last):
File "my.py", line 9, in <module>
ob.stackoverflow()
TypeError: stackoverflow() takes exactly 2 arguments (1 given)
How do I make this work?
It's method overloading, not method overriding. And in Python, you historically do it all in one function:
class A:
def stackoverflow(self, i='some_default_value'):
print('only method')
ob=A()
ob.stackoverflow(2)
ob.stackoverflow()
See the Default Argument Values section of the Python tutorial. See "Least Astonishment" and the Mutable Default Argument for a common mistake to avoid.
See PEP 443 for information about the single dispatch generic functions added in Python 3.4:
>>> from functools import singledispatch
>>> #singledispatch
... def fun(arg, verbose=False):
... if verbose:
... print("Let me just say,", end=" ")
... print(arg)
>>> #fun.register(int)
... def _(arg, verbose=False):
... if verbose:
... print("Strength in numbers, eh?", end=" ")
... print(arg)
...
>>> #fun.register(list)
... def _(arg, verbose=False):
... if verbose:
... print("Enumerate this:")
... for i, elem in enumerate(arg):
... print(i, elem)
You can also use pythonlangutil:
from pythonlangutil.overload import Overload, signature
class A:
#Overload
#signature()
def stackoverflow(self):
print('first method')
#stackoverflow.overload
#signature("int")
def stackoverflow(self, i):
print('second method', i)
While agf was right with the answer in the past, pre-3.4, now with PEP-3124 we got our syntactic sugar.
See typing documentation for details on the #overload decorator, but note that this is really just syntactic sugar and IMHO this is all people have been arguing about ever since.
Personally, I agree that having multiple functions with different signatures makes it more readable then having a single function with 20+ arguments all set to a default value (None most of the time) and then having to fiddle around using endless if, elif, else chains to find out what the caller actually wants our function to do with the provided set of arguments. This was long overdue following the Python Zen:
Beautiful is better than ugly.
and arguably also
Simple is better than complex.
Straight from the official Python documentation linked above:
from typing import overload
#overload
def process(response: None) -> None:
...
#overload
def process(response: int) -> Tuple[int, str]:
...
#overload
def process(response: bytes) -> str:
...
def process(response):
<actual implementation>
EDIT: for anyone wondering why this example is not working as you'd expect if from other languages I'd suggest to take a look at this discussion. The #overloaded functions are not supposed to have any actual implementation. This is not obvious from the example in the Python documentation.
In Python, you don't do things that way. When people do that in languages like Java, they generally want a default value (if they don't, they generally want a method with a different name). So, in Python, you can have default values.
class A(object): # Remember the ``object`` bit when working in Python 2.x
def stackoverflow(self, i=None):
if i is None:
print 'first form'
else:
print 'second form'
As you can see, you can use this to trigger separate behaviour rather than merely having a default value.
>>> ob = A()
>>> ob.stackoverflow()
first form
>>> ob.stackoverflow(2)
second form
You can't, never need to and don't really want to.
In Python, everything is an object. Classes are things, so they are objects. So are methods.
There is an object called A which is a class. It has an attribute called stackoverflow. It can only have one such attribute.
When you write def stackoverflow(...): ..., what happens is that you create an object which is the method, and assign it to the stackoverflow attribute of A. If you write two definitions, the second one replaces the first, the same way that assignment always behaves.
You furthermore do not want to write code that does the wilder of the sorts of things that overloading is sometimes used for. That's not how the language works.
Instead of trying to define a separate function for each type of thing you could be given (which makes little sense since you don't specify types for function parameters anyway), stop worrying about what things are and start thinking about what they can do.
You not only can't write a separate one to handle a tuple vs. a list, but also don't want or need to.
All you do is take advantage of the fact that they are both, for example, iterable (i.e. you can write for element in container:). (The fact that they aren't directly related by inheritance is irrelevant.)
I write my answer in Python 3.2.1.
def overload(*functions):
return lambda *args, **kwargs: functions[len(args)](*args, **kwargs)
How it works:
overload takes any amount of callables and stores them in tuple functions, then returns lambda.
The lambda takes any amount of arguments,
then returns result of calling function stored in functions[number_of_unnamed_args_passed] called with arguments passed to the lambda.
Usage:
class A:
stackoverflow=overload( \
None, \
#there is always a self argument, so this should never get called
lambda self: print('First method'), \
lambda self, i: print('Second method', i) \
)
I think the word you're looking for is "overloading". There isn't any method overloading in Python. You can however use default arguments, as follows.
def stackoverflow(self, i=None):
if i != None:
print 'second method', i
else:
print 'first method'
When you pass it an argument, it will follow the logic of the first condition and execute the first print statement. When you pass it no arguments, it will go into the else condition and execute the second print statement.
I write my answer in Python 2.7:
In Python, method overloading is not possible; if you really want access the same function with different features, I suggest you to go for method overriding.
class Base(): # Base class
'''def add(self,a,b):
s=a+b
print s'''
def add(self,a,b,c):
self.a=a
self.b=b
self.c=c
sum =a+b+c
print sum
class Derived(Base): # Derived class
def add(self,a,b): # overriding method
sum=a+b
print sum
add_fun_1=Base() #instance creation for Base class
add_fun_2=Derived()#instance creation for Derived class
add_fun_1.add(4,2,5) # function with 3 arguments
add_fun_2.add(4,2) # function with 2 arguments
In Python, overloading is not an applied concept. However, if you are trying to create a case where, for instance, you want one initializer to be performed if passed an argument of type foo and another initializer for an argument of type bar then, since everything in Python is handled as object, you can check the name of the passed object's class type and write conditional handling based on that.
class A:
def __init__(self, arg)
# Get the Argument's class type as a String
argClass = arg.__class__.__name__
if argClass == 'foo':
print 'Arg is of type "foo"'
...
elif argClass == 'bar':
print 'Arg is of type "bar"'
...
else
print 'Arg is of a different type'
...
This concept can be applied to multiple different scenarios through different methods as needed.
In Python, you'd do this with a default argument.
class A:
def stackoverflow(self, i=None):
if i == None:
print 'first method'
else:
print 'second method',i
Python does not support method overloading like Java or C++. We may overload the methods, but we can only use the latest defined method.
# First sum method.
# Takes two argument and print their sum
def sum(a, b):
s = a + b
print(s)
# Second sum method
# Takes three argument and print their sum
def sum(a, b, c):
s = a + b + c
print(s)
# Uncommenting the below line shows an error
# sum(4, 5)
# This line will call the second sum method
sum(4, 5, 5)
We need to provide optional arguments or *args in order to provide a different number of arguments on calling.
Courtesy Python | Method Overloading
I just came across overloading.py (function overloading for Python 3) for anybody who may be interested.
From the linked repository's README file:
overloading is a module that provides function dispatching based on
the types and number of runtime arguments.
When an overloaded function is invoked, the dispatcher compares the
supplied arguments to available function signatures and calls the
implementation that provides the most accurate match.
Features
Function validation upon registration and detailed resolution rules
guarantee a unique, well-defined outcome at runtime. Implements
function resolution caching for great performance. Supports optional
parameters (default values) in function signatures. Evaluates both
positional and keyword arguments when resolving the best match.
Supports fallback functions and execution of shared code. Supports
argument polymorphism. Supports classes and inheritance, including
classmethods and staticmethods.
Python 3.x includes standard typing library which allows for method overloading with the use of #overload decorator. Unfortunately, this is to make the code more readable, as the #overload decorated methods will need to be followed by a non-decorated method that handles different arguments.
More can be found here here but for your example:
from typing import overload
from typing import Any, Optional
class A(object):
#overload
def stackoverflow(self) -> None:
print('first method')
#overload
def stackoverflow(self, i: Any) -> None:
print('second method', i)
def stackoverflow(self, i: Optional[Any] = None) -> None:
if not i:
print('first method')
else:
print('second method', i)
ob=A()
ob.stackoverflow(2)
Python added the #overload decorator with PEP-3124 to provide syntactic sugar for overloading via type inspection - instead of just working with overwriting.
Code example on overloading via #overload from PEP-3124
from overloading import overload
from collections import Iterable
def flatten(ob):
"""Flatten an object to its component iterables"""
yield ob
#overload
def flatten(ob: Iterable):
for o in ob:
for ob in flatten(o):
yield ob
#overload
def flatten(ob: basestring):
yield ob
is transformed by the #overload-decorator to:
def flatten(ob):
if isinstance(ob, basestring) or not isinstance(ob, Iterable):
yield ob
else:
for o in ob:
for ob in flatten(o):
yield ob
In the MathMethod.py file:
from multipledispatch import dispatch
#dispatch(int, int)
def Add(a, b):
return a + b
#dispatch(int, int, int)
def Add(a, b, c):
return a + b + c
#dispatch(int, int, int, int)
def Add(a, b, c, d):
return a + b + c + d
In the Main.py file
import MathMethod as MM
print(MM.Add(200, 1000, 1000, 200))
We can overload the method by using multipledispatch.
There are some libraries that make this easy:
functools - if you only need the first argument use #singledispatch
plum-dispatch - feature rich method/function overloading.
multipledispatch - alternative to plum less features but lightweight.
python 3.5 added the typing module. This included an overload decorator.
This decorator's intended purpose it to help type checkers. Functionally its just duck typing.
from typing import Optional, overload
#overload
def foo(index: int) -> str:
...
#overload
def foo(name: str) -> str:
...
#overload
def foo(name: str, index: int) -> str:
...
def foo(name: Optional[str] = None, index: Optional[int] = None) -> str:
return f"name: {name}, index: {index}"
foo(1)
foo("bar", 1)
foo("bar", None)
This leads to the following type information in vs code:
And while this can help, note that this adds lots of "weird" new syntax. Its purpose - purely type hints - is not immediately obvious.
Going with Union of types usually is a better option.
Let's say that I have the function
def add(a,b,c):
return a+b+c
I want a decorator that fixes the value of b, say to 5, and return a function with only two parameters a and c.
def add5(a,c):
return a+c+5
The function add5 should not have any other parameter. I'm not looking to solve this with a default parameters for b.
You can use functools.partial:
functools.partial(func, /, *args, **keywords)
Return a new partial
object which when called will behave like func called with the
positional arguments args and keyword arguments keywords.
from functools import partial
def add(a,b,c):
return a+b+c
If you want to give a fixed value to the first positional argument, you can do
add5 = partial(add, 5)
print(add5(1, 2))
# 8
As the first positional argument (a) will be replaced by 5, you can't do:
print(add5(a=3, b=4))
# TypeError: add() got multiple values for argument 'a'
If you want to control which parameter you fix, use keyword arguments:
add5 = partial(add, b=5)
print(add5(a=1, c=2))
# 8
In Python, functions are the first class objects, which means that –
Functions are objects; they can be referenced to, passed to a variable and returned from other functions as well.
Functions can be defined inside another function and can also be passed as argument to another function.
Decorators are very powerful and useful tool in Python since it allows programmers to modify the behavior of function or class. Decorators allow us to wrap another function in order to extend the behavior of wrapped function, without permanently modifying it.
In Decorators, functions are taken as the argument into another function and then called inside the wrapper function.
in your case:
def my_custom_decorator(f):
def outer_function(*args):
res = f(*args)
return res + 5
return outer_function
#my_custom_decorator
def A_and_C(a, c):
return a+c
print(A_and_C(2,3))
You can do it by
def add5(*arg):
return sum(args)+5
print(add5(1,2))
This will sum all the argument that you are passing to the function and will add 5 to the sum of the args.
Output
8
Why do the following lines give me the same result?
str.upper('hello')
and
'hello'.upper()
I tried to do the same with list.append but got a TypeError.
list.append([1])
Is the str type in Python overloaded? How can this be achieved by writing a class/function? I would appreciate an example.
list.append takes two arguments - the list to modify and the element to append. So you need to do it like this:
ls = [1]
list.append(ls, 2)
which is equivalent to the much more popular:
ls.append(2)
str.upper and list.append are both functions.
str.upper takes one argument.
>>> str.upper('test')
'TEST'
list.append takes two arguments.
>>> my_list = []
>>> list.append(my_list, 1)
>>> my_list
[1]
str.upper and list.append (like other functions) are also non-data-descriptors with a __get__ method which in this context has two implications:
When you access the function through the class via the dot notation (str.upper, list.append) the function's __get__ method (i.e. string.upper.__get__ and list.append.__get__) is called but it returns just the function itself.
When you access the function through an instance (my_string.upper, my_list.append) the function's __get__ method is called and it will return a new callable acting like the original function, but with whatever was "in front of the dot" automatically passed as the first argument. .
That's why you need to pass 1 - 1 = 0 arguments when calling my_string.upper() and 2 - 1 = 1 argument when calling my_list.append(1).
>>> 'my_string'.upper()
'MY_STRING'
>>>
>>> my_list = []
>>> my_list.append(1)
>>> my_list
[1]
You could even get these modified callables (methods) by explicitly calling __get__ and passing the argument to be bound (what has been before the dot) as its argument.
>>> my_string = 'my_string'
>>> upper_maker = str.upper.__get__(my_string)
>>> upper_maker()
'MY_STRING'
>>>
>>> my_list = []
>>> appender = list.append.__get__(my_list)
>>> appender(1)
>>> my_list
[1]
Finally, here's a short example demonstrating how descriptor instances can detect whether they are being accessed via their owner-class or via an instance.
class Descriptor:
def __get__(self, instance, owner_class):
if instance is None:
print('accessed through class')
# list.append.__get__ would return list.append here
else:
print('accessed through instance')
# list.append.__get__ would build a new callable here
# that takes one argument x and that internally calls
# list.append(instance, x)
class Class:
attribute = Descriptor()
Class.attribute # prints 'accessed through class'
instance = Class()
instance.attribute # prints 'accessed through instance'
Quoting Dave Kirbys answer from Relationship between string module and str:
There is some overlap between the string module and the str type,
mainly for historical reasons. In early versions of Python str objects
did not have methods, so all string manipulation was done with
functions from the string module. When methods were added to the str
type (in Python 1.5?) the functions were left in the string module for
compatibility, but now just forward to the equivalent str method.
However the string module also contains constants and functions that
are not methods on str, such as formatting, character translation etc.
There is nothing at all magical going on with str (except that we have a nice syntactic shortcut to creating one using ""). You can write a class that behaves like str and list to see more clearly what is happening here.
class MyClass():
def __init__(self, arg):
self.val=str(arg)
def do_thing(self):
self.val = "asdf"
def do_thing_with_arg(self, arg):
self.val = "asdf " + str(arg)
def __repr__(self):
return self.val
my_thing = MyClass("qwerty")
# this is like 'hello'.upper()
my_thing.do_thing()
print(my_thing)
# it prints 'asdf'
my_thing = MyClass("qwerty")
# this is like str.upper('hello')
MyClass.do_thing(my_thing)
print(my_thing)
# it prints 'asdf'
my_thing = MyClass("qwerty")
# this is like my_list.append('qwerty')
my_thing.do_thing_with_arg('zxcv')
print(my_thing)
# it prints 'asdf zxcv'
my_thing = MyClass("qwerty")
# this is like list.append(my_list, 'qwerty')
MyClass.do_thing_with_arg(my_thing, 'zxcv')
print(my_thing)
# it prints 'asdf zxcv'
The short version is, you're invoking what looks like an "instance method" on a class, but you are supplying the instance ('self') yourself as the first argument to the function call.
This question already has answers here:
Can a variable number of arguments be passed to a function?
(6 answers)
How can I find the number of arguments of a Python function?
(14 answers)
Expanding tuples into arguments
(5 answers)
Closed 8 years ago.
I wondering if it is possible to pass functions with different number of arguments to another function in Python.
For example
def aux_1(arg1,arg2):
if arg1 > arg2: return 1
else: return 0
def aux_2(arg1,arg2,arg3):
return arg1 + arg2 * arg3
def Main_fn(function_to_use,arg1, arg2, arg3):
return function_to_use(?)
>>>Main_fn(aux_1,1,2,1) #Example cases
>>>Main_fn(aux_2,1,2,1)
So this is my doubt, if I pass function aux_1 how could I ensure it uses only arg1 and arg2. Similarly, if I use aux_2 all the three arguments need to used.
I know it is possible to use a set of if conditions in the Main_fn, using inspect.getargspec()/varargs to appropriately pass on the arguments. However, such a logic would require modifications to the Main_fn each time I make a new aux function to handle its arguments, which is not practical in a large development scenario.
Hence, I'm looking for a way around this.
Please feel free to question me, if in need of further clarifications.
Yes, you can use inspect.getargspec to figure out how many arguments a function was defined to take:
import inspect
def get_num_args(f):
return len(inspect.getargspec(f)[0])
Therefore, you may use get_num_args to decide how many arguments to send in:
if get_num_args(function_to_use) == 2:
function_to_use(arg1, arg2)
else:
function_to_use(arg1, arg2, arg3)
Its possible but first when you want to return a function you must be aware that how you invoke the main function ! see this example:
>>> def a():
... print 3
...
>>> def b(func):
... return func
...
>>> print b(a)
<function a at 0x7feb48e30de8>
>>> b(a())
3
and when you want to use args on inner function when you pass your function to main func you must pass your arguments not when you define the main func ... so you must change this
def Main_fn(function_to_use,arg1, arg2, arg3):
return function_to_use(?)
to this:
def Main_fn(function_to_use,arg1, arg2, arg3):
return function_to_use
#Also note that when you pass arg1, arg2, arg3 in func you must pass them when you invoke the function , if you dont need them you must not pass them in define
an example :
>>> def a(i,j):
... print i+j
...
>>> def b(func):
... return func
...
>>> b(a(3,5))
8
>>>