Problem
As the title suggest, I am trying to use functools.partial to create a default setting for a callable. However, some of the parameters depend on each other.
Background
Imagine I have a function _print_slice(items: list[int], start: int, end: int) that takes in a list and prints items from start to end.
_print_slice definition
def _print_slice(items: list[int], start: int, end: int) -> None:
print(items[start:end])
I want to make a partial version of this function called print_list that prints the entire list, i.e, start = 0 and end = len(items)
print_list definition
def print_list(items: list[int]) -> None:
_print_slice(items=items, start=0, end=len(items))
notice that print_list is just a wrapper around _print_slice. If I am not mistaken, this would be a perfect use case for functools.partial, however, I am not sure to use use partial to accomplish this given that end = len(items), please help.
Disclaimer
This is a very simplified version of the problem to highlight what I am trying to accomplish.
This is not a job for functools.partial. It cannot perform the kind of dependent argument computation you want. functools.partial is designed to fix argument values, not to compute them from other argument values.
(Due to the way slicing handles None endpoints, partial(_print_slice, start=None, end=None) would have the behavior you want, but that's something the list slicing implementation would handle, not something partial handles - it wouldn't generalize to other uses of partial. Also, it violates the _print_slice argument annotations.)
Related
I was trying to create some code that can solve problems with sums.
For example, if you want to take the sum of 4*i for all values of i from 3, 109, I want this code to be able to do that. However, it should also be able to deal with more complicated things than just multiplication. See a sample of what I want to do below
from typing import Callable
class MyClass:
def __init__(self):
pass
def function_sum(self, lower_bound: int, upper_bound: int, function: Callable, *args):
return sum((function(*args) for i in range(lower_bound, upper_bound+1)))
print(MyClass().function_sum(1, 10, lambda x: x*i, 1))
Is there a way to use the iterable i which is used inside the function, as a part of the function in the parameter, without forcing i to be a parameter?
from typing import Callable
class MyClass:
def __init__(self):
pass
def function_sum(self, lower_bound: int, upper_bound: int, func: Callable, *args):
# i is forced to be a parameter in function
return sum((func(*args, i) for i in range(lower_bound, upper_bound+1)))
print(MyClass().function_sum(1, 10, lambda x: x*i, 1))
There's no reasonable way to do this; functions bind their scope at definition time, based on where they're defined, not called. Python doesn't directly support swapping out scopes like that.
You typically don't want to do this anyway, as it's a truly terrible design. Think about it: Even if this works, it requires the caller of the function to have detailed knowledge of the names and types of (some) local variables in the callee. What if the callee changed the name of the iteration variable from i to j (maybe i was being used for something else)? Now every caller that relied on it has to change. It ends up ridiculously tightly coupled (a bad thing) to no real benefit, since, if the function being called imposes the proper requirements on the callable passed, it can safely pass the argument by itself anyway (as your fixed code would demonstrate, if the lambda were properly defined as lambda x,i: x*i).
I'd suggest having the func only take i as a parameter, and skip the *args thing entirely in favor of just binding that information inside func:
from typing import Callable
def function_sum(lower_bound: int, upper_bound: int, func: Callable[[int], int]) -> int:
return sum(func(i) for i in range(lower_bound, upper_bound+1))
print(function_sum(1, 10, lambda i: 4*i))
In general, any value that a function needs to get from its caller should be passed into it as a parameter; this is the reason that parameters exist.
The x value is not being supplied by func's caller (function_sum), though, it's being supplied at the point where func is defined -- so that's the value that doesn't need to be a parameter, and can simply be defined as part of the function body (in this case 4*i, as opposed to making x a parameter, having the body be x*i, and then passing 4 into function_sum to in turn pass to func).
Python typing documentation includes examples for typing of generator functions. Yield is still a concept I struggle to understand, but I have a scenario where I'm not quite sure, how to properly use typing. The code is a very basic example of my current struggle and shows where my question arises from. If I have two yields in a function, how would I use typing for that function? The documentation on typing gives me no clear answer, my current preference would be, to use Iterator[dict].
def yield_func() -> ?:
A: dict = {}
B: dict = {}
yield A
yield B
I would currently use Iterator[dict] as typing annotations for the given function.
The most flexible form is the one explained here already and alluded to again by #tomerar. It was already mentioned in PEP 484 way back in the day. The generic alias Generator from collections.abc is what you can use.
Since your example generator is very simple in that it does not have a return value and does not utilize sending anything to it, you can use the more basic Iterator type, which is a supertype of Generator (see here). An Iterator is generic only in terms of the type its __next__ method returns, which is equivalent to the type you yield in a generator function.
I would suggest always using those generic alias types from collections.abc and not from typing (unless you are on Python <3.9) because the typing equivalents are deprecated (see for example here).
By the way, if you are already taking the time to annotate your functions (which is great), you should properly utilize generic types, which include dict. Here is how I would annotate your example function assuming the keys in your dictionaries are str and without knowing anything about the value-type:
from collections.abc import Iterator
MyDictT = dict[str, object] # for example
def yield_func() -> Iterator[MyDictT]:
a: MyDictT = {}
b: MyDictT = {}
...
yield a
yield b
I run into the following two issues at the same time fairly frequently
I have a function with an argument that's expected to be a container of strings
I would like to simplify calls by passing the function either a container of strings or a single string that is not a singleton list
I usually handle this with something like the following, which seems somehow not pythonic to me (I don't know why). Is there a more pythonic way to handle the situation? Is this actually a bad habit and it's most appropriate to require function calls like my_func(['just_deal_with_it'])?
Note the function iterable_not_string below is from sorin's answer to another question
from collections.abc import Iterable
def iterable_not_string(x):
is_iter = isinstance(x, Iterable)
is_not_str = (not isinstance(x, str))
return (is_iter and is_not_str)
def my_func(list_or_string):
if iterable_not_string(list_or_string):
do_stuff(list_or_string)
else:
do_stuff([list_or_string])
I use the following idiom, which works with any flexibly typed language:
def my_func(arg):
"""arg can be a list, or a single string"""
if isinstance(arg, str):
arg = [ arg ]
# the rest of the code can just treat `arg` as a list
do_stuff_with_a_list(arg)
By normalizing the argument to a list at the start, you avoid code duplication and type-checking later... and the attendant bugs if you forget a check.
Another options is to accept arbitrary arguments
def my_func(*strings):
do_stuff(strings)
my_func('string')
l = ['list', 'of', 'strings']
my_func(*l)
However, I advise to only do this, if the amount of elements is expected to be small, since the unpacking of the iterable may take some time and consume a lot of memory (i.e. on long generators).
You can do this with the python library typing
Typing provids type hinting for python
For example:
from typing import List, Union
def my_func(list_or_string: Union[List, str]):
...
Python 3.10 provides a cleaner approach to this:
from typing import List
def my_func(list_or_string: List | str):
...
In this case Union is replaced by the | (pipe)
I have a piece of code that generates a function from many smaller functions while making the outermost one accept an argument x.
In other words, I have an input x and I need to do various transformations to x that are decided at runtime.
This is done by iteratively calling this function (it essentially wraps a function in another function).
Here is the function:
def build_layer(curr_layer: typing.Callable, prev_layer: Union[typing.Callable, int]) -> typing.Callable:
def _function(x):
return curr_layer(prev_layer(x) if callable(prev_layer) else x)
return _function
Sidenote: as you can see if prev_layer is not callable it gets substituted for input x so I am using dummy integers to indicate where input goes.
The problem: this code cannot be pickled.
I do not seem to be able to figure out a way to rewrite this code in such a way to be pickleable.
Note: I need this object to be persisted on disk, but also its used in multiprocessing where its pickled for IPC (these functions are not used there, so technically they could be moved)
I have also a more complex version of this function that handles multiple inputs (using fixed aggregation function, in this case torch.cat) I know these two can be merged into one generic function and I will do that once I get it to work.
Here is the code for the second function:
def build_layer_multi_input(curr_layer: typing.Callable, prev_layers: list) -> typing.Callable:
def _function(x):
return curr_layer(torch.cat([layer(x) if callable(layer) else x for layer in prev_layers]))
return _function
I resolved this by attaching the return value of these functions to a class instance as described in this thread.
I'm running into this problem often: I'm creating a function that needs to perform a series of operations on a value, whether that value be a single value or a list of values.
Is there an elegant way to do this:
def convert_val(val):
do a series of things to each value, whether list or single val
return answer or list of answers
rather than what I've been doing?:
def convert_val(val):
if isinstance(val, list):
... do a series of things to each list item,
return a list of answers
else:
... do the same series, just on a single value
return a single answer
One solution would be to create a sub_convert() that would do the series of actions, and then just call it once or iteratively, depending on the type passed in to convert().
Another would be to create a single convert() that would accept the arguments (value, sub_convert()).
Other suggestions that would be more compact, elegant and preferably all in one function?
(I've done several searches here to see if my issue has already been addressed. My appologies if it has.)
Thanks,
JS
You need to fix your design to make all uses of the function actually correct.
Ralph Waldo Emerson. "A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines."
We're not talking about a foolish consistency. You have what might be a design problem based on inconsistent use of this function.
Option 1. Don't call convert_val( x ) where x is a non-list. Do this. convert_val( [x] ). Don't fix your function, fix all the places that use your function. Consistency helps reduce bugs.
Option 2. Change the design of convert_val to use multiple positional arguments. This doesn't generalize well.
def convert_val( *args ):
whatever it's supposed to do to the arguments.
Then fix all the places you provide a list to be convert_val( *someList ). That's okay, and may be closer to your intent.
Note.
You can find your design errors using the warnings module.
def convert_val( arg ):
if isinstance( arg, collections.Sequence ):
return convert_val_list( arg )
else:
warnings.warn( "Fix this" )
return convert_val_list( [arg] )[0]
def convert_val_list( arg ):
assert isinstance( arg, collections.Sequence )
the original processing
Once you've fixed all the design problems, you can then do this
convert_val = convert_val_list
And delete the original function.
If the function makes sense for a single value, as well as for a list, then logically the function's result for a certain list item will not depend on the other items in the list.
For example, a and b should end up identical:
items = [1, 2]
a = convert_val(items)
b = map(convert_val, items)
This example already hints at the solution: the caller knows whether a list or a single value is passed in. When passing a single value, the function can be used as-is. When passing a list, a map invocation is easily added, and makes it clearer what's happening on the side of the caller.
Hence, the function you describe should not exist in the first place!
I'm late to the party here and I'm not sure if this is what OP wants.
I much prefer to keep the implementation details hidden inside the function. The caller shouldn't care about what happens inside.
def convert_val(val):
values = []
values.extend(val)
for value in values:
# do things to each list item,
return a list of answers
This would make the convert_val put val into the values list (if not a list) or all values of val into the values list.
In addition should predictably get a list back (since you'd be using the same logic).
In the end:
assert convert_val([1]) == convert_val(1)