I have a method that takes a list and returns an object:
# input a list, returns an object
def map_to_obj(lst):
a_list = f(lst)
return a_list[0] if a_list else None
I want to get a list that contains all the mapped elements that aren't None.
Like this:
v_list = [v1, v2, v3, v4]
[map_to_obj(v) for v in v_list if map_to_obj(v)]
But it doesn't seem good to call the map_to_obj method twice in the list comprehension.
Is there a way to have local variables in list comprehensions so that it can have better performance?
Or does the compiler optimize it automatically?
Here is what I want:
(sml like)
[let mapped = map_to_obj(v) in for v in v_list if mapped end]
Starting in Python 3.8, and the introduction of assignment expressions (PEP 572) (:= operator), it's possible to use a local variable within a list comprehension in order to avoid calling the same function twice.
In our case, we can name the evaluation of map_to_obj(v) as a variable o while using the result of the expression to filter the list; and thus use o as the mapped value:
[o for v in [v1, v2, v3, v4] if (o := map_to_obj(v))]
Use nested list comprehension:
[x for x in [map_to_obj(v) for v in v_list] if x]
or better still, a list comprehension around a generator expression:
[x for x in (map_to_obj(v) for v in v_list) if x]
A variable assignment is just a singular binding:
[x for v in l for x in [v]]
This is a more general answer and also closer to what you proposed.
So for your problem you can write:
[x for v in v_list for x in [map_to_obj(v)] if x]
You can avoid re-calculation by using python built-in filter:
list(filter(lambda t: t is not None, map(map_to_obj, v_list)))
A local variable can be set within a comprehension by cheating a bit and using an extra 'for' which "iterates" through a 1-element tuple containing the desired value for the local variable. Here's a solution to the OP's problem using this approach:
[o for v in v_list for o in (map_to_obj(v),) if o]
Here, o is the local variable being set equal to map_to_obj(v) for each v.
In my tests this is slightly faster than Lying Dog's nested generator expression (and also faster than the OP's double-call to map_to_obj(v), which, surprisingly, can be faster than the nested generator expression if the map_to_obj function isn't too slow).
List comprehensions are fine for the simple cases, but sometimes a plain old for loop is the simplest solution:
other_list = []
for v in v_list:
obj = map_to_obj(v)
if obj:
other_list.append(obj)
Now if you really want a list comp and dont want to build an tmp list, you can use the iterator versions of filter and map:
import itertools as it
result = list(it.ifilter(None, it.imap(map_to_obj, v_list)))
or more simply :
import itertools as it
result = filter(None, it.imap(map_to_obj, v_list)))
The iterator versions don't build a temporary list, they use lazy evaluation.
I have figured out a way of using reduce:
def map_and_append(lst, v):
mapped = map_to_obj(v)
if mapped is not None:
lst.append(mapped)
return lst
reduce(map_and_append, v_list, [])
How about the performance of this?
The best way I have found to do this, which is cryptic but succinct is like this:
[f(y) for x in some_list()
for y in [some_item(x)] # wastefully making a singleton list
if some_condition(y)]
Related
Why this problem has no trivial solution is because I needs to be solved using only pure functions.
Using only pure functions from Python's functional programming page (https://docs.python.org/3/howto/functional.html#), how can one create a list with a value in it? If we'd like to create a list with a value in it, we'd (in code) just do
x = [1]
I do not consider [] to be to be a part of the functions we're looking at here, since it has no signature and is not callable like any other function.
Using only functions to do this is not so trivial. One thought I had was to create a new list using list() and then append values to it. But list().append is mutable and does not return a new, or the, list with the item in it.
What I really want to do is to turn ["a","b","c"] into [["a"],["b"],["c"]], with above constraints.
Other proposals has been made like creating my own (pure) function doing what I want:
def create_list(value) -> list:
return [value]
and then just do map(create_list, ["a","b","c"]) to get solution.
But this is a custom made function and is not from any of the python package functions (within https://docs.python.org/3/howto/functional.html, as mentioned)
lst=[1,2,3];
#this will print [[1],[2],[3]]
print(list(map(lambda x: [x],lst)));
Single element:
def to_list(elem):
return list(range(elem, elem+1)))
To convert [1,2,3] into [[1], [2], [3]] with list comprehesion (it can be easily changed to map):
return [to_list(el) for el in input_list]
And without (ugly, but works ^^)
import itertools
def make_gen(elem):
yield elem
def to_list(elem):
return list(make_gen(elem))
def helper(elem, l):
return list(itertools.chain(to_list(to_list(elem)), l))
def convert(l):
if not l:
return []
return helper(l[0], convert(l[1:]))
print(convert([1, 2, 3]))
To ensure non-mutability, you probably want to use tuples instead of lists (or be very disciplined with your lists).
Using a list comprehension would be a valid functional approach:
A = [1,2,3]
B = [ [i] for i in A ] # [[1], [2], [3]]
or with tuples:
A = (1,2,3)
B = tuple( (i,) for i in A ) # ((1,), (2,), (3,))
If you must use functions, then map() is probably a good solution to this:
A = [1,2,3]
B = list(map(lambda i:[i],A))
If even [i] is proscribed (but why would it be), you can use a a function to make a list directly from its arguments:
def makeList(*v): return list(*v)
A = makeList(1,2,3)
B = makeList(*map(makeList,A))
# combined
makeList(*map(makeList,makeList(1,2,3)))
BTW functional programming is not about "only using functions", it is more about non-mutability of results (and avoidance of side effects). You may want to question whoever is sending you on this wild goose chase.
Using only pure functions from Python's functional programming page
(https://docs.python.org/3/howto/functional.html#), how can one create
a list with a value in it? If we'd like to create a list with number 1
in it
You might exploit generator as generator are described therein as follows
def justone():
yield 1
lst = list(justone())
print(lst)
output
[1]
justone is function (which might be checked using inspect.isfunction) and is pure (as it does not alter anything outside)
In the documentation you link, there are references to Iterators and Generators, which are powerful constructs present in Python (and other languages). You can consider a function to build a list as follows:
def list_from_args(*args):
return [*args]
This is a (superfluous) wrapper around Iterator functionality. You can leverage the Iterator pattern in Python to accomplish a lot, whether that be creating/consuming objects (e.g. lists, tuples, dictionaries), or for processing data (e.g. reading/writing to a file line-by-line, paginating an API or DB Query, etc.)
The code above does the following, for example:
>>> example = list_from_args(1, 'a', 'ham', 'eggs', 44)
>>> example
[1, 'a', 'ham', 'eggs', 44]
The reason I labeled the above function as superfluous: Oftentimes, if you need to create a list on the fly, you can use list comprehensions.
This does it using only functions from https://docs.python.org/3/library/functional.html
import functools
import itertools
map(
list,
map(
functools.partial(
itertools.repeat,
times=1,
),
[1,2,3]
)
)
functools.partial creates a new function of itertools.repeat with "times" parameter set to 1. Each value in the list is then repeated once and turned into a new list using list function.
>>> [[1], [2], [3]]
I'd have assumed the results of purge and purge2 would be the same in the following code (remove duplicate elements, keeping the first occurrences and their order):
def purge(a):
l = []
return (l := [x for x in a if x not in l])
def purge2(a):
d = {}
return list(d := {x: None for x in a if x not in d})
t = [2,5,3,7,2,6,2,5,2,1,7]
print(purge(t), purge2(t))
But it looks like with dict comprehensions, unlike with lists, the value of d is built incrementally. Is this what's actually happening? Do I correctly infer the semantics of dict comprehensions from this sample code and their difference from list comprehensions? Does it work only with comprehensions, or also with other right-hand sides referring to the dictionary being assigned to (e.g. comprehensions nested inside other expressions, something involving iterators, comprehensions of types other than dict)? Where is it specified and full semantics can be consulted? Or is it just an undocumented behaviour of the implementation, not to be relied upon?
There's nothing "incremental" going on here. The walrus operator doesn't assign to the variable until the dictionary comprehension completes. if x not in d is referring to the original empty dictionary, not the dictionary that you're building with the comprehension, just as the version with the list comprehension is referring to the original l.
The reason the duplicates are filtered out is simply because dictionary keys are always unique. Trying to create a duplicate key simply ignores the second one. It's the same as if you'd written:
return {2: None, 2: None}
you'll just get {2: None}.
So your function can be simplified to
def purge2(a):
return list({x: None for x in a})
I have a for loop in which I use the iterator always in the same manner, like this
for dict in mylist:
var = dict['prop']
var2 = f( dict['prop'] )
...
This is clumsy. The only alternatives I can think of:
local variable
wrapping the mylist in a list comprehension. But that seems overkill and is probably inefficient
Any better way?
One map call would work to give you a list of tuples of values:
listOfTuples = map(lambda x: (dict['prop'], f(dict['prop']), myList)
Or if you want two separate lists of values:
[varList, var2List] = zip(*zip(listOfTuples))
Assume you have a list such as
x = [('Edgar',), ('Robert',)]
What would be the most efficient way to get to just the strings 'Edgar' and 'Robert'?
Don't really want x[0][0], for example.
Easy solution, and the fastest in most cases.
[item[0] for item in x]
#or
[item for (item,) in x]
Alternatively if you need a functional interface to index access (but slightly slower):
from operator import itemgetter
zero_index = itemgetter(0)
print map(zero_index, x)
Finally, if your sequence is too small to fit in memory, you can do this iteratively. This is much slower on collections but uses only one item's worth of memory.
from itertools import chain
x = [('Edgar',), ('Robert',)]
# list is to materialize the entire sequence.
# Normally you would use this in a for loop with no `list()` call.
print list(chain.from_iterable(x))
But if all you are going to do is iterate anyway, you can also just use tuple unpacking:
for (item,) in x:
myfunc(item)
This is pretty straightforward with a list comprehension:
x = [('Edgar',), ('Robert',)]
y = [s for t in x for s in t]
This does the same thing as list(itertools.chain.from_iterable(x)) and is equivalent in behavior to the following code:
y = []
for t in x:
for s in t:
y.append(s)
I need to send this string to another function.
If your intention is just to call a function for each string in the list, then there's no need to build a new list, just do...
def my_function(s):
# do the thing with 's'
x = [('Edgar',), ('Robert',)]
for (item,) in x:
my_function(item)
...or if you're prepared to sacrifice readability for performance, I suspect it's quickest to do...
def my_function(t):
s = t[0]
# do the thing with 's'
return None
x = [('Edgar',), ('Robert',)]
filter(my_function, x)
Both map() and filter() will do the iteration in C, rather than Python bytecode, but map() will need to build a list of values the same length of the input list, whereas filter() will only build an empty list, as long as my_function() returns a 'falsish' value.
Here is one way:
>>> [name for name, in x]
['Edgar', 'Robert']
Note the placement of the comma, which unpacks the tuple.
>>> from operator import itemgetter
>>> y = map(itemgetter(0), x)
>>> y
['Edgar', 'Robert']
>>> y[0]
'Edgar'
>>> y[1]
'Robert'
I have 5 dictionaries and I want a union of their keys.
alldict = [dict1, dict2, dict3, dict4, dict5]
I tried
allkey = reduce(lambda x, y: set(x.keys()).union(y.keys()), alldict)
but it gave me an error
AttributeError: 'set' object has no attribute 'keys'
Am I doing it wrong ? I using normal forloop but I wonder why the above code didn't work.
I think #chuck already answered the question why it doesn't work, but a simpler way to do this would be to remember that the union method can take multiple arguments:
allkey = set().union(*alldict)
does what you want without any loops or lambdas.
Your solution works for the first two elements in the list, but then dict1 and dict2 got reduced into a set and that set is put into your lambda as the x. So now x does not have the method keys() anymore.
The solution is to make x be a set from the very beginning by initializing the reduction with an empty set (which happens to be the neutral element of the union).
Try it with an initializer:
allkey = reduce(lambda x, y: x.union(y.keys()), alldict, set())
An alternative without any lambdas would be:
allkey = reduce(set.union, map(set, map(dict.keys, alldict)))
A simple strategy for non-functional neurons (pun intended):
allkey = []
for dictio in alldict:
for key in dictio:
allkey.append(key)
allkey = set(allkey)
We can convert this code to a much sorter form using set comprehensions:
allkey = {key for dictio in alldict for key in dictio}
This one-liner is still very readable in comparison with the conventional for loop.
The key to convert a nested loop to a list or set comprehension is to write the inner loop (the one that varies faster in the nested loop) as the last index (that is, for key in dictio).
set().union(dict1.keys(),dict2.keys()...)
I tried the list and it didnt work so just putting it up here for anyone.
Just one more way, 'cause what the hay:
a={}; [ a.update(b) for b in alldict ] and a.keys()
or the slightly-more-mysterious
reduce(lambda a, b: a.update(b) or a, alldict, {}).keys()
(I'm bummed that there's no built-in function equivalent to
def f(a,b):
r = {}
r.update(a)
r.update(b)
return r
is there?)
If you only want to union keys of 2 dicts you could use operator |.
Quote from docs:
Return a new set with elements from the set and all others.
Example:
all_keys = (dict1.keys() | dict2.keys())