In a list comprehension with a condition that has a function call in it, does Python (specifically CPython 3.9.4) call the function each time, or does it calculate the value once and then uses it?
For example if you have:
list_1 = [1, 2, 3, 4, 5, 6, 7, 8, 9]
list_2 = [x for x in list_1 if x > np.average(list_1)]
Will Python actually calculate the np.average(list_1) len(list_1) times? So would it be more optimized to write
list_1 = [1, 2, 3, 4, 5, 6, 7, 8, 9]
np_avg = np.average(list_1)
list_2 = [x for x in list_1 if x > np_avg]
instead? Or does Python already "know" to just calculate the average beforehand?
Python has to call the function each time. It cannot optimize that part, because successive calls of the function might return different results (for example because of side effects). There is no easy way for Python’s compiler to be sure that this can’t happen.
Therefore, if you (the programmer) know that the result will always be the same – like in this case – it is probably advisable to calculate the result of the function in advance and use it inside the list comprehension.
Assuming standard CPython - Short answer: Yes. Your second snippet is more efficient.
A function call in the filter part of a list comprehension will be called for each element.
We can test this quite easily with a trivial example:
def f(value):
""" Allow even values only """
print('function called')
return value % 2 == 0
mylist = [x for x in range(5) if f(x)]
# 'function called' will be printed 5 times
The above is somewhat equivalent to doing:
mylist = []
for x in range(5):
if f(x):
mylist.append(x)
Since you're comparing against the same average each time, you can indeed just calculate it beforehand and use the same value as you did in your second code snippet.
It is easy to convert an entire iterator sequence into a list using list(iterator), but what is the best/fastest way to directly create a sublist from an iterator without first creating the entire list, i.e. how to best create list(iterator)[m:n] without first creating the entire list?
It seems obvious that it should not* (at least not always) be possible to do so directly for m > 0, but it should be for n less than the length of the sequence. [p for i,p in zip(range(n), iterator)] comes to mind, but is that the best way?
The context is simple: Creating the entire list would cause a RAM overflow, so it needs to be broken down. So how do you do this efficiently and/or python-ic-ly?
*The list comprehension I mentioned could obviously be used for m > 0 by calling next(iterator) m times prior to execution, but I don't enjoy the lack of python-ness here.
itertools.islice:
from itertools import islice
itr = (i for i in range(10))
m, n = 3, 8
result = list(islice(itr, m, n))
print(result)
# [3, 4, 5, 6, 7]
In addition, you can add an argument as the step if you wanted:
itr = (i for i in range(10))
m, n, step = 3, 8, 2
result = list(islice(itr, m, n, step))
print(result)
# [3, 5, 7]
list =[1,2,3,4,5]
This is my list. Basically what i want to perform is, difference between all individual list elements, like X=1-2-3-4-5. How can I do it?
The functional approach with a reduction:
from operator import sub
from functools import reduce
X = reduce(sub, [1, 2, 3, 4, 5])
Is basically equivalent to ((((1 - 2) - 3) - 4) - 5).
You can use reduce to perform what you are asking. It is built-in to python 2, but has been moved to the functools library for python 3.
from functools import reduce
a = [1,2,3,4,5]
reduce(lambda x,y: x-y, a)
# returns
-13
You can try this:
givenlist = [1,2,3,4,5]
difference = givenlist[0]
for i in range(1,len(list)):
difference = difference - givenlist[i]
print(difference)
It prints -13
So what I have is a list of integers such as [2, 2, 2, 3, ..., n] the length of the list can be anywhere from 1 to 100. What I need to do is calculate the powers of all the numbers. This should be super easy but there is also the stipulation that you must raise each number to the power of the next number to the power of the next number and so on. FOR EXAMPLE: if the list contained [2, 3, 4] first I would need to calculate the power of 3^4 and then 2^(answer of 3^4). If the list is longer it would need to calculate the value for all of them. The example above [2, 3, 4] should return 2^81 which should be something like 2417851639229258349412352 according to wolfram. Any help would be awesome even if its just an algorithm (I can figure out the code from there) I've just been struggling to come up with a sufficient algorithm for some time now.
here is some code I have now...
temp = []
length = 0
for num in powernumbers:
for index in num:
if index.isdigit():
temp.append(index)
length = len(temp)
if length > 0:
for j in reversed(range(len(temp))):
_temp = math.pow(int(temp[j-1]), int(temp[j]))
#THE ABOVE CODE WILL ONLY WORK FOR A LIST OF LEN 2
print(_temp)
#needs math.pow(0,(math.pow(1,(math.pow(2,...)))))
print("TEMP:", temp)
Once again any help is super appreciated!
You could use functools.reduce with reversed list:
>>> from functools import reduce
>>> l = [2, 3, 4]
>>> reduce(lambda x, y: y**x, reversed(l))
2417851639229258349412352
reduce takes two arguments: function and iterable. Then it will cumulatively apply the function to reduce the iterable to single value. First parameter of the function is reduced value and second parameter is item from iterable. Since we want to process the list in reverse order we're using reversed so that 3**4 will be executed first.
Note that on Python 2 reduce is a builtin so there's no need to import anything.
>>> numbers = [2,3,4] # your list
>>> result = 1
>>> for n in reversed(numbers):
result = n**result
>>> result
2417851639229258349412352
>>>
first initialize the result on 1, then go through the list in reverse order raising the number to the previous result, which the first time is 1 resulting for this example in
result = 4**1 -> 4
result = 3**4 -> 81
result = 2**81 -> 2417851639229258349412352
but be aware, this Nested exponentials will grow very very fast, and you more likely would get a memory error for the insanely big numbers
>>> result = 1
>>> powers = [2,2,2,2,2,2]
>>> for n in reversed(powers):
result = n**result
Traceback (most recent call last):
File "<pyshell#60>", line 2, in <module>
result = n**result
MemoryError
>>>
Pop the last element off of the list, then go through the list backwards and keep exponentiating.
powernumbers = [2, 3, 4]
result = powernumbers.pop()
for num in powernumbers[::-1]:
result = num**result
The result:
>>> result
2417851639229258349412352
Let's say we have a function add as follows
def add(x, y):
return x + y
we want to apply map function for an array
map(add, [1, 2, 3], 2)
The semantics are I want to add 2 to every element of the array. But the map function requires a list in the third argument as well.
Note: I am putting the add example for simplicity. My original function is much more complicated. And of course option of setting the default value of y in add function is out of question as it will be changed for every call.
One option is a list comprehension:
[add(x, 2) for x in [1, 2, 3]]
More options:
a = [1, 2, 3]
import functools
map(functools.partial(add, y=2), a)
import itertools
map(add, a, itertools.repeat(2, len(a)))
The docs explicitly suggest this is the main use for itertools.repeat:
Make an iterator that returns object over and over again. Runs indefinitely unless the times argument is specified. Used as argument to map() for invariant parameters to the called function. Also used with zip() to create an invariant part of a tuple record.
And there's no reason for pass len([1,2,3]) as the times argument; map stops as soon as the first iterable is consumed, so an infinite iterable is perfectly fine:
>>> from operator import add
>>> from itertools import repeat
>>> list(map(add, [1,2,3], repeat(4)))
[5, 6, 7]
In fact, this is equivalent to the example for repeat in the docs:
>>> list(map(pow, range(10), repeat(2)))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
This makes for a nice lazy-functional-language-y solution that's also perfectly readable in Python-iterator terms.
Use a list comprehension.
[x + 2 for x in [1, 2, 3]]
If you really, really, really want to use map, give it an anonymous function as the first argument:
map(lambda x: x + 2, [1,2,3])
Map can contain multiple arguments, the standard way is
map(add, a, b)
In your question, it should be
map(add, a, [2]*len(a))
The correct answer is simpler than you think.
Simply do:
map(add, [(x, 2) for x in [1,2,3]])
And change the implementation of add to take a tuple i.e
def add(t):
x, y = t
return x+y
This can handle any complicated use case where both add parameters are dynamic.
Sometimes I resolved similar situations (such as using pandas.apply method) using closures
In order to use them, you define a function which dynamically defines and returns a wrapper for your function, effectively making one of the parameters a constant.
Something like this:
def add(x, y):
return x + y
def add_constant(y):
def f(x):
return add(x, y)
return f
Then, add_constant(y) returns a function which can be used to add y to any given value:
>>> add_constant(2)(3)
5
Which allows you to use it in any situation where parameters are given one at a time:
>>> map(add_constant(2), [1,2,3])
[3, 4, 5]
edit
If you do not want to have to write the closure function somewhere else, you always have the possibility to build it on the fly using a lambda function:
>>> map(lambda x: add(x, 2), [1, 2, 3])
[3, 4, 5]
If you have it available, I would consider using numpy. It's very fast for these types of operations:
>>> import numpy
>>> numpy.array([1,2,3]) + 2
array([3, 4, 5])
This is assuming your real application is doing mathematical operations (that can be vectorized).
If you really really need to use map function (like my class assignment here...), you could use a wrapper function with 1 argument, passing the rest to the original one in its body; i.e. :
extraArguments = value
def myFunc(arg):
# call the target function
return Func(arg, extraArguments)
map(myFunc, itterable)
Dirty & ugly, still does the trick
I believe starmap is what you need:
from itertools import starmap
def test(x, y, z):
return x + y + z
list(starmap(test, [(1, 2, 3), (4, 5, 6)]))
def func(a, b, c, d):
return a + b * c % d
map(lambda x: func(*x), [[1,2,3,4], [5,6,7,8]])
By wrapping the function call with a lambda and using the star unpack, you can do map with arbitrary number of arguments.
You can include lambda along with map:
list(map(lambda a: a+2, [1, 2, 3]))
To pass multiple arguments to a map function.
def q(x,y):
return x*y
print map (q,range(0,10),range(10,20))
Here q is function with multiple argument that map() calls.
Make sure, the length of both the ranges i.e.
len (range(a,a')) and len (range(b,b')) are equal.
In :nums = [1, 2, 3]
In :map(add, nums, [2]*len(nums))
Out:[3, 4, 5]
Another option is:
results = []
for x in [1,2,3]:
z = add(x,2)
...
results += [f(z,x,y)]
This format is very useful when calling multiple functions.
#multi argument
def joke(r):
if len(r)==2:
x, y = r
return x + y
elif len(r)==3:
x,y,z=r
return x+y+z
#using map
print(list(map(joke,[[2,3],[3,4,5]])))
output = [6,12]
if the case like above and just want use function
def add(x,y):
ar =[]
for xx in x:
ar.append(xx+y)
return ar
print(list(map(add,[[3,2,4]],[2]))[0])
output = [5,4,6]
Note: you can modified as you want.