Sorry, kind of a hard question to title.
If I want to iterate over a potentially empty list, which is more efficient? I'm expecting the list to be empty the majority of the time.
for x in list:
dostuff()
OR
if len(list)>0:
for x in list:
dostuff()
Based on timings from the timeit module:
>>> from timeit import timeit
>>> timeit('for x in lst:pass', 'lst=[]')
0.08301091194152832
>>> timeit('if len(lst)>0:\n for x in lst:\n pass', 'lst=[]')
0.09223318099975586
It looks like just doing the for loop will be faster when the list is empty, making it faster option regardless of the state of the list.
However, there is a significantly faster option:
>>> timeit('if lst:\n for x in lst:\n pass', 'lst=[]')
0.03235578536987305
Using if lst is much faster than either checking the length of the list or always doing the for loop. However, all three methods are quite fast, so if you are trying to optimize your code I would suggest trying to find what the real bottleneck is - take a look at When is optimisation premature?.
You can just use if list:
In [15]: if l:
....: print "hello"
....:
In [16]: l1= [1]
In [17]: if l1:
....: print "hello from l1"
....:
hello from l1
In [21]: %timeit for x in l:pass
10000000 loops, best of 3: 54.4 ns per loop
In [22]: %timeit if l:pass
10000000 loops, best of 3: 22.4 ns per loop
If a list is empty if list will evaluate to False so no need to check the len(list).
Firstly if len(list) > 0: should be if list: to improve readability. I personally would have thought having the if statement is redundant but timeit seems to prove me wrong. It seems (unless I've made a silly mistake) that having the check for an empty list makes the code faster (for an empty list):
$ python -m timeit 'list = []' 'for x in list:' ' print x'
10000000 loops, best of 3: 0.157 usec per loop
$ python -m timeit 'list = []' 'if list:' ' for x in list:' ' print x'
10000000 loops, best of 3: 0.0766 usec per loop
First variant is more efficient, because Python's for loop checks length of list anyway, making additional explicit check just a waste of CPU cycles.
Related
What is the fastest way to check if a set contains at least one number within a given range?
For example setA = set(1,4,7,9,10), lowerRange=6, upperRange=8, will return True because of 7.
Currently I am using:
filtered = filter(lambda x: lowerRange<=x<=upperRange,setA)
Then if filtered is not empty, returns a True.
Assuming that setA can be a very large set, is this the optimal solution? Or is this iterating through the entire setA?
Since the membership chick is approximately O(1) for sets you can use a generator expression within any() built-in function:
rng = range(6, 9)
any(i in setA for i in rng)
Note that for a short range you'll give a better performance with set.intersection():
In [2]: a = {1,4,7,9,10}
In [3]: rng = range(6, 9)
In [8]: %timeit bool(a.intersection(rng))
1000000 loops, best of 3: 344 ns per loop
In [9]: %timeit any(i in a for i in rng)
1000000 loops, best of 3: 620 ns per loop
But in this case for longer ranges you'd definitely go with any():
In [10]: rng = range(6, 9000)
In [11]: %timeit any(i in a for i in rng)
1000000 loops, best of 3: 620 ns per loop
In [12]: %timeit bool(a.intersection(rng))
1000 loops, best of 3: 233 µs per loop
And note that the reason that any() performs better is because it returns True right after it encounter an item that exist in your set (based on our membership condition) an since the number 8 is at the beginning of the range it makes the any() per forms so fast. Also as mentioned in comment as a more pythonic way for checking the validity of the intersection of an iterable within a set you can use isdisjoint() method. Here is a benchmark with this method for small rage:
In [26]: %timeit not a.isdisjoint(rng)
1000000 loops, best of 3: 153 ns per loop
In [27]: %timeit any(i in a for i in rng)
1000000 loops, best of 3: 609 ns per loop
And here is a benchmark that makes the any() checks the membership for all the numbers which shows that isdisjoint() performs so better:
In [29]: rng = range(8, 1000)
In [30]: %timeit any(i in a for i in rng)
1000000 loops, best of 3: 595 ns per loop
In [31]: %timeit not a.isdisjoint(rng)
10000000 loops, best of 3: 142 ns per loop
The fastest way is to work with a sorted list or tuple instead of a set. That way you can do the range searches using the bisect module.
Unless you plan to use those values, using the filter function is unnecessary, because it stores data that you won't end up using. It also keeps going even after it finds one that fits the criteria, slowing you down quite a bit.
My solution would have been to write and use the following function.
def check(list, lower, upper):
for i in list:
if i >= lower and i <= upper:
return True
return False
Like with #Kasramvd's answer, and your idea, this is the brute-force search (O(n) solution). That's impossible to beat unless there are some constraints on the data beforehand, like that it has to be sorted.
I have to make a very large number of simulations on a R*C grid.
These simulations are altering the grid, so I need to copy my reference grid before each, and then apply my simulating function on the fresh new grid.
What is the fastest way to do this in Python?
Since I have not found a similar question on StackOverflow, I did the tests myself and decided to post them here thinking they could be useful to other people.
The answer will be a community response so that other people can add new measurements with possibly other techniques.
If you add another method, remember to measure all the old tests and update them because the time depends on the computer used, avoid biasing the results.
I used a bash variable for setting up the timeit tests:
setup="""
R = 100
C = 100
from copy import deepcopy
import numpy as np
ref = [[i for i in range(C)] for _ in range(R)]
ref_np = np.array(ref)
cp = [[100 for i in range(C)] for _ in range(R)]
cp_np = np.array(cp)
"""
Just for convenience, I also set a temporary alias pybench:
alias pybench='python3.5 -m timeit -s "$setup" $1'
Python 3
Python 3.5.0+ (default, Oct 11 2015, 09:05:38)
Deepcopy:
>>> pybench "cp = deepcopy(ref)"
100 loops, best of 3: 8.29 msec per loop
Modifying pre-created array using index:
>>> pybench \
"for y in range(R):
for x in range(C):
cp[y][x] = ref[y][x]"
1000 loops, best of 3: 1.16 msec per loop
Nested list comprehension:
>>> pybench "cp = [[x for x in row] for row in ref]"
1000 loops, best of 3: 390 usec per loop
Slicing:
>>> pybench "cp = [row[:] for row in ref]"
10000 loops, best of 3: 45.8 usec per loop
NumPy copy:
>>> pybench "cp_np = np.copy(ref_np)"
100000 loops, best of 3: 6.03 usec per loop
Copying to pre-created NumPy array:
>>> pybench "np.copyto(cp_np, ref_np)"
100000 loops, best of 3: 4.52 usec per loop
There is nothing very surprising in these results, as you might have guessed, use NumPy is enormously faster, especially if one avoids creating a new table each time.
To add to the answer from Delgan, numpy copy's documentation says to use numpy.ndarray.copy as the preferred method. So for now, without doing a timing test, I will use numpy.ndarray.copy
https://numpy.org/doc/stable/reference/generated/numpy.copy.html
https://numpy.org/doc/stable/reference/generated/numpy.ndarray.copy.html
I have some data which is either 1 or 2 dimensional. I want to iterate through every pattern in the data set and perform foo() on it. If the data is 1D then add this value to a list, if it's 2D then take the mean of the inner list and append this value.
I saw this question, and decided to implement it checking for instance of a list. I can't use numpy for this application.
outputs = []
for row in data:
if isinstance(row, list):
vals = [foo(window) for window in row]
outputs.append(sum(vals)/float(len(vals)))
else:
outputs.append(foo(row))
Is there a neater way of doing this? On each run, every pattern will have the same dimensionality, so I could make a separate class for 1D/2D but that will add a lot of classes to my code. The datasets can get quite large so a quick solution is preferable.
Your code is already almost as neat and fast as it can be. The only slight improvement is replacing [foo(window) for window in row] with map(foo, row), which can be seen by the benchmarks:
> python -m timeit "foo = lambda x: x+1; list(map(foo, range(1000)))"
10000 loops, best of 3: 132 usec per loop
> python -m timeit "foo = lambda x: x+1; [foo(a) for a in range(1000)]"
10000 loops, best of 3: 140 usec per loop
isinstance() already seems faster than its counterparts hasattr() and type() ==:
> python -m timeit "[isinstance(i, int) for i in range(1000)]"
10000 loops, best of 3: 117 usec per loop
> python -m timeit "[hasattr(i, '__iter__') for i in range(1000)]"
1000 loops, best of 3: 470 usec per loop
> python -m timeit "[type(i) == int for i in range(1000)]"
10000 loops, best of 3: 130 usec per loop
However, if you count short as neat, you can also simplify your code (after replacingmap) to:
mean = lambda x: sum(x)/float(len(x)) #or `from statistics import mean` in python3.4
output = [foo(r) if isinstance(r, int) else mean(map(foo, r)) for r in data]
I've been playing around with the map function in Python and I was looking for some help in understanding the following behaviour:
foo="12345"
print map(int,foo)
gives you [1, 2, 3, 4, 5]. Obviously int(foo) spits out 12345. So what exactly is happening? Since strings are iterable by character, would the above two lines be synonymous with
print [int(x) for x in foo]
I know they will output the same result but is there anything different going on behind the scenes? Is one more efficient or better than another? Is one more "pythonic"?
Thanks a lot!
map() may be somewhat faster than using list comprehension in some cases and in some cases map is slower than list comprehensions.
when using a built-in function:
python -mtimeit -s'xs=xrange(1000)' 'map(int,"1234567890")'
10000 loops, best of 3: 18.3 usec per loop
python -mtimeit -s'xs=xrange(1000)' '[int(x) for x in "1234567890"]'
100000 loops, best of 3: 20 usec per loop
with lambda,map() becomes slow:
python -mtimeit -s'xs=xrange(1000)' '[x*10 for x in "1234567890"]'
100000 loops, best of 3: 6.11 usec per loop
python -mtimeit -s'xs=xrange(1000)' 'map(lambda x:x*10,"1234567890")'
100000 loops, best of 3: 11.2 usec per loop
But, in python 3x map() returns a map object, i.e. an iterator
Apply function to every item of iterable and return a list of the
results.
From the documentation for map
int() attempts to convert what is passed into an integer and will raise a ValueError if you try something silly, like this:
>>> int('Hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: 'Hello'
map() will return a list, which has the return value of the function that you ask it to call for any iterable. If your function returns nothing, then you'll get a list of Nones, like this:
>>> def silly(x):
... pass
...
>>> map(silly,'Hello')
[None, None, None, None, None]
It is the short and efficient way to do something like this:
def verbose_map(some_function,something):
results = []
for i in something:
results.append(some_function(i))
return results
map can be thought of to work like this:
def map(func, iterable):
answer = []
for elem in iterable:
answer.append(func(elem))
return answer
Basically, it returns a list L such that the ith element of L is the result of computing func on the ith element of your iterable.
So, with int and a string of ints, in each iteration of the for loop, the element is a specific character, which when given to int comes back as an actual int. The result of calling map on such a string is a list, whose elements correspond to the inted values of the corresponding character in the string.
So yes, if L = "12345", then map(int, L) is synonymous with [int(x) for x in L]
Hope this helps
foo="12345"
In [507]: dis.dis('map(int,foo)')
0 <109> 28769
3 STORE_SLICE+0
4 LOAD_ATTR 29806 (29806)
7 <44>
8 BUILD_TUPLE 28527
11 STORE_SLICE+1
def map(func, iterable):
answer = []
for elem in iterable:
answer.append(func(elem))
return answer
dis.dis('map(int,foo)')
0 <109> 28769
3 STORE_SLICE+0
4 LOAD_ATTR 29806 (29806)
7 <44>
8 BUILD_TUPLE 28527
11 STORE_SLICE+1
dis.dis('[int(x) for x in foo]')
0 DELETE_NAME 28265 (28265)
3 LOAD_GLOBAL 30760 (30760)
6 STORE_SLICE+1
7 SLICE+2
8 BUILD_TUPLE 29295
11 SLICE+2
12 SETUP_LOOP 26912 (to 26927)
15 JUMP_FORWARD 26144 (to 26162)
18 JUMP_IF_FALSE 23919 (to 23940)
And timing:
In [512]: timeit map(int,foo)
100000 loops, best of 3: 6.89 us per loop
In [513]: def mymap(func, iterable):
...: answer = []
...: for elem in iterable:
...: answer.append(func(elem))
...: return answer
In [514]: timeit mymap(int,foo)
100000 loops, best of 3: 8.29 us per loop
In [515]: timeit [int(x) for x in foo]
100000 loops, best of 3: 7.5 us per loop
"More efficient" is a can of worms. On this computer, it's faster to use map with CPython, but the list comprehension is faster for pypy
$ python -mtimeit 'map(int,"1234567890")'
100000 loops, best of 3: 8.05 usec per loop
$ python -mtimeit '[int(x) for x in "1234567890"]'
100000 loops, best of 3: 9.33 usec per loop
$ pypy -mtimeit 'map(int,"1234567890")'
1000000 loops, best of 3: 1.18 usec per loop
$ pypy -mtimeit '[int(x) for x in "1234567890"]'
1000000 loops, best of 3: 0.938 usec per loop
Python3 shows map() to be faster even with the extra call to list() that is required
$ python3 -mtimeit 'list(map(int,"1234567890"))'
100000 loops, best of 3: 11.8 usec per loop
$ python3 -mtimeit '[int(x) for x in "1234567890"]'
100000 loops, best of 3: 13.6 usec per loop
Yes, there is a huge difference behind the scenes. If you print(map) you'll see it is a builtin. A builtin function executes faster than one written in python or than most that are based off of how the language is parsed, map uses the fast iter method, a list comprehension does not. Other that there is no difference.
map(int, '1'*1000000)
vs.
[int(i) for i in '1'*1000000]
Using CPython, and the unix time program, map completes in ~3 seconds, the list comprehension in ~5.
Oh, one thing to note, this only pertains when the function passed to map is written in C.
This is a problem I've come across a lot lately. Google doesn't seem to have an answer so I bring it to the good people of stack overflow.
I am looking for a simple way to populate a list with the output of a function. Something like this:
fill(random.random(), 3) #=> [0.04095623, 0.39761869, 0.46227642]
Here are other ways I've found to do this. But I'm not really happy with them, as they seem inefficient.
results = []
for x in xrange(3): results.append(random.random())
#results => [0.04095623, 0.39761869, 0.46227642]
and
map(lambda x: random.random(), [None] * 3)
#=> [0.04095623, 0.39761869, 0.46227642]
Suggestions?
Thanks for all the answers. I knew there was a more python-esque way.
And to the efficiency questions...
$ python --version
Python 2.7.1+
$ python -m timeit "import random" "map(lambda x: random.random(), [None] * 3)"
1000000 loops, best of 3: 1.65 usec per loop
$ python -m timeit "import random" "results = []" "for x in xrange(3): results.append(random.random())"
1000000 loops, best of 3: 1.41 usec per loop
$ python -m timeit "import random" "[random.random() for x in xrange(3)]"
1000000 loops, best of 3: 1.09 usec per loop
How about a list comprehension?
[random.random() for x in xrange(3)]
Also, in many cases, you need the values just once. In these cases, a generator expression which computes the values just-in-time and does not require a memory allocation is preferable:
results = (random.random() for x in xrange(3))
for r in results:
...
# results is "used up" now.
# We could have used results_list = list(results) to convert the generator
By the way, in Python 3.x, xrange has been replaced by range. In Python 2.x, range allocates the memory and calculates all values beforehand (like a list comprehension), whereas xrange calculates the values just-in-time and does not allocate memory (it's a generator).
why do you think they are inefficient?
There is another way to do it,a list-comprehension
listt= [random.random() for i in range(3)]
something more generic...
from random import random
fill = lambda func, num: [func() for x in xrange(num)]
# for generating tuples:
fill = lambda func, num: (func() for x in xrange(num))
# then just call:
fill(random, 4)
# or...
fill(lambda : 1+2*random(), 4)
list = [random.random() for i in xrange(3)]
list = [random.random() for i in [0]*3]
list = [i() for i in [random.random]*3]
Or :
fill =lambda f,n: [f() for i in xrange(n)]
fill(random.random , 3 ) #=> [0.04095623, 0.39761869, 0.46227642]
List comprehension is probably clearest, but for the itertools afficionado:
>>> list(itertools.islice(iter(random.random, None), 3))
[0.42565379345946064, 0.41754360645917354, 0.797286438646947]
A quick check with timeit shows that the itertools version is ever so slightly faster for more than 10 items, but still go with whatever seems clearest to you:
C:\Python32>python lib\timeit.py -s "import random, itertools" "list(itertools.islice(iter(random.random, None), 10))"
100000 loops, best of 3: 2.93 usec per loop
C:\Python32>python lib\timeit.py -s "import random, itertools" "[random.random() for _ in range(10)]"
100000 loops, best of 3: 3.19 usec per loop