Develop Reference

(Why) is there a performance benefit of using list.clear?

I've recently noticed that the built-in list has a list.clear() method. So far, when I wanted to ensure a list is empty, I just create a new list: l = [].
I was curious if it makes a difference, so I measured it:
$ python --version
Python 3.11.0
$ python -m timeit 'a = [1, 2, 3, 4]; a= []'
5000000 loops, best of 5: 61.5 nsec per loop
$ python -m timeit 'a = [1, 2, 3, 4]; a.clear()'
5000000 loops, best of 5: 57.4 nsec per loop
So creating a new empty list is about 7% slower than using clear() for small lists.
For bigger lists, it seems to be faster to just create a new list:
$ python -m timeit 'a = list(range(10_000)); a = []'
2000 loops, best of 5: 134 usec per loop
$ python -m timeit 'a = list(range(10_000)); a = []'
2000 loops, best of 5: 132 usec per loop
$ python -m timeit 'a = list(range(10_000)); a = []'
2000 loops, best of 5: 134 usec per loop
$ python -m timeit 'a = list(range(10_000)); a.clear()'
2000 loops, best of 5: 143 usec per loop
$ python -m timeit 'a = list(range(10_000)); a.clear()'
2000 loops, best of 5: 139 usec per loop
$ python -m timeit 'a = list(range(10_000)); a.clear()'
2000 loops, best of 5: 139 usec per loop
why is that the case?
edit: Small clarification: I am aware that both are (in some scenarios) not semantically the same. If you clear a list you could still have two pointers to it. When you create a new list object, there will not be a pointer to it. For this question, I don't care about this potential difference. Just assume there is exactly one reference to that list.

I was curious if it makes a difference, so I measured it:
This is not the right way to do the measurement. The setup code should be separated out, so that it is not included in the timing. In fact, all of these operations are O(1); you see O(N) results because creating the original data is O(N).
So far, when I wanted to ensure a list is empty, I just create a new list: l = [].
list.clear is not equivalent to creating a new list. Consider:
a = [1,2,3]
b = a
Doing a = [] will not cause b to change, because it only rebinds the a name. But doing a.clear() will cause b to change, because both names refer to the same list object, which was mutated.
(Using a = [] is fine when that result is intentional, of course. Similarly, user-defined objects are often best "reset" by re-creating them.)
Instead, a.clear() is equivalent to a[:] = [] (or the same with some other empty sequence). However, this is harder to read. The Zen of Python tells us: "Beautiful is better than ugly"; "Explicit is better than implicit"; "There should be one-- and preferably only one --obvious way to do it".
It's also clearly (sorry) faster:
$ python -m timeit --setup 'a = list(range(10_000))' 'a.clear()'
10000000 loops, best of 5: 34.9 nsec per loop
$ python -m timeit --setup 'a = list(range(10_000))' 'a[:] = []'
5000000 loops, best of 5: 67.1 nsec per loop
$ python -m timeit --setup 'a = list(range(10_000))' 'a[:] = ()'
5000000 loops, best of 5: 51.7 nsec per loop
The "clever" ways require creating temporary objects, both for the iterable and for the slice used to index the list. Either way, it will end up in C code that does relatively simple, O(1) pointer bookkeeping (and a memory deallocation).

Why is a set comprehension slower than set() with a list comprehension?

I've been doing some benchmarking of Python code and have noticed a strange thing:
$ python -m timeit '{x**2 for x in range(10_000)}'
100 loops, best of 3: 3.3 msec per loop
$ python -m timeit 'set([x**2 for x in range(10_000)])'
100 loops, best of 3: 3.21 msec per loop
Why is a set comprehension slower than using the set() function with a list comprehension, which creates an intermediate list? I'm using Python 3.6.6.

deque.popleft() vs list.pop(0), performance analysis

According to this question, I checked the performance on my laptop.
Surprisingly, I found that pop(0) from a list is faster than popleft() from a deque stucture:
python -m timeit 'l = range(10000)' 'l.pop(0)'
gives:
10000 loops, best of 3: 66 usec per loop
While:
python -m timeit 'import collections' 'l = collections.deque(range(10000))' 'l.popleft()'
gives:
10000 loops, best of 3: 123 usec per loop
Moreover, I checked the performance on jupyter finding the same outcome:
%timeit l = range(10000); l.pop(0)
10000 loops, best of 3: 64.7 µs per loop
from collections import deque
%timeit l = deque(range(10000)); l.popleft()
10000 loops, best of 3: 122 µs per loop
What is the reason?

The problem is that your timeit call also times the deque/list creation, and creating a deque is obviously much slower because of the chaining.
In the command line, you can pass the setup to timeit using the -s option like this:
python -m timeit -s"import collections, time; l = collections.deque(range(10000000))" "l.popleft()"
Also, since setup is only run once, you get a pop error (empty list) after a whule, since I haven't changed default number of iterations, so I created a large deque to make it up, and got
10000000 loops, best of 3: 0.0758 usec per loop
on the other hand with list it's slower:
python -m timeit -s "l = list(range(10000000))" "l.pop(0)"
100 loops, best of 3: 9.72 msec per loop
I have also coded the bench in a script (more convenient), with a setup (to avoid clocking the setup) and 99999 iterations on a 100000-size list:
import timeit
print(timeit.timeit(stmt='l.pop(0)',setup='l = list(range(100000))',number=99999))
print(timeit.timeit(setup='import collections; l = collections.deque(range(100000))', stmt='l.popleft()', number=99999))
no surprise: deque wins:
2.442976927292288 for pop in list
0.007311641921253109 for pop in deque
note that l.pop() for the list runs in 0.011536903686244897 seconds, which is very good when popping the last element, as expected.

Python3 vs Python2 list/generator range performance

I have this simple function that partitions a list and returns an index i in the list such that elements at indices less that i are smaller than list[i] and elements at indices greater than i are bigger.
def partition(arr):
first_high = 0
pivot = len(arr) - 1
for i in range(len(arr)):
if arr[i] < arr[pivot]:
arr[first_high], arr[i] = arr[i], arr[first_high]
first_high = first_high + 1
arr[first_high], arr[pivot] = arr[pivot], arr[first_high]
return first_high
if __name__ == "__main__":
arr = [1, 5, 4, 6, 0, 3]
pivot = partition(arr)
print(pivot)
The runtime is substantially bigger with python 3.4 that python 2.7.6
on OS X:
time python3 partition.py
real 0m0.040s
user 0m0.027s
sys 0m0.010s
time python partition.py
real 0m0.031s
user 0m0.018s
sys 0m0.011s
Same thing on ubuntu 14.04 / virtual box
python3:
real 0m0.049s
user 0m0.034s
sys 0m0.015s
python:
real 0m0.044s
user 0m0.022s
sys 0m0.018s
Is python3 inherently slower that python2.7 or is there any specific optimizations to the code do make run as fast as on python2.7

As mentioned in the comments, you should be benchmarking with timeit rather than with OS tools.
My guess is the range function is probably performing a little slower in Python 3. In Python 2 it simply returns a list, in Python 3 it returns a range which behave more or less like a generator. I did some benchmarking and this was the result, which may be a hint on what you're experiencing:
python -mtimeit "range(10)"
1000000 loops, best of 3: 0.474 usec per loop
python3 -mtimeit "range(10)"
1000000 loops, best of 3: 0.59 usec per loop
python -mtimeit "range(100)"
1000000 loops, best of 3: 1.1 usec per loop
python3 -mtimeit "range(100)"
1000000 loops, best of 3: 0.578 usec per loop
python -mtimeit "range(1000)"
100000 loops, best of 3: 11.6 usec per loop
python3 -mtimeit "range(1000)"
1000000 loops, best of 3: 0.66 usec per loop
As you can see, when input provided to range is small, it tends to be fast in Python 2. If the input grows, then Python 3's range behave better.
My suggestion: test the code for larger arrays, with a hundred or a thousand elements.
Actually, I went further and test a complete iteration through the elements. The results were totally in favor of Python 2:
python -mtimeit "for i in range(1000):pass"
10000 loops, best of 3: 31 usec per loop
python3 -mtimeit "for i in range(1000):pass"
10000 loops, best of 3: 45.3 usec per loop
python -mtimeit "for i in range(10000):pass"
1000 loops, best of 3: 330 usec per loop
python3 -mtimeit "for i in range(10000):pass"
1000 loops, best of 3: 480 usec per loop
My conclusion is that, is probably faster to iterate through a list than through a generator. Although the latter is definitely more efficient regarding memory consumption. This is a classic example of the trade off between speed and memory. Although the speed difference is not that big per se (less than miliseconds). So you should value this and what's better for your program.

Check for multidimensional list in Python

I have some data which is either 1 or 2 dimensional. I want to iterate through every pattern in the data set and perform foo() on it. If the data is 1D then add this value to a list, if it's 2D then take the mean of the inner list and append this value.
I saw this question, and decided to implement it checking for instance of a list. I can't use numpy for this application.
outputs = []
for row in data:
if isinstance(row, list):
vals = [foo(window) for window in row]
outputs.append(sum(vals)/float(len(vals)))
else:
outputs.append(foo(row))
Is there a neater way of doing this? On each run, every pattern will have the same dimensionality, so I could make a separate class for 1D/2D but that will add a lot of classes to my code. The datasets can get quite large so a quick solution is preferable.

Your code is already almost as neat and fast as it can be. The only slight improvement is replacing [foo(window) for window in row] with map(foo, row), which can be seen by the benchmarks:
> python -m timeit "foo = lambda x: x+1; list(map(foo, range(1000)))"
10000 loops, best of 3: 132 usec per loop
> python -m timeit "foo = lambda x: x+1; [foo(a) for a in range(1000)]"
10000 loops, best of 3: 140 usec per loop
isinstance() already seems faster than its counterparts hasattr() and type() ==:
> python -m timeit "[isinstance(i, int) for i in range(1000)]"
10000 loops, best of 3: 117 usec per loop
> python -m timeit "[hasattr(i, '__iter__') for i in range(1000)]"
1000 loops, best of 3: 470 usec per loop
> python -m timeit "[type(i) == int for i in range(1000)]"
10000 loops, best of 3: 130 usec per loop
However, if you count short as neat, you can also simplify your code (after replacingmap) to:
mean = lambda x: sum(x)/float(len(x)) #or `from statistics import mean` in python3.4
output = [foo(r) if isinstance(r, int) else mean(map(foo, r)) for r in data]