Related
To copy a nested list in an existing list, it is unfortunately not sufficient to simply multiply it, otherwise references are created and not independent lists in the list, see this example:
x = [[1, 2, 3]] * 2
x[0] is x[1] # will evaluate to True
To achieve your goal, you could use the range function in a list comprehension, for example, see this:
x = [[1, 2, 3] for _ in range(2)]
x[0] is x[1] # will evaluate to False (wanted behaviour)
This is a good way to multiply items in a list without just creating references, and this is also explained multiple times on many different websites.
However, there is a more efficient way to copy the list elements. That code seems a little faster to me (measured by timeit via command line and with different paramater n ∈ {1, 50, 100, 10000} for code below and range(n) in code above):
x = [[1, 2, 3] for _ in [0] * n]
But I wonder, why does this code run faster? Are there other disadvantages (more memory consumption or similar)?
python -m timeit '[[1, 2, 3] for _ in range(1)]'
1000000 loops, best of 3: 0.243 usec per loop
python -m timeit '[[1, 2, 3] for _ in range(50)]'
100000 loops, best of 3: 3.79 usec per loop
python -m timeit '[[1, 2, 3] for _ in range(100)]'
100000 loops, best of 3: 7.39 usec per loop
python -m timeit '[[1, 2, 3] for _ in range(10000)]'
1000 loops, best of 3: 940 usec per loop
python -m timeit '[[1, 2, 3] for _ in [0] * 1]'
1000000 loops, best of 3: 0.242 usec per loop
python -m timeit '[[1, 2, 3] for _ in [0] * 50]'
100000 loops, best of 3: 3.77 usec per loop
python -m timeit '[[1, 2, 3] for _ in [0] * 100]'
100000 loops, best of 3: 7.3 usec per loop
python -m timeit '[[1, 2, 3] for _ in [0] * 10000]'
1000 loops, best of 3: 927 usec per loop
# difference will be greater for larger n
python -m timeit '[[1, 2, 3] for _ in range(1000000)]'
10 loops, best of 3: 144 msec per loop
python -m timeit '[[1, 2, 3] for _ in [0] * 1000000]'
10 loops, best of 3: 126 msec per loop
This is correct; range, even in Python 3 where it produces a compact range object, is more complicated than a list, in the classical tradeoff between computation and storage.
As the list grows too large to fit in cache (the primary question if we're concerned with performance), the range object runs into a different issue: as each number in the range is created, it destroys and creates new int objects (the first 256 or so are less costly because they're interned, but their difference can still cost a few cache misses). The list would keep referring to the same one.
There are still more efficient options, though; a bytearray, for instance, would consume far less memory than the list. Probably the best function for the task is hidden away in itertools: repeat. Like a range object, it doesn't need storage for all the copies, but like the repeated list it doesn't need to create distinct objects. Something like for _ in repeat(None, x) would therefore just poke at the same few cache lines (iteration count and reference count for the object).
In the end, the main reason people stick to using range is because it's what's prominently presented (both in the idiom for a fixed count loop and among the builtins).
In other Python implementations, it's quite possible for range to be faster than repeat; this would be because the counter itself already holds the value. I'd expect such behaviours from Cython or PyPy.
Let's say I have something like this:
list(range(9:12))
Which gives me a list:
[9,10,11]
However I want it to be like:
[9,1,0,1,1]
Which splits every integer into single digits, is there anyway of achieving this without sacrificing too much performance? Or is there a way of generating list like these in the first place?
You can build the final result efficiently without having to build one large and/or small intermediate strings using itertools.chain.from_iterable.
In [18]: list(map(int, chain.from_iterable(map(str, range(9, 12)))))
Out[18]: [9, 1, 0, 1, 1]
In [12]: %%timeit
...: list(map(int, chain.from_iterable(map(str, range(9, 20)))))
...:
100000 loops, best of 3: 8.19 µs per loop
In [13]: %%timeit
...: [int(i) for i in ''.join(map(str, range(9, 20)))]
...:
100000 loops, best of 3: 9.15 µs per loop
In [14]: %%timeit
...: [int(x) for i in range(9, 20) for x in str(i)]
...:
100000 loops, best of 3: 9.92 µs per loop
Timings scale with input. The itertools version also uses memory efficiently although it is marginally slower than the str.join version if used with list(map(int, ...)):
In [15]: %%timeit
...: list(map(int, chain.from_iterable(map(str, range(9, 200)))))
...:
10000 loops, best of 3: 138 µs per loop
In [16]: %%timeit
...: [int(i) for i in ''.join(map(str, range(9, 200)))]
...:
10000 loops, best of 3: 159 µs per loop
In [17]: %%timeit
...: [int(x) for i in range(9, 200) for x in str(i)]
...:
10000 loops, best of 3: 182 µs per loop
In [18]: %%timeit
...: list(map(int, ''.join(map(str, range(9, 200)))))
...:
10000 loops, best of 3: 130 µs per loop
simplest way is,
>>> [int(i) for i in range(9,12) for i in str(i)]
[9, 1, 0, 1, 1]
>>>
Convert the integers to strings, then split() the string and reconvert the digits back to ints.
li = range(9,12)
digitlist = [int(d) for number in li for d in str(number)]
Output:
[9,1,0,1,1]
I've investigated how performant I can make this a little more. The first function I wrote was naive_single_digits, which uses the str approach, with a pretty efficient list comprehension.
def naive_single_digits(l):
return [int(c) for n in l
for c in str(n)]
As you can see, this approach works:
In [2]: naive_single_digits(range(9, 15))
Out[2]: [9, 1, 0, 1, 1, 1, 2, 1, 3, 1, 4]
However, I thought that it would surely be unecessary to always build a str object for each item in the list - all we actually need is a base conversion to digits. Out of laziness, I copied this function from here. I've optimised it a bit by specifying it to base 10.
def base10(n):
if n == 0:
return [0]
digits = []
while n:
digits.append(n % 10)
n //= 10
return digits[::-1]
Using this, I made
def arithmetic_single_digits(l):
return [i for n in l
for i in base10(n)]
which also behaves correctly:
In [3]: arithmetic_single_digits(range(9, 15))
Out[3]: [9, 1, 0, 1, 1, 1, 2, 1, 3, 1, 4]
Now to time it. I've also tested against one other answer (full disclosure: I modified it a bit to work in Python2, but that shouldn't have affected the performance much)
In [11]: %timeit -n 10 naive_single_digits(range(100000))
10 loops, best of 3: 173 ms per loop
In [10]: %timeit -n 10 list(map(int, itertools.chain(*map(str, range(100000)))))
10 loops, best of 3: 154 ms per loop
In [12]: %timeit arithmetic_single_digits(range(100000))
10 loops, best of 3: 93.3 ms per loop
As you can see arithmetic_single_digits is actually somewhat faster, although this is at the cost of more code and possibly less clarity. I've tested against ridiculously large inputs, so you can see a difference in performance - at any kind of reasonable scale, every answer here will be blazingly fast. Note that python's integer arithmetic is probably actually relatively slow, as it doesn't use a primitive integer type. If this were to be implemented in C, I'd suspect my approach to get a bit faster.
Comparing this to viblo's answer, using (pure) Python 3 (to my shame I haven't installed ipython for python 3):
print(timeit.timeit("digits(range(1, 100000))", number=10, globals=globals()))
print(timeit.timeit("arithmetic_single_digits(range(1, 100000))", number=10, globals=globals()))
This has the output of:
3.5284318959747907
0.806847038998967
My approach is quite a bit faster, presumably because I'm purely using integer arithmetic.
Another way to write an arithmetic solution. Compared to Izaak van Dongens solution this doesnt use a while loop but calculates upfront how many iterations it need in the list comprehension/loop.
import itertools, math
def digits(ns):
return list(itertools.chain.from_iterable(
[
[
(abs(n) - (abs(n) // 10 **x)*10**x ) // 10**(x-1)
for x
in range(1+math.floor(math.log10(abs(n) if n!=0 else 1)), 0, -1)]
for n in ns
]
))
digits([-11,-10,-9,0,9,10,11])
Turn it to a string then back into a list :)
lambda x: list(''.join(str(e) for e in x))
You can also do with map function
a=range(9,12)
res = []
b=[map(int, str(i)) for i in a]
for i in b:
res.extend(i)
print(res)
here is how I did it:
ls = range(9,12)
lsNew = []
length = len(ls)
for i in range(length):
item = ls[i]
string = str(item)
if len(string) > 1:
split = list(string)
lsNew = lsNew + split
else:
lsNew.append(item)
ls = lsNew
print(ls)
def breakall(L):
if L == []:
return []
elif L[0] < 10:
return [L[0]] + breakall(L[1:])
else:
return breakall([L[0]//10]) + [L[0] % 10] + breakall(L[1:])
print(breakall([9,10,12]))
-->
[9, 1, 0, 1, 2]
I'd like to take in a list of functions, funclist, and return a new function which takes in a list of arguments, arglist, and applies the ith function in funclist to the ith element of arglist, returning the results in a list:
def myfunc(funclist):
return lambda arglist: [ funclist[i](elt) for i, elt in enumerate(arglist) ]
This is not optimized for parallel/vectorized application of the independent functions in funclist to the independent arguments in argvec. Is there a built-in function in python or numpy (or otherwise) that will return a more optimized version of the lambda above? It would be similar in spirit to map or numpy.vectorize (but obviously not the same), and so far I haven't found anything.
In numpy terms true vectorization means performing the iterative stuff in compiled code. Usually that requires using numpy functions that work with whole arrays, doing thing like addition and indexing.
np.vectorize is a way of iterate of several arrays, and using their elements in a function that does not handle arrays. It doesn't do much in compiled code, so does not improve the speed much. It's most valuable as a way of applying numpy broadcasting rules to your own scalar function.
map is a variant on list comprehension, and has basically the same speed. And a list comprehension has more expressive power, working with several lists.
#Tore's zipped comprehension is a clear expression this task
[f(args) for f, args in zip(funclist, arglist)]
map can work with several input lists:
In [415]: arglist=[np.arange(3),np.arange(1,4)]
In [416]: fnlist=[np.sum, np.prod]
In [417]: [f(a) for f,a in zip(fnlist, arglist)]
Out[417]: [3, 6]
In [418]: list(map(lambda f,a: f(a), fnlist, arglist))
Out[418]: [3, 6]
Your version is a little wordier, but functionally the same.
In [423]: def myfunc(funclist):
...: return lambda arglist: [ funclist[i](elt) for i, elt in enumerate(arglist) ]
In [424]: myfunc(fnlist)
Out[424]: <function __main__.myfunc.<locals>.<lambda>>
In [425]: myfunc(fnlist)(arglist)
Out[425]: [3, 6]
It has the advantage of generating a function that can be applied to different arglists:
In [426]: flist=myfunc(fnlist)
In [427]: flist(arglist)
Out[427]: [3, 6]
In [428]: flist(arglist[::-1])
Out[428]: [6, 0]
I would have written myfunc more like:
def altfun(funclist):
def foo(arglist):
return [f(a) for f,a in zip(funclist, arglist)]
return foo
but the differences are just stylistic.
================
Time test for zip v enumerate:
In [154]: funclist=[sum]*N
In [155]: arglist=[list(range(N))]*N
In [156]: sum([funclist[i](args) for i,args in enumerate(arglist)])
Out[156]: 499500000
In [157]: sum([f(args) for f,args in zip(funclist, arglist)])
Out[157]: 499500000
In [158]: timeit [funclist[i](args) for i,args in enumerate(arglist)]
10 loops, best of 3: 43.5 ms per loop
In [159]: timeit [f(args) for f,args in zip(funclist, arglist)]
10 loops, best of 3: 43.1 ms per loop
Basically the same. But map is 2x faster
In [161]: timeit list(map(lambda f,a: f(a), funclist, arglist))
10 loops, best of 3: 23.1 ms per loop
Packaging the iteration in a callable is also faster
In [165]: timeit altfun(funclist)(arglist)
10 loops, best of 3: 23 ms per loop
In [179]: timeit myfunc(funclist)(arglist)
10 loops, best of 3: 22.6 ms per loop
Good evening!
I was running some tests on lists and list creation vs iterator creation and I came across some staggering time differences. Observe the following:
>>> timeit.timeit('map(lambda x: x**3, [1, 2, 3, 4, 5])')
0.4515998857965542
>>> timeit.timeit('list(map(lambda x: x**3, [1, 2, 3, 4, 5]))')
2.868906182460819
The iterator version returned by the first test runs more than 6x as fast as converting to a list. I understand basically why this might be occurring, but what I'm more interested in is a solution. Does anyone know of a data structure similar to a list that offers fast creation time? (Basically, I want to know if there is a way to go straight from iterator (i.e. map or filter function, etc.), to a list without any major performance hits)
Things I can sacrifice for speed:
Appending, inserting, popping and deleting elements.
Slicing of elements.
Reversing the list or any inplace operators like sort.
Contains (in) operator.
Concatenation and multiplication.
All suggestions are welcome thanks!
EDIT: Indeed this is for python 3.
In Python 3.x, map doesn't create a list, but just an iterator, unlike Python 2.x.
print(type(map(lambda x: x**3, [1, 2, 3, 4, 5])))
# <class 'map'>
To really get a list, iterate it with the list function, like this
print(type(list(map(lambda x: x**3, [1, 2, 3, 4, 5]))))
# <class 'list'>
So, you are really not comparing two similar things.
Expanding on thefourtheye's answer; The expressions inside the map function will not be evaluated before you iterate over it. This example should be pretty clear:
from time import sleep
def badass_heavy_function():
sleep(3600)
# Method call isn't evaluated
foo = map(lambda x: x(), [badass_heavy_function, badass_heavy_function])
# Methods call will be evaluated, please wait 2 hours
bar = list(map(lambda x: x(), [badass_heavy_function, badass_heavy_function]))
for _ in foo:
# Please wait one hour
pass
To further extend the answers of the other two guys:
You had a misconception about the iterator. But you refer to as "slow creation time", and then you look for a "faster container", because of your misinterpretation.
Note that the creation of a list object in python is fast:
%timeit list(range(10000))
10000 loops, best of 3: 164 µs per loop
What you experience as slow is the actual loop that you need to do calculate the values that need to go into the list.
see a very unoptimized example of slowly "creating" a new list of another list:
x = list(range(10000))
def slow_loop(x):
new = []
for i in x:
new.append(i**2)
return new
%timeit slow_loop(x)
100 loops, best of 3: 4.17 ms per loop
the time that is spent is actually on the loop, that is "slow" in python.
This is actually what you are doing here technically if you compare:
def your_loop(x):
return list(map(lambda y: y**2, x))
%timeit your_loop(x)
100 loops, best of 3: 4.5 ms per loop
There is a way to speed this up though:
def faster_loop(x):
return [i**2 for i in x]
%timeit faster_loop(x)
100 loops, best of 3: 3.67 ms per loop
although not by much given this kind of function. The thing is: the slow part here is the math, not the list and not the container. You can prove this by using numpy
arr = np.array(x)
%timeit arr ** 2
100000 loops, best of 3: 7.44 µs per loop
Woah... crazy speedup.
With the benchmarking - I find myself guilty of this quite often as well - people doubt the system too often but themselves not often enough. So it's not like python is very unoptimized or "slow" it's just that you're doing it wrong. Don't doubt the python list efficiency. Doubt your slow, inefficient code. You will probably get it right quicker...
It seems here the pure python ** operator is very slow, as a simple multiplication
is much quicker:
def faster_loop2(x):
return [i * i for i in x]
%timeit faster_loop2(x)
1000 loops, best of 3: 534 µs per loop
Hi I need to calculate distances between every numbers pair in list, including the distance between the last and the first (it's a circle).
Naively i can do something like that:
l = [10,-12,350]
ret = []
for i in range(len(l)-1):
ret.append(abs(l[i] - l[i+1]))
ret.append(l[-1] - l[0])
print ret
out: [22, 362, 340]
I tried "enumerate" which is a little bit better way:
print [abs(v - (l+[l[0]])[i+1]) for i, v in enumerate(l)]
out: [22, 362, 340]
Is there more elegant and "pythonic" way?
I'd argue this is a small improvement. There could well be a cleaner way than this though:
print [abs(v - l[(i+1)%len(l)]) for i, v in enumerate(l)]
Another method:
print map(lambda x,y: abs(x-y), l[1:] + l[:1], l)
Not a huge improvement:
>>> [abs(a - b) for a, b in zip(l, l[1:] + l[:-1])]
[22, 362, 340]
It's probably not as good as other answers in this case, but if used as part of a larger codebase, it could be useful to define an iterator which returns pairs of items over a list, e.g.:
def pairs(l):
if len(l) < 2:
return
for i in range(len(l)-1):
yield l[i], l[i+1]
yield l[-1], l[0]
print [abs(a - b) for a,b in pairs([10,-12,350])]
It's not a one-liner, but is fairly readable.
If you're happy to use numpy...
list(numpy.abs(numpy.ediff1d(l, to_end=l[0]-l[-1])))
This scales well with longer l. Not converting to or from a list will speed things up quite a bit (very often a numpy array can be used in place of a list anyway).
Or you can construct it youself using numpy.roll:
list(numpy.abs(l - numpy.roll(l, -1)))
A few timings:
In [37]: l = list(numpy.random.randn(1000))
In [38]: timeit [abs(v - l[(i+1)%len(l)]) for i, v in enumerate(l)]
1000 loops, best of 3: 936 us per loop
In [39]: timeit list(numpy.abs(numpy.ediff1d(l, to_end=l[0]-l[-1])))
1000 loops, best of 3: 367 us per loop
In [40]: _l = numpy.array(l)
In [41]: timeit numpy.abs(numpy.ediff1d(_l, to_end=l[0]-l[-1]))
10000 loops, best of 3: 48.9 us per loop
In [42]: timeit _l = numpy.array(l); list(numpy.abs(_l - numpy.roll(_l, -1)))
1000 loops, best of 3: 350 us per loop
In [43]: timeit numpy.abs(_l - numpy.roll(_l, -1))
10000 loops, best of 3: 32.2 us per loop
If raw speed is your thing, quicker still, but not so neat, you can use sliced arrays directly:
In [78]: timeit a = numpy.empty(_l.shape, _l.dtype); a[:-1] = _l[:-1] - _l[1:]; a[-1] = _l[-1] - _l[0]; a = numpy.abs(a)
10000 loops, best of 3: 20.5 us per loop
Combining the answer from icecrime with this answer provides another pythonic possibility:
print [numpy.linalg.norm(a-b) for a, b in zip(l, l[1:] + l[:-1])]