My code is something like this:
def fun -> list:
array = ... # creating of array
return array
def fun2:
array2 = []
# some code
for i in range(some_value):
array2.append(fun)
Note that it isn't known how many values function fun returns in each step of the algorithm so it is impossible to allocate array2 at the beginning. Is there any way how to fasten this performance?
If you take a look at the time complexity table of different python operations on lists, you can see that that the speed of appending k elements to a list is always the same regardless to the number of elements to append.
Related
I have a file with millions of lines, each of which is a list of integers (these sublists are in the range of tens to hundreds of items). What I want is to read through the file contents once and create 3 numpy arrays -- one with the average of each sublist, one with the length of each sublist, and one which is a flattened list of all the values in all the sublists.
If I just wanted one of these things, I'd do something like:
counts = np.fromiter((len(json.loads(line.rstrip())) for line in mystream), int)
but if I write 3 of those, my code would iterate through my millions of sublists 3 times, and I obviously only want to iterate through them once. So I want to do something like this:
averages = []
counts = []
allvals = []
for line in mystream:
sublist = json.loads(line.rstrip())
averages.append(np.average(sublist))
counts.append(len(sublist))
allvals.extend(sublist)
I believe that creating regular arrays as above and then doing
np_averages = np.array(averages)
Is very inefficient (basically creating the list twice). What is the right/efficient way to iteratively create a numpy array if it's not practical to use fromiter? Or do I want to create a function that returns the 3 values and do something like list comprehension for multiple return function? with fromiter instead of traditional list comprehension?
Or would it be efficient to create a 2D array of
[[count1, average1, sublist1], [count1, average2, sublist2], ...] and then doing additional operations to slice off (and in the 3rd case also flatten) the columns as their own 1D arrays?
First of all, the json library is not the most optimized library for that. You can use the pysimdjson package based on the optimized simdjson library to speed up the computation. For small integer lists, it is about twice faster on my machine.
Moreover, Numpy functions are not great for relatively small arrays as they introduce a pretty big overhead. For example, np.average takes about 8-10 us on my machine to compute an array of 20 items. Meanwhile, sum(sublist)/len(sublist) only takes 0.25-0.30 us.
Finally, np.array needs to iterate twice to convert the list into an array because it does not know the type of all objects. You can specify it so to make the convertion faster: np.array(averages, np.float64).
Here is a significantly faster implementation:
import simdjson
averages = []
counts = []
allvals = []
for line in mystream:
sublist = simdjson.loads(line.rstrip())
averages.append(sum(sublist) / len(sublist))
counts.append(len(sublist))
allvals.extend(sublist)
np_averages = np.array(averages, np.float64)
One issue with this implementation is that allvals will contain all the values in the form of a big list of objects. CPython objects are quite big in memory compared to native Numpy integers (especially compared to 32-bit=4bytes integers) since each object takes usually 32 bytes and the reference in the list takes usually 8 bytes (resulting in 40 bytes per items, that is to say 10 times more than Numpy 32-bit-integer-based arrays). Thus, I may be better to use a native implementation, possibly based on Cython.
I am very new to Python, and I am trying to get used to performing Python's array operations rather than looping through arrays. Below is an example of the kind of looping operation I am doing, but am unable to work out a suitable pure array operation that does not rely on loops:
import numpy as np
def f(arg1, arg2):
# an arbitrary function
def myFunction(a1DNumpyArray):
A = a1DNumpyArray
# Create a square array with each dimension the size of the argument array.
B = np.zeros((A.size, A.size))
# Function f is a function of two elements of the 1D array. For each
# element, i, I want to perform the function on it and every element
# before it, and store the result in the square array, multiplied by
# the difference between the ith and (i-1)th element.
for i in range(A.size):
B[i,:i] = f(A[i], A[:i])*(A[i]-A[i-1])
# Sum through j and return full sums as 1D array.
return np.sum(B, axis=0)
In short, I am integrating a function which takes two elements of the same array as arguments, returning an array of results of the integral.
Is there a more compact way to do this, without using loops?
The use of an arbitrary f function, and this [i, :i] business complicates by passing a loop.
Most of the fast compiled numpy operations work on the whole array, or whole rows and/or columns, and effectively do so in parallel. Loops that are inherently sequential (value from one loop depends on the previous) don't fit well. And different size lists or arrays in each loop are also a good indicator that 'vectorizing' will be difficult.
for i in range(A.size):
B[i,:i] = f(A[i], A[:i])*(A[i]-A[i-1])
With a sample A and known f (as simple as arg1*arg2), I'd generate a B array, and look for patterns that treat B as a whole. At first glance it looks like your B is a lower triangle. There are functions to help index those. But that final sum might change the picture.
Sometimes I tackle these problems with a bottom up approach, trying to remove inner loops first. But in this case, I think some sort of big-picture approach is needed.
I was going through an example in this computer-vision book and was a bit surprised by the code:
descr = []
descr.append(sift.read_features_from_file(featurefiles[0])[1])
descriptors = descr[0] #stack all features for k-means
for i in arange(1,nbr_images):
descr.append(sift.read_features_from_file(featurefiles[i])[1])
descriptors = vstack((descriptors,descr[i]))
To me it looks like this is copying the array over and over again and a more efficient implementation would be:
descr = []
descr.append(sift.read_features_from_file(featurefiles[0])[1])
for i in arange(1,nbr_images):
descr.append(sift.read_features_from_file(featurefiles[i])[1])
descriptors = vstack((descr))
Or am I missing something here and the two codes are not identical. I ran a small test:
print("ATTENTION")
print(descriptors.shape)
print("ATTENTION")
print(descriptors[1:10])
And it seems the list is different?
You're absolutely right - repeatedly concatenating numpy arrays inside a loop is extremely inefficient. Concatenation always generates a copy, which becomes more and more costly as your array gets bigger and bigger inside the loop.
Instead, do one of two things:
As you have done, store the intermediate values in a regular Python list and convert this to a numpy array outside the loop. Appending to a list is O(1), whereas concatenating np.ndarrays is O(n+k).
If you know how large the final array will be ahead of time, you can pre-allocate it and then fill in the rows inside your for loop, e.g.:
descr = np.empty((nbr_images, nbr_features), dtype=my_dtype)
for i in range(nbr_image):
descr[i] = sift.read_features_from_file(featurefiles[i])[1]
Another variant would be to use np.fromiter to lazily generate the array from an iterable object, for example in this recent question.
These codes gives the sum of even integers in a list without using loop statement. I would like to know the time complexity and space complexity of both codes. which is best?
CODE 1:
class EvenSum:
#Initialize the class
def __init__(self):
self.res = 0
def sumEvenIntegers(self, integerlist):
if integerlist:
if not integerlist[0] % 2:
self.res += integerlist[0]
del integerlist[0]
self.sumEvenIntegers(integerlist)
else:
del integerlist[0]
self.sumEvenIntegers(integerlist)
return self.res
#main method
if __name__ == "__main__":
l = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
even = EvenSum()
print even.sumEvenIntegers(l)
CODE 2:
import numpy as np
def sum_of_all_even_integers(list):
list_sum = sum(list)
bin_arr = map(lambda x:x%2, list)
return list_sum - sum(list*bin_arr)
if __name__ == "__main__":
list = np.array([1,2,3,4,5,6,7,8,9,10])
print sum_of_all_even_integers(list)
According to the Python wiki, deleting an item from a list takes linear time proportional to the number of elements in the list. Since you delete every item in the list, and each deletion takes linear time, the overall runtime is proportional to the square of number of items in the list.
In your second code snippet, both sum as well as map take linear time. So the overall complexity is linear proportional to the number of elements in the list. Interestingly, sum_of_elements isn't used at all (but it doesn't sum all even elements either).
what about the following?
import numpy as np
a = np.arange(20)
print np.sum(a[a%2==0])
It seems to be much more lightweight compared to your two code snippets.
Small timings with an np.arange(998):
Pure numpy:
248502
0.0
Class recursion:
248502
0.00399994850159
List/Numpy one:
248502
0.00200009346008
And, if there's a 999 element array, your class runs in failure, because the maximum recursion depth is reached.
First code use item deletion in list and recursivity, two thing at which python is not so good : time deletion take an O(n) time, since you recreate the whole list, and python does not optimize recursive calls (to keep full info about the traceback I think).
So I would go for the second code (which I think actually use "for loops", only the loops are hidden in the reduce and map).
If you use numpy, you could actually do something like :
a = np.array([1,2,3,4,5,6,7,8,9,10])
np.sum(np.where((a+1)%2,a,0))
Or like anki proposed :
np.sum( a[a%2 == 0] )
Which I think would be best since numpy is optimized for array manipulation.
By the way, never name an object list, as it overwrites the list constructor.
EDIT :
If you just want the sum of all even number in [0,n], you don't need a sum or anything. There is a mathematical formula for that :
s=(n//2)*(n//2+1)
First have O(N^2) time complexity and O(N) space complexity. The second have O(N) time complexity and space complexity.
The first uses one stack frame (one piece of stack memory of constant but quite large size) for each element in the array. In addition it executes the function for each element, but for each time it deletes the first element of the array, which is an O(N) operation.
The second happens much behind the scene. The map function generates a new list of the same size of the original, in addition it calls a function for each element - giving the complexity directly. Similarily for the reduce and sum functions - they do the same operation for each element in the list, although they don't use more memory than constant. Adding up these doesn't make the complexities worse - twice or thrice O(N) is still O(N).
The best is probably the later, but then again - it depends on what your preferences are. Maybe you want to consume a lot of time and stack space? In that case the first would suit your preferences much better.
Also note that the first solution modifies the input data - they don't do the same thing in other words. After calling the first the list sent to the function would be empty (which may or may not be a bad thing).
(I am quite a newbie in Python, so lots of things puzzle me even after reading the tutorial...)
Initially, I had the code like the following:
strings = ['>abc', 'qwertyu', '>def', 'zxcvbnm']
matrix = zip(*strings)
for member in matrix:
print("".join(member)) # characters are printed as expected
-- which did what I expected. But then for some reason I wanted to determine the number of members in matrix; as len(matrix) gave an error, I decided to copy it with converting to the list: mtxlist = list(matrix). Surprisingly, after this line the content of matrix seems to be changed - or at least I cannot use it the same way as above:
strings = ['>abc', 'qwertyu', '>def', 'zxcvbnm']
matrix = zip(*strings)
mtxlist = list(matrix) # this assignment empties (?) the matrix
for member in matrix:
print("".join(member)) # nothing printed
Can anybody explain what is going on there?
You're using Python 3, correct?
zip returns a generator that can only be iterated once. If you want to use it more than once, then your options are:
Write zip(*strings) each time you need it.
matrix = tuple(zip(*strings))
(iterate matrix as many times as you like. This is the easy option. The downside is that if zip(*strings) is big then it uses a lot of memory that the generator doesn't.)
matrix1, matrix2 = itertools.tee(zip(*strings))
(iterate each of matrix1 and matrix2 once. This is worse than the tuple in your usage, but it's useful if you want to partially consume matrix1, then use some of matrix2, more of matrix1, etc)
def matrix():
return zip(*strings)
# or
matrix = lambda: zip(*strings)
(iterate but using matrix(), not matrix, as many times as you like. Doesn't use extra memory for a copy of the result like the tuple solution, but the syntax for using it is a bit annoying)
class ReusableIterable:
def __init__(self, func):
self.func = func
def __iter__(self):
return iter(self.func())
matrix = ReusableIterable(lambda: zip(*strings))
(iterate using matrix as many times as you like. Deals with the syntax annoyance, although you still have to beware that if you modify strings between iterations over matrix then you'll get different results.)