I am new in the developing community and I was wondering if there is any function or method you use to decide which algorithm has the best performance and therefore use it instead of any other.
For example:
I am using a decorator to know how long the functions are taking to solve problems, but I dont think that is extrapolable, hence, I was thinking maybe there is a general method or function you use to decide which algorithm to use.
Can you help me please?
Example I was using the time library to know how long two independent functions take to count the negative numbers in an array:
import time
def time_it(func):
def wrapper(*args,**kwargs):
start=time.time()
result=func(*args,**kwargs)
end=time.time()
print(func.__name__ +" took " +str((end-start)*1000) + " mil seconds")
return result
return wrapper
array=[
[-4, -3, -1, 1],
[-2, -2, 1, 2],
[-1, 1, 2, 3],
[1, 2, 4, 5]
]
#time_it
def count_negatives(array):
count=0
for i in array:
for j in i:
if j < 0:
count +=1
return count
#time_it
def count_neg(array):
count=0
row=0
column=0
while row<len(array) and column<len(array[0]):
if array[row][column]<0:
count +=1
column +=1
else:
row +=1
column=0
return count
print(count_negatives(array))
print(count_neg(array))
An algorithm runs according to some input given to it and as well the operations its doing on it (and other variables).
With enough sampling, you can plot graphs (I prefer the matplotlib library) and see which one handles the input you're giving it the best.
Keep in mind these will be only samples from your own "computer" - meaning it may run faster or slower for others.
Here we can use the time_it decorator you've written with a little change:
I would prefer using time.perf_counter() as it uses the fastest clock cycles thus can give more accurate results than just time.time()
The decorator will return the actual time it took in milliseconds.
I'll change some names so it will be easier to follow + remove the return value as we don't care about the answer of whether an array contains a negative number.
import time
def time_it(func):
def wrapper(*args,**kwargs):
start=time.perf_counter()
result=func(*args,**kwargs)
end=time.perf_counter()
return (end - start) * 1_000 # return is in milliseconds!
return wrapper
#time_it
def count_negatives_v1(array):
count = 0
for i in array:
for j in i:
if j < 0:
count += 1
#time_it
def count_negatives_v2(array):
count = 0
row = 0
column = 0
while row < len(array) and column < len(array[0]):
if array[row][column] < 0:
count += 1
column += 1
else:
row += 1
column = 0
We can now build some function that generates list of lists containing random integers between any range we choose! I've chosen that it could generate a list that contain 500-1000 lists, and these "inner" lists contain 50 numbers each can be between -1000 and 1000
def generate_arrays(inner_lists_amount=(500, 1000), numbers=(-1_000, 1_000), inner_lists_length=50):
inner_arrays_count = random.choice(range(*inner_lists_amount))
return [list(random.choices(range(*numbers), k=inner_lists_length)) for _ in range(inner_arrays_count)]
This will generate up inner_arrays_quantity inner arrays to inner arrays, each one containing 50 number between -1000 and 1000.
Then we will pass it to each of the function you've written: (e.g. v1, v2) and get the result, we will save the output as our "y" values on the graph, the "x" values will be the sample index, here I've chosen sample amount of 1000 meaning it will call 1000 times the generate_arrays, pass it to v1 and v2 and save the results for each of these methods in a different "y" value lists:
import matplotlib.pyplot as plt
def build_graphs(sample_count=100):
x = range(sample_count)
y_v1 = []
y_v2 = []
for _ in range(sample_count):
print(_)
arrays = generate_arrays()
y_v1.append(count_negatives_v1(arrays))
y_v2.append(count_negatives_v2(arrays))
plt.plot(x, y_v1, 'r')
plt.plot(x, y_v2, 'g')
plt.show()
Using the matplotlib module we coloured the second method (v2) with green and v1 in red.
This will give us results as following:
Now this is not 100% accurate and will never be as it depends on a lot of things such as:
PC memory
CPU clock rate sampling the time
and much more, but can be somewhat be improved if for each call of the generate_arrays we do X more tests and check the average time it takes on each specific array. Because here we tested only once how much time it takes for v1, v2 to run on each array... however because the sample amount is 1000 it gives fairly the same results as expected.
Note: this does not give the actual order of the functions (big-o) - if you want to do it, then you can give it increasing amount of data, plotting it into excel and use a trendline with the highest R value to find the best graph function that has the nearest to 100%.
More info using the openpyxl module
Related
I'm learning about algorithms, complexity, and data science. I wrote two functions that break apart a list of 1000 numbers into a list of each number in it's own list.
For example,
[1,2,3,4,5] would become [[1], [2], [3], [4], [5]]
It isn't practical, but the point is to see how the functions scale. 'divsep' separates the list by using recursion and 'ifsep' separates the list using iteration. The script I wrote generates lists of length 1 to 1000 and tests the amount of time it takes to sort those list sizes with each different function.
However, I'm getting consistent outliers at specific points in the data, particularly at list sizes of 400 and 800 and I'm not sure why. The code is below.
import time
def divsep(prime_list,fresh_list = []):
'''takes an input list of size x and breaks it down into x lists of size 1
using recursion'''
if len(prime_list) == 0:
return fresh_list
else:
new = prime_list.pop()
list_entry = [new]
fresh_list.append(list_entry)
return divsep(prime_list, fresh_list)
def ifsep(list_input):
'''takes an input list of size x and breaks it down into x lists of size 1
using iteration'''
new_list = []
for item in list_input:
ind = [item]
new_list.append(ind)
return new_list
with open('divsep.csv', 'w') as f:
f.write('"Algorithm","Range","Time"\n')
for number in range(1,1000):
list1 = [x for x in range(1,number)]
sum1 = 0
for numnum in range(100):
t1 = time.perf_counter()
divsep(list1)
t2 = time.perf_counter()
sum1 += t2-t1
f.write(f'"divsep","{number}","{sum1/100}"\n')
with open('ifsep.csv', 'w') as f:
f.write('"Algorithm","Range","Time"\n')
for number in range(1,1000):
list1 = [x for x in range(1,number)]
sum1 = 0
for number in range(100):
t3 = time.perf_counter()
ifsep(list1)
t4 = time.perf_counter()
sum1 += t4-t3
f.write(f'"ifsep","{number}","{sum1/100}"\n')
print('End program')
I use this code in Jupyter Notebook to display the results on line graphs.
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
recursion_separators = pd.io.parsers.read_csv('divsep.csv')
iteration_separators = pd.io.parsers.read_csv('ifsep.csv')
recursive_values = list(recursion_separators[['Time'][0]])
plt.plot(range(1,1000), recursive_values, 'm')
plt.show()
iteration_values = list(iteration_separators[['Time'][0]])
plt.plot(range(1,1000), iteration_values, 'b')
plt.show()
For both graphs, the x axis is the list size and the y axis is the time it takes to run the algorithm. The purple (top graph) are the results for the recursion separator, while the blue (bottom graph) displays the results for the iterative separator.
My assumption is that these values should scale consistently, so when I initially saw these outliers, I thought that maybe it extra time added because my computer is running other software, making the testing slower than it should be. To fix that, I added to the program and made it to where it runs each list of n size 100 times and then gets an average. I figured this would offset any outliers, but I does not. Besides, the ticks always appear in the same tick marks, particularly 400 and 800 for the top graph.
I'm not sure if it's a problem with my code, computer, or understanding of stats, so I hope that someone will be able to shed some light on the topic.
Thanks!
My versions are:
python: 3.7,
spyder: 3.3.6,
jupyter core: 4.5.0,
jupyter-notebook: 6.0.1,
I am trying to find the total possibilities of how to place 90 apples in 90 boxes. Any amount of apples can be placed in one box (0 to 90 apples), but all apples have to be placed into boxes. I used recursion but it took way too much time to complete the calculation. I was only able to test my code with small amounts of apples and boxes. Could anyone help me on reduce the time complexity of my code? Thanks in advance.
import math
boxes = 3
apples = 3
def possibilities(apples, boxes):
if apples == 0:
return 1
if boxes == 0:
return 0
start_point = 0 if boxes > 1 else math.floor(apples/boxes)
p = 0
for n in range(start_point, apples+1):
p += possibilities(apples-n, boxes-1)
return p
t = possibilities(apples,boxes)
print(t)
The way I see it, the problem consists in finding the number of sorted list of max 90 elements which have a sum equal to 90.
There is a concept which is quite close to this and we call it the partitions of a number.
For example, the partitions of 4 are [4], [3, 1], [2, 2], [2, 1, 1], [1, 1, 1, 1].
After a bit of research I found this article which is relevant to your problem.
As explained in there, the recursion method results in a very long calculation for large numbers, but...
A much more efficient approach is via an approach called dynamic programming. Here we compute a function psum(n,k), which is the total number of n-partitions with largest component of k or smaller. At any given stage we will have computed the values of psum(1,k), psum(2,k), psum(3,k), ..., psum(n,k) for some fixed k. Given this vector of n values we compute the values for k+1 as follows:
psum(i,k+1) = psum(i,k) + p(i,k) for any value i
But recall that p(i,k) = Σj p(i-k,j) = psum(i-k,k)
So psum(i,k+1) = psum(i,k) + psum(i-k,k)
So with a little care we can reuse the vector of values and compute the values of psum(i,k) in a rolling value for successively greater values of k. Finally, we have a vector whose values are psum(i,n). The value psum(n,n) is the desired value p(n). As an additional benefit we see that we have simultaneously computed the values of p(1), p(2), ..., p(n).
Basically, if you keep the intermediate values in a list and use the recurrence presented in the article,
psum(i,k+1) = psum(i,k) + psum(i-k,k)
you can use the following function:
def partitionp(n):
partpsum = [1] * (n + 1)
for i in range(2, n + 1):
for j in range(i, n + 1):
partpsum[j] += partpsum[j - i]
return partpsum[n]
At each iteration of the outer for loop, the list partpsum contains all the value psum(1,k), psum(2,k), psum(3,k), ..., psum(n,k). At the end of the iteration, you only need to return psum(n,n).
I am trying to create a loop in Python with numpy that will give me a variable "times" with 5 numbers generated randomly between 0 and 20. However, I want there to be one condition: that none of the differences between two adjacent elements in that list are less than 1. What is the best way to achieve this? I tried with the last two lines of code, but this is most likely wrong.
for j in range(1,6):
times = np.random.rand(1, 5) * 20
times.sort()
print times
da = np.diff(times)
if da.sum < 1: break
For instance, for one iteration, this would not be good:
4.25230915 4.36463992 10.35915732 12.39446368 18.46893283
But something like this would be perfect:
1.47166904 6.85610453 10.81431629 12.10176092 15.53569052
Since you are using numpy, you might as well use the built-in functions for uniform random numbers.
def uniform_min_range(a, b, n, min_dist):
while True:
x = np.random.uniform(a, b, size=n)
np.sort(x)
if np.all(np.diff(x) >= min_dist):
return x
It uses the same trial-and-error approach as the previous answer, so depending on the parameters the time to find a solution can be large.
Use a hit and miss approach to guarantee uniform distribution. Here is a straight-Python implementation which should be tweakable for numpy:
import random
def randSpacedPoints(n,a,b,minDist):
#draws n random numbers in [a,b]
# with property that their distance apart is >= minDist
#uses a hit-miss approach
while True:
nums = [a + (b-a)*random.random() for i in range(n)]
nums.sort()
if all(nums[i] + minDist < nums[i+1] for i in range(n-1)):
return nums
For example,
>>> randSpacedPoints(5,0,20,1)
[0.6681336968970486, 6.882374558960349, 9.73325447748434, 11.774594560239493, 16.009157676493903]
If there is no feasible solution this will hang in an infinite loop (so you might want to add a safety parameter which controls the number of trials).
I was attempting to solve a programing challenge and the program i wrote solved the small test data correctly for this question. But When they run it against the larger datasets, my program timed out on some of the occasions . I am mostly a self taught programmer, if there is a better algorithm/implementation than my logic can you guys tell me.thanks.
Question
Given an array of integers, a, return the maximum difference of any
pair of numbers such that the larger integer in the pair occurs at a
higher index (in the array) than the smaller integer. Return -1 if you
cannot find a pair that satisfies this condition.
My Python Function
def maxDifference( a):
diff=0
find=0
leng = len(a)
for x in range(0,leng-1):
for y in range(x+1,leng):
if(a[y]-a[x]>=diff):
diff=a[y]-a[x]
find=1
if find==1:
return diff
else:
return -1
Constraints:
1 <= N <= 1,000,000
-1,000,000 <= a[i] <= 1,000,000 i belongs to [1,N]
Sample Input:
Array { 2,3,10,2,4,8,1}
Sample Output:
8
Well... since you don't care for anything else than finding the highest number following the lowest number, provided that difference is the highest so far, there's no reason to do several passes or using max() over a slice of the array:
def f1(a):
smallest = a[0]
result = 0
for b in a:
if b < smallest:
smallest = b
if b - smallest > result:
result = b - smallest
return result if result > 0 else -1
Thanks #Matthew for the testing code :)
This is very fast even on large sets:
The maximum difference is 99613 99613 99613
Time taken by Sojan's method: 0.0480000972748
Time taken by #Matthews's method: 0.0130000114441
Time taken by #GCord's method: 0.000999927520752
The reason your program takes too long is that your nested loop inherently means quadratic time.
The outer loop goes through N-1 indices. The inner loop goes through a different number of indices each time, but the average is obviously (N-1)/2 rounded up. So, the total number of times through the inner loop is (N-1) * (N-1)/2, which is O(N^2). For the maximum N=1000000, that means 499999000001 iterations. That's going to take a long time.
The trick is to find a way to do this in linear time.
Here's one solution (as a vague description, rather than actual code, so someone can't just copy and paste it when they face the same test as you):
Make a list of the smallest value before each index. Each one is just min(smallest_values[-1], arr[i]), and obviously you can do this in N steps.
Make a list of the largest value after each index. The simplest way to do this is to reverse the list, do the exact same loop as above (but with max instead of min), then reverse again. (Reversing a list takes N steps, of course.)
Now, for each element in the list, instead of comparing to every other element, you just have to compare to smallest_values[i] and largest_values[i]. Since you're only doing 2 comparisons for each of the N values, this takes 2N time.
So, even being lazy and naive, that's a total of N + 3N + 2N steps, which is O(N). If N=1000000, that means 6000000 steps, which is a whole lot faster than 499999000001.
You can obviously see how to remove the two reverses, and how to skip the first and last comparisons. If you're smart, you can see how to take the whole largest_values out of the equation entirely. Ultimately, I think you can get it down to 2N - 3 steps, or 1999997. But that's all just a small constant improvement; nowhere near as important as fixing the basic algorithmic problem. You'd probably get a bigger improvement than 3x (maybe 20x), for less work, by just running the naive code in PyPy instead of CPython, or by converting to NumPy—but you're not going to get the 83333x improvement in any way other than changing the algorithm.
Here's a linear time solution. It keeps a track of the minimum value before each index of the list. These minimum values are stored in a list min_lst. Finally, the difference between corresponding elements of the original and the min list is calculated into another list of differences by zipping the two. The maximum value in this differences list should be the required answer.
def get_max_diff(lst):
min_lst = []
running_min = lst[0]
for item in lst:
if item < running_min:
running_min = item
min_lst.append(running_min)
val = max(x-y for (x, y) in zip(lst, min_lst))
if not val:
return -1
return val
>>> get_max_diff([5, 6, 2, 12, 8, 15])
13
>>> get_max_diff([2, 3, 10, 2, 4, 8, 1])
8
>>> get_max_diff([5, 4, 3, 2, 1])
-1
Well, I figure since someone in the same problem can copy your code and run with that, I won't lose any sleep over them copying some more optimized code:
import time
import random
def max_difference1(a):
# your function
def max_difference2(a):
diff = 0
for i in range(0, len(a)-1):
curr_diff = max(a[i+1:]) - a[i]
diff = max(curr_diff, diff)
return diff if diff != 0 else -1
my_randoms = random.sample(range(100000), 1000)
t01 = time.time()
max_dif1 = max_difference1(my_randoms)
dt1 = time.time() - t01
t02 = time.time()
max_dif2 = max_difference2(my_randoms)
dt2 = time.time() - t02
print("The maximum difference is", max_dif1)
print("Time taken by your method:", dt1)
print("Time taken by my method:", dt2)
print("My method is", dt1/dt2, "times faster.")
The maximum difference is 99895
Time taken by your method: 0.5533690452575684
Time taken by my method: 0.08005285263061523
My method is 6.912546237558299 times faster.
Similar to what #abarnert said (who always snipes me on these things I swear), you don't want to loop over the list twice. You can exploit the fact that you know that your larger value has to be in front of the smaller one. You also can exploit the fact that you don't care for anything except the largest number, that is, in the list [1,3,8,5,9], the maximum difference is 8 (9-1) and you don't care that 3, 8, and 5 are in there. Thus: max(a[i+1:]) - a[i] is the maximum difference for a given index.
Then you compare it with diff, and take the larger of the 2 with max, as calling default built-in python functions is somewhat faster than if curr_diff > diff: diff = curr_diff (or equivalent).
The return line is simply your (fixed) line in 1 line instead of 4
As you can see, in a sample of 1000, this method is ~6x faster (note: used python 3.4, but nothing here would break on python 2.x)
I think the expected answer for
1, 2, 4, 2, 3, 8, 5, 6, 10
will be 8 - 2 = 6 but instead Saksham Varma code will return 10 - 1 = 9.
Its max(arr) - min(arr).
Don't we have to reset the min value when there is a dip
. ie; 4 -> 2 will reset current_smallest = 2 and continue diff the calculation with value '2'.
def f2(a):
current_smallest = a[0]
large_diff = 0
for i in range(1, len(a)):
# Identify the dip
if a[i] < a[i-1]:
current_smallest = a[i]
if a[i] - current_smallest > large_diff:
large_diff = a[i] - current_smallest
I am profiling some genetic algorithm code with some nested loops and from what I see most of the time is spent in two of my functions which involve slicing and adding up numpy arrays. I tried my best to further optimize them but would like to see if others come up with ideas.
Function 1:
The first function is called 2954684 times for a total time spent inside the function of 19 seconds
We basically just create views inside numpy arrays contained in data[0], according to coordinates contained in data[1]
def get_signal(data, options):
#data[0] contains bed, data[1] contains position
#forward = 0, reverse = 1
start = data[1][0] - options.halfwinwidth
end = data[1][0] + options.halfwinwidth
if data[1][1] == 0:
normals_forward = data[0]['normals_forward'][start:end]
normals_reverse = data[0]['normals_reverse'][start:end]
else:
normals_forward = data[0]['normals_reverse'][end - 1:start - 1: -1]
normals_reverse = data[0]['normals_forward'][end - 1:start - 1: -1]
row = {'normals_forward': normals_forward,
'normals_reverse': normals_reverse,
}
return row
Function 2:
Called 857 times for a total time of 13.674 seconds spent inside the function:
signal is a list of numpy arrays of equal length with dtype float, options is just random options
The goal of the function is just to add up the lists of each numpy arrays to a single one, calculate the intersection of the two curves formed by the forward and reverse arrays and return the result
def calculate_signal(signal, options):
profile_normals_forward = np.zeros(options.halfwinwidth * 2, dtype='f')
profile_normals_reverse = np.zeros(options.halfwinwidth * 2, dtype='f')
#here i tried np.sum over axis = 0, its significantly slower than the for loop approach
for b in signal:
profile_normals_forward += b['normals_forward']
profile_normals_reverse += b['normals_reverse']
count = len(signal)
if options.normalize == 1:
#print "Normalizing to max counts"
profile_normals_forward /= max(profile_normals_forward)
profile_normals_reverse /= max(profile_normals_reverse)
elif options.normalize == 2:
#print "Normalizing to number of elements"
profile_normals_forward /= count
profile_normals_reverse /= count
intersection_signal = np.fmin(profile_normals_forward, profile_normals_reverse)
intersection = np.sum(intersection_signal)
results = {"intersection": intersection,
"profile_normals_forward": profile_normals_forward,
"profile_normals_reverse": profile_normals_reverse,
}
return results
As you can see the two are very simple, but account for > 60% of my execution time on a script that can run for hours / days (genetic algorithm optimization), so even minor improvements are welcome :)
One simple thing I would do to increase the speed of the first function is to use different notation for the accessing of the list indices as detailed here.
For example:
foo = numpyArray[1][0]
bar = numpyArray[1,0]
The second line will execute much faster because you don't have to return the entire element at numpyArray[1] and then find the first element of that. Try it out