I am attempting to use timeit to keep track of how long a sorting algorithm takes to finish. However, it seems I can't even find an answer online how to exactly run timeit with functions that were originally written in another module. I've tried various setups and string inputs but am finding myself lost on this.
I tried using "t = timeit.Timer('sort.bubble(temp_array)')," but printing out the timer objects only gives me the memory addresses and cannot be converted to an integer...
In this case, I am calling bubble sort from another module.
#This section is on a timetests.py file
import random
import sort
import timeit
test_array1 = [random.randint(0, 500) for i in range(10)]
arrays_to_sort = [test_array1]
bubble_times = []
for a in range(len(arrays_to_sort)):
temp_array = arrays_to_sort[a]
t = timeit(sort.bubble(temp_array))
bubble_times.append(t)
**t = timeit(sort.bubble(temp_array))** #code is definitely not correct here
#This file is on sort.py
def bubble(list):
for current_pass in range(len(list) - 1, 0, -1):
for element in range(current_pass):
#Swap the element if the current one is smaller than
#next one
if list[element] > list[element + 1]:
temp = list[element]
list[element] = list[element + 1]
list[element + 1] = temp
return list
You need to function bubble and variable temp_array in the local environment.
Try:
t = timeit.timeit("bubble(temp_array)", setup = "from sort import bubble; from __main__ import temp_array"),
bubble_times.append(t)
Explanation:
So we use bubble() since that's the way to access an imported function
you would use sort.bubble() if you had imported the sort module rather than just a function from the module
We have to also bring in temp_array (assuming we are running timetests.py as the main module)
Using lambda function
Another option is use lambda to create a zero argument function which we pass to timeit. Note: checkout how to pass parameters of a function when using timeit.Timer()
t = timeit.timeit(lambda: sort.bubble(temp_array))
Related
I wrote some code (named exercise 2) where I define a function (named is_divisible) and it has worked perfectly.
Afterwards to learn how to import functions, I wrote the same code but without the defined function, and created a second module (named is_divisible). But whenever I import this module into the original "exercise 2" I get
No module named 'is_divisible'
I have checked that both python files are in the same folder, the name of the file is correct, and I know the code is well written because it has worked before and it is from a lecturer's of mine. I have also attempted to name the module and the function differently and to instead write:
from divis import is_divisible
but this was also unsuccessful.
Where am I going wrong? I will leave the code below:
import random
import math
import numpy as np
random_list=[]
for i in range (0,5):
r=random.randint(0,10)
random_list.append(r)
print(random_list) #five numbers from 0 to 10 are chosen and appended to a list
new_result=[print('right' for x in random_list if round(np.cosh(x)**2 - np.sinh(x)**2,2) == 1]
#checking the numbers following a maths rule
import is_divisible #trying to import the function is_divisible
divisor=3
idx = is_divisible(random_list, divisor)
for i in idx:
print(f'Value {random_list[i]} (at index {i}) is divisible by {divisor}')
the code for the function is_divisible is:
def is_divisible(x, n):
""" Find the indices of x where the element is exactly divisible by n.
Arguments:
x - list of numbers to test
n - single divisor
Returns a list of the indices of x for which the value of the element is
divisible by n (to a precision of 1e-6 in the case of floats).
Example:
>>> is_divisible([3, 1, 3.1415, 6, 7.5], 3)
[0, 3]
"""
r = []
small = 1e-6
for i, m in enumerate(x):
if m % n < small:
r.append(i)
return r
I know this question has been answered multiple times, but none of the answers seem to work for me or maybe I am not doing it correctly.
Generally, when you type import <Module> the module is the name of the file. So, if you had the function is_divisible inside a Python file named a.py, then to import it you will write from a import is_divisible. If instead, you would like to import the whole file, then you'd write import a.py, then to use the function you would use a.is_divisible(random_list, divisor).
You should also make sure that both files are in the same folder.
Not sure if it is relevant, but I use Spyder(version 3.3.3) as Python IDE and this python version is 3.7.3
The following is my code:
import os
import numpy as np
import multiprocessing as mp
from itertools import product
def func1(a_tup):
return (a_tup[0][0], *a_tup[1]), (a_tup[0][1]+a_tup[1][0])*a_tup[1][1]
if __name__ == '__main__':
lst1 = np.arange(4)
lst2 = np.arange(2, 8, 3)
lst3 = product(enumerate(lst2),
(item for item in product(lst1, lst1) if item[0] < item[1])
)
result1 = np.zeros((len(lst2), len(lst1) -1, len(lst1)), dtype = np.float32)
with mp.Pool(os.cpu_count() - 1) as pool:
mp_result1 = pool.imap(func1, lst3)#<--this line right here
for item in mp_result1:
print(item)
The problem is that as soon as I run this code, it will just sit there and never finish.
I have found that as soon as I change the line mp_result1 = pool.imap(func1, lst3) to mp_result1 = list(pool.imap(func1, lst3)), the coded will run perfectly. But the thing is, I am looking for a way to make the code work without converting mp_result1 to a list, because when the generator mp_result1 becomes big enough in terms of memory usage, converting it to a list object will instantly consume all of the available memory which makes the system freezes. So it is definitely not an ideal situation to be in.
So here is my question:
How to iterate over the result of pool.imap without converting it to a list object first?
Thank you in advance
I am quite new to python.I have been thinking of making the below code to parellel calls where a list of doj values are formatted with help of lambda,
m_df[['doj']] = m_df[['doj']].apply(lambda x: formatdoj(*x), axis=1)
def formatdoj(doj):
doj = str(doj).split(" ")[0]
doj = datetime.strptime(doj, '%Y' + "-" + '%m' + "-" + "%d")
return doj
Since the list has million records, the time it takes to format all takes a lot of time.
How to make parellel function call in python similar to Parellel.Foreach in c#?
I think that in your case using parallel computation is a bit of an overkill. The slowness comes from the code, not from using a single processor. I'll show you in some steps how to make it faster, guessing a bit that you're working with a Pandas dataframe and what your dataframe contains (please stick to SO guidelines and include a complete working example!!)
For my test, I've used the following random dataframe with 100k rows (scale times up to get to your case):
N=int(1e5)
m_df = pd.DataFrame([['{}-{}-{}'.format(y,m,d)]
for y,m,d in zip(np.random.randint(2007,2019,N),
np.random.randint(1,13,N),
np.random.randint(1,28,N))],
columns=['doj'])
Now this is your code:
tstart = time()
m_df[['doj']] = m_df[['doj']].apply(lambda x: formatdoj(*x), axis=1)
print("Done in {:.3f}s".format(time()-tstart))
On my machine it runs in around 5.1s. It has several problems. The first one is you're using dataframes instead of series, although you work only on one column, and creating a useless lambda function. Simply doing:
m_df['doj'].apply(formatdoj)
Cuts down the time to 1.6s. Also joining strings with '+' is slow in python, you can change your formatdoj to:
def faster_formatdoj(doj):
return datetime.strptime(doj.split()[0], '%Y-%m-%d')
m_df['doj'] = m_df['doj'].apply(faster_formatdoj)
This is not a great improvement but does cut down a bit to 1.5s. If you need to join the strings for real (because e.g. they are not fixed), rather use '-'.join('%Y','%m','%d'), that's faster.
But the true bottleneck comes from using datetime.strptime a lot of times. It is intrinsically a slow command - dates are a bulky thing. On the other hand, if you have millions of dates, and assuming they're not uniformly spread since the beginning of humankind, chances are they are massively duplicated. So the following is how you should truly do it:
tstart = time()
# Create a new column with only the first word
m_df['doj_split'] = m_df['doj'].apply(lambda x: x.split()[0])
converter = {
x: faster_formatdoj(x) for x in m_df['doj_split'].unique()
}
m_df['doj'] = m_df['doj_split'].apply(lambda x: converter[x])
# Drop the column we added
m_df.drop(['doj_split'], axis=1, inplace=True)
print("Done in {:.3f}s".format(time()-tstart))
This works in around 0.2/0.3s, more than 10 times faster than your original implementation.
After all this, if you still are running to slow, you can consider working in parallel (rather parallelizing separately the first "split" instruction and, maybe, the apply-lambda part, otherwise you'd be creating many different "converter" dictionaries nullifying the gain). But I'd take that as a last step rather than the first solution...
[EDIT]: Originally in the first step of the last code box I used m_df['doj_split'] = m_df['doj'].str.split().apply(lambda x: x[0]) which is functionally equivalent but a bit slower than m_df['doj_split'] = m_df['doj'].apply(lambda x: x.split()[0]). I'm not entirely sure why, probably because it's essentially applying two functions instead of one.
Your best bet is to use dask. Dask has a data_frame type which you can use to create this a similar dataframe, but, while executing compute function, you can specify number of cores with num_worker argument. this will parallelize the task
Since I'm not sure about your example, I will give you another one using the multiprocessing library:
# -*- coding: utf-8 -*-
import multiprocessing as mp
input_list = ["str1", "str2", "str3", "str4"]
def format_str(str_input):
str_output = str_input + "_test"
return str_output
if __name__ == '__main__':
with mp.Pool(processes = 2) as p:
result = p.map(format_str, input_list)
print (result)
Now, let's say you want to map a function with several arguments, you should then use starmap():
# -*- coding: utf-8 -*-
import multiprocessing as mp
input_list = ["str1", "str2", "str3", "str4"]
def format_str(str_input, i):
str_output = str_input + "_test" + str(i)
return str_output
if __name__ == '__main__':
with mp.Pool(processes = 2) as p:
result = p.starmap(format_str, [(input_list, i) for i in range(len(input_list))])
print (result)
Do not forget to place the Pool inside the if __name__ == '__main__': and that multiprocessing will not work inside an IDE such as spyder (or others), thus you'll need to run the script in the cmd.
To keep the results, you can either save them to a file, or keep the cmd open at the end with os.system("pause") (Windows) or an input() on Linux.
It's a fairly simple way to use multiprocessing with python.
I'm going to improve the performance of my code snippet which will frequently getting sub-array recursively.
So I used numpy.array instead of build-in list. Because, as I know, when fetching the sub-array, numpy.array don't copy the orginal list.
But when I changed to numpy.array, the performance got worse. So I want to know the reason. Thanks!
Following is my code snippet and the execution times by using the different objects I got:
import timeit
stat = '''
import numpy
def func(a):
a[len(a)-1] += 1
if len(a) == 1:
return a[0]
else:
return func(a[1:len(a)])
a1=[1,2,3,4,5,6,7,8,9,10]
a2=numpy.array([1,2,3,4,5,6,7,8,9,10])
'''
if __name__ == "__main__":
print "Execution time with build-in list: {0}".format(timeit.timeit('func(a1)', setup = stat, number = 1000))
print "Execution time with Numpy array: {0}".format(timeit.timeit('func(a2)', setup = stat, number = 1000))
And on my 64-bit mac(Python 2.7.6 + Numpy 1.8.0rc1) the output is:
Execution time with build-in list: 0.00507998466492
Execution time with Numpy array: 0.0195469856262
You will get the same execution times if you modify your two last lines of code as follows:
print "Execution time with build-in list: {0}".format(timeit.timeit(
'func(a1)', setup = stat, number = 1000), 'gc.enable()')
print "Execution time with Numpy array: {0}".format(timeit.timeit(
'func(a2)', setup = stat, number = 1000), 'gc.enable()')
where in both cases we allowed timeit to switch on so-called garbage collection, i.e. the process of freeing memory when it is not used anymore. The abovementioned modification returns, e.g.:
Execution time with build-in list: 0.00580596923828
Execution time with Numpy array: 0.00822710990906
to be of the same order of magnitude. According to the documentation of timeit "by default, it temporarily turns off garbage collection during the timing. The advantage of this approach is that it makes independent timings more comparable. This disadvantage is that garbage collection may be an important component of the performance of the function being measured."
There is a thin understanding what method, i.e. with or without garbage collection, should be used and when. Please also note, that you will obtain much longer times if you apply time.time() block from time module.
Thanks for all guys' answers and comments for this question, you have shown valuable info to me to make this answer.
The answer is this. The reason, which caused bad performance of numpy array in my quesion, is that accessing to individual items and assigning to built-in type on numpy array is slower than on built-in list. And, actually, the performance gain of fetching sub-array of numpy array than built-in list do exist. But the gain is too small in short array, e.g. array with len = 10 in my example, so the small gain was overtaken by the lost is this line: a[len(a)-1] += 1 in which we have accessed individual items and converting between built-in type int.
Following code proved the reason:
import numpy
from timeit import timeit
stat = '''
import numpy
a1 = range(4000)
a2 = numpy.array(a1)
i = 0
'''
if __name__ == "__main__":
test_times = 1000
print '1. {0:.8f}'.format(timeit('a1[i]', setup = stat, number = test_times))
print '2. {0:.8f}'.format(timeit('a2[i]', setup = stat, number = test_times))
print '3. {0:.8f}'.format(timeit('i += a1[i]; ++i', setup = stat, number = test_times))
print '4. {0:.8f}'.format(timeit('i += a2[i]; ++i', setup = stat, number = test_times))
print '5. {0:.8f}'.format(timeit('a = a1[i:len(a1)]; ++i', setup = stat, number = test_times))
print '6. {0:.8f}'.format(timeit('a = a2[i:len(a2)]; ++i', setup = stat, number = test_times))
The running result in my mac is following:
1. 0.00005913
2. 0.00017881
3. 0.00008607
4. 0.00084305
5. 0.01492000
6. 0.00053406
We can get these from the above result:
1 vs 2: Numpy array is slower when accessing individual item than built-in list.
2 vs 4: It seems need data conversion, which spend additional time, when assign data of item in numpy array to built-in type(int). For built-in list, the time spend for that is much little.
5 vs 6: Numpy truly saved much time when fetching sub-array comparing with built-in list.
If I want to use only the index within a loop, should I better use the range/xrange function in combination with len()
a = [1,2,3]
for i in xrange(len(a)):
print i
or enumerate? Even if I won't use p at all?
for i,p in enumerate(a):
print i
I would use enumerate as it's more generic - eg it will work on iterables and sequences, and the overhead for just returning a reference to an object isn't that big a deal - while xrange(len(something)) although (to me) more easily readable as your intent - will break on objects with no support for len...
Using xrange with len is quite a common use case, so yes, you can use it if you only need to access values by index.
But if you prefer to use enumerate for some reason, you can use underscore (_), it's just a frequently seen notation that show you won't use the variable in some meaningful way:
for i, _ in enumerate(a):
print i
There's also a pitfall that may happen using underscore (_). It's also common to name 'translating' functions as _ in i18n libraries and systems, so beware to use it with gettext or some other library of such kind (thnks to #lazyr).
That's a rare requirement – the only information used from the container is its length! In this case, I'd indeed make this fact explicit and use the first version.
xrange should be a little faster, but enumerate will mean you don't need to change it when you realise that you need p afterall
I ran a time test and found out range is about 2x faster than enumerate. (on python 3.6 for Win32)
best of 3, for len(a) = 1M
enumerate(a): 0.125s
range(len(a)): 0.058s
Hope it helps.
FYI: I initialy started this test to compare python vs vba's speed...and found out vba is actually 7x faster than range method...is it because of my poor python skills?
surely python can do better than vba somehow
script for enumerate
import time
a = [0]
a = a * 1000000
time.perf_counter()
for i,j in enumerate(a):
pass
print(time.perf_counter())
script for range
import time
a = [0]
a = a * 1000000
time.perf_counter()
for i in range(len(a)):
pass
print(time.perf_counter())
script for vba (0.008s)
Sub timetest_for()
Dim a(1000000) As Byte
Dim i As Long
tproc = Timer
For i = 1 To UBound(a)
Next i
Debug.Print Timer - tproc
End Sub
I wrote this because I wanted to test it.
So it depends if you need the values to work with.
Code:
testlist = []
for i in range(10000):
testlist.append(i)
def rangelist():
a = 0
for i in range(len(testlist)):
a += i
a = testlist[i] + 1 # Comment this line for example for testing
def enumlist():
b = 0
for i, x in enumerate(testlist):
b += i
b = x + 1 # Comment this line for example for testing
import timeit
t = timeit.Timer(lambda: rangelist())
print("range(len()):")
print(t.timeit(number=10000))
t = timeit.Timer(lambda: enumlist())
print("enum():")
print(t.timeit(number=10000))
Now you can run it and will get most likely the result, that enum() is faster.
When you comment the source at a = testlist[i] + 1 and b = x + 1 you will see range(len()) is faster.
For the code above I get:
range(len()):
18.766527627612255
enum():
15.353173553868345
Now when commenting as stated above I get:
range(len()):
8.231641875551514
enum():
9.974262515773656
Based on your sample code,
res = [[profiel.attr[i].x for i,p in enumerate(profiel.attr)] for profiel in prof_obj]
I would replace it with
res = [[p.x for p in profiel.attr] for profiel in prof_obj]
Just use range(). If you're going to use all the indexes anyway, xrange() provides no real benefit (unless len(a) is really large). And enumerate() creates a richer datastructure that you're going to throw away immediately.