Develop Reference

Process time, timeit()

I have several functions to create a list within a range. I'm using a time function I wrote, but it isn't timing the list functions I input. My list functions return the created list, currently. And the error is telling me, when I use time_it() that result cannot pass through.
# one of my list functions
def for_list(x):
x = range(x)
list_1 = []
for i in x:
i = str(i)
list_1 += i
return list_1
# timing function
def time_limit(tx):
start_time = process_time()
tx()
end_time = process_time()
time = (end_time - start_time)
print(f'{tx.__name__}, {time:.15f}')
SIZE = 10000
time_limit(for_list(SIZE))
Am I suppose to return something differently or is my time_limit() incorrect?

Inside the function time_limit() you are calling the for list twice.
It is called once when passed through and called again on the tx() line.
When removing that line it should look like this:
# one of my list functions
def for_list(x):
x = range(x)
list_1 = []
for i in x:
i = str(i)
list_1 += i
return list_1
# timing function
def time_limit(tx):
start_time = process_time()
end_time = process_time()
time = (end_time - start_time)
print(f'{tx.__name__}, {time:.15f}')
SIZE = 10000
time_limit(for_list(SIZE))

Can't use Python multiprocessing with large amount of calculations

I have to speed up my current code to do around 10^6 operations in a feasible time. Before I used multiprocessing in my the actual document I tried to do it in a mock case. Following is my attempt:
def chunkIt(seq, num):
avg = len(seq) / float(num)
out = []
last = 0.0
while last < len(seq):
out.append(seq[int(last):int(last + avg)])
last += avg
return out
def do_something(List):
# in real case this function takes about 0.5 seconds to finish for each
iteration
turn = []
for e in List:
turn.append((e[0]**2, e[1]**2,e[2]**2))
return turn
t1 = time.time()
List = []
#in the real case these 20's can go as high as 150
for i in range(1,20-2):
for k in range(i+1,20-1):
for j in range(k+1,20):
List.append((i,k,j))
t3 = time.time()
test = []
List = chunkIt(List,3)
if __name__ == '__main__':
with concurrent.futures.ProcessPoolExecutor() as executor:
results = executor.map(do_something,List)
for result in results:
test.append(result)
test= np.array(test)
t2 = time.time()
T = t2-t1
T2 = t3-t1
However, when I increase the size of my "List" my computer tires to use all of my RAM and CPU and freezes. I even cut my "List" into 3 pieces so it will only use 3 of my cores. However, nothing changed. Also, when I tried to use it on a smaller data set I noticed the code ran much slower than when it ran on a single core.
I am still very new to multiprocessing in Python, am I doing something wrong. I would appreciate it if you could help me.

To reduce memory usage, I suggest you use instead the multiprocessing module and specifically the imap method method (or imap_unordered method). Unlike the map method of either multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor, the iterable argument is processed lazily. What this means is that if you use a generator function or generator expression for the iterable argument, you do not need to create the complete list of arguments in memory; as a processor in the pool become free and ready to execute more tasks, the generator will be called upon to generate the next argument for the imap call.
By default a chunksize value of 1 is used, which can be inefficient for a large iterable size. When using map and the default value of None for the chunksize argument, the pool will look at the length of the iterable first converting it to a list if necessary and then compute what it deems to be an efficient chunksize based on that length and the size of the pool. When using imap or imap_unordered, converting the iterable to a list would defeat the whole purpose of using that method. But if you know what that size would be (more or less) if the iterable were converted to a list, then there is no reason not to apply the same chunksize calculation the map method would have, and that is what is done below.
The following benchmarks perform the same processing first as a single process and then using multiprocessing using imap where each invocation of do_something on my desktop takes approximately .5 seconds. do_something now has been modified to just process a single i, k, j tuple as there is no longer any need to break up anything into smaller lists:
from multiprocessing import Pool, cpu_count
import time
def half_second():
HALF_SECOND_ITERATIONS = 10_000_000
sum = 0
for _ in range(HALF_SECOND_ITERATIONS):
sum += 1
return sum
def do_something(tpl):
# in real case this function takes about 0.5 seconds to finish for each iteration
half_second() # on my desktop
return tpl[0]**2, tpl[1]**2, tpl[2]**2
"""
def generate_tpls():
for i in range(1, 20-2):
for k in range(i+1, 20-1):
for j in range(k+1, 20):
yield i, k, j
"""
# Use smaller number of tuples so we finish in a reasonable amount of time:
def generate_tpls():
# 64 tuples:
for i in range(1, 5):
for k in range(1, 5):
for j in range(1, 5):
yield i, k, j
def benchmark1():
""" single processing """
t = time.time()
for tpl in generate_tpls():
result = do_something(tpl)
print('benchmark1 time:', time.time() - t)
def compute_chunksize(iterable_size, pool_size):
""" This is more-or-less the function used by the Pool.map method """
chunksize, remainder = divmod(iterable_size, 4 * pool_size)
if remainder:
chunksize += 1
return chunksize
def benchmark2():
""" multiprocssing """
t = time.time()
pool_size = cpu_count() # 8 logical cores (4 physical cores)
N_TUPLES = 64 # number of tuples that will be generated
pool = Pool(pool_size)
chunksize = compute_chunksize(N_TUPLES, pool_size)
for result in pool.imap(do_something, generate_tpls(), chunksize=chunksize):
pass
print('benchmark2 time:', time.time() - t)
if __name__ == '__main__':
benchmark1()
benchmark2()
Prints:
benchmark1 time: 32.261038303375244
benchmark2 time: 8.174998044967651

The nested For loops creating the array before the main definition appears to be the problem. Moving that part to underneath the main definition clears up any memory problems.
def chunkIt(seq, num):
avg = len(seq) / float(num)
out = []
last = 0.0
while last < len(seq):
out.append(seq[int(last):int(last + avg)])
last += avg
return out
def do_something(List):
# in real case this function takes about 0.5 seconds to finish for each
iteration
turn = []
for e in List:
turn.append((e[0]**2, e[1]**2,e[2]**2))
return turn
if __name__ == '__main__':
t1 = time.time()
List = []
#in the real case these 20's can go as high as 150
for i in range(1,20-2):
for k in range(i+1,20-1):
for j in range(k+1,20):
List.append((i,k,j))
t3 = time.time()
test = []
List = chunkIt(List,3)
with concurrent.futures.ProcessPoolExecutor() as executor:
results = executor.map(do_something,List)
for result in results:
test.append(result)
test= np.array(test)
t2 = time.time()
T = t2-t1
T2 = t3-t1

implementing python multithreading into the calculation for a progressbar

i am trying to create a progressbar similar to tqdm's. Everything works just fine, but i noticed that the calculation for every step of the progressbar (for big iterables, len > 50) takes a lot of time. this is my code.
def progressbar(iterable):
def new(index):
#... print the progressbar
for i in range(len(iterable)):
new(i)
yield iterable[i]
the problem is that while on small iterables the time that new() takes to execute is indifferent, on larger iterables it becomes a problem (which does not occur in the tqdm library). For example the following code takes a few seconds to execute. It should be instant!
iterator = progressbar(range(1000))
for i in iterator: pass
can you tell me a way to remedy this thing? maybe implementing multithreading?

It's not clear what the issue is (you are not showing all of your calculations), but I believe your approach can be improved with the way your progress bar is handling the iterable it is being passed:
First, you are assuming that the iterable is indexable, which may not always be the case.
If it is a generator function, then the length may not be determinable with the len function nor would converting the generator to a list to get its length be necessarily efficient and it would probably defeat the purpose of having a progress bar, as in the example below. Your interface should therefore allow the user to pass an optional total parameter (as tqdm does) to explicitly specify the length of the iterable.
You can do some upfront calculations outside of function new so thatnew can quickly calculate based on the value of the index argument how wide the bar should be.
I would suggest the following changes:
def progressbar(iterable, total=None):
def new(index):
#... print the progressbar
from math import floor
nonlocal division, width
n_division = floor(index / division + .5)
remainder = width - n_division
print('|', '.' * n_division, ' ' * remainder, '|', sep='', end='\r')
if total is None:
iterable = list(iterable)
# we must convert to a list
total = len(iterable)
it = iter(iterable)
width = 60 # with of progress bar
division = total / 60 # each division represents this many completions
try:
for i in range(total):
# ensure next value exists before printing it:
yield next(it)
new(i)
except StopIteration:
pass
print()
def fun():
import time
for i in range(1000):
time.sleep(.03)
yield i
iterator = progressbar(fun(), total=1000)
values = [i for i in iterator]
print(values[0], values[-1])
Multithreading
Incorporating multithreading as a way of speeding up processing is problematic. The following is a (naive) attempt to do so that fails because although multithreading is being used to get the values from the generator function fun, the generator function is still generating values only once every .03 seconds. It's should also be clear that if the iterable is, for example, a simple list that multithreading is not going to be able to iterate the list more quickly than using a single thread:
from multiprocessing.pool import ThreadPool
def progressbar(iterable, total=None):
def new(index):
#... print the progressbar
from math import floor
nonlocal division, width
n_division = floor(index / division + .5)
remainder = width - n_division
print('|', '.' * n_division, ' ' * remainder, '|', sep='', end='\r')
if total is None:
iterable = list(iterable)
# we must convert to a list
total = len(iterable)
it = iter(iterable)
width = 60 # with of progress bar
division = total / 60 # each division represents this many completions
with ThreadPool(20) as pool:
for i, result in enumerate(pool.imap(lambda x: x, iterable)):
yield result
new(i)
print()
def fun():
import time
for i in range(1000):
time.sleep(.03)
yield i
iterator = progressbar(fun(), total=1000)
values = [i for i in iterator]
print(values[0], values[-1])
What would have sped up processing would have been if the generator function itself had used multithreading. But, of course, one has no control over how the iterable is being created:
from multiprocessing.pool import ThreadPool
def progressbar(iterable, total=None):
def new(index):
#... print the progressbar
from math import floor
nonlocal division, width
n_division = floor(index / division + .5)
remainder = width - n_division
print('|', '.' * n_division, ' ' * remainder, '|', sep='', end='\r')
if total is None:
iterable = list(iterable)
# we must convert to a list
total = len(iterable)
it = iter(iterable)
width = 60 # with of progress bar
division = total / 60 # each division represents this many completions
try:
for i in range(total):
# ensure next value exists before printing it:
yield next(it)
new(i)
except StopIteration:
pass
print()
def fun():
import time
def fun2(i):
time.sleep(.03)
return i
with ThreadPool(20) as pool:
for i in pool.imap(fun2, range(1000)):
yield i
iterator = progressbar(fun(), total=1000)
values = [i for i in iterator]
print(values[0], values[-1])

Optimizing a parallel implementation of a list comprehension

I have a dataframe, where each row contains a list of integers. I also have a reference-list that I use to check what integers in the dataframe appear in this list.
I have made two implementations of this, one single-threaded and one multi-threaded. The single-threaded implementation is quite fast (takes roughly 0.1s on my machine), whereas the multithreaded takes roughly 5s.
My question is: Is this due to my implementation being poor, or is this merely a case where the overhead due to multithreading is so large that it doesn't make sense to use multiple threads?
The example is below:
import time
from random import randint
import pandas as pd
import multiprocessing
from functools import partial
class A:
def __init__(self, N):
self.ls = [[randint(0, 99) for i in range(20)] for j in range(N)]
self.ls = pd.DataFrame({'col': self.ls})
self.lst_nums = [randint(0, 99) for i in range(999)]
#classmethod
def helper(cls, lst_nums, col):
return any([s in lst_nums for s in col])
def get_idx_method1(self):
method1 = self.ls['col'].apply(lambda nums: any(x in self.lst_nums for x in nums))
return method1
def get_idx_method2(self):
pool = multiprocessing.Pool(processes=1)
method2 = pool.map(partial(A.helper, self.lst_nums), self.ls['col'])
pool.close()
return method2
if __name__ == "__main__":
a = A(50000)
start = time.time()
m1 = a.get_idx_method1()
end = time.time()
print(end-start)
start = time.time()
m2 = a.get_idx_method2()
end = time.time()
print(end - start)

First of all, multiprocessing is useful when the cost of data communication between the main process and the others is less comparable to the time cost of the function.
Another thing is that you made an error in your code:
def helper(cls, lst_nums, col):
return any([s in lst_nums for s in col])
VS
any(x in self.lst_nums for x in nums)
You have that list [] in the helper method, which will make the any() method to wait for the entire array to be computed, while the second any() will just stop at the first True value.
In conclusion if you remove list brackets from the helper method and maybe increase the randint range for lst_nums initializer, you will notice an increase in speed when using multiple processes.
self.lst_nums = [randint(0, 10000) for i in range(999)]
and
def helper(cls, lst_nums, col):
return any(s in lst_nums for s in col)

How to use timeit where each test requires random setup

I have a function f(x) that takes as input a list x of 100 random floats between 0 and 1. Different lists will result in different running times of f.
I want to find out how long f takes to run on average, over a large number of different random lists. What's the best way to do this? Should I use timeit and if so is there a way I can do this without including the time it takes to generate each random list in each trial?
This is how I would do it without timeit (pseudocode):
for i = 1 to 10000:
x = random list
start = current time
f(x)
end = current time
results.append(end - start)
return mean(results)

You can make a timer decorator:
Here is some example code:
from time import time
class Timer(object):
def __init__(self, func):
"""
Decorator that times a function
#param func: Function being decorated
#type func: callable
"""
self.func = func
def __call__(self, *args, **kwargs):
start = time()
self.func(*args, **kwargs)
end = time()
return end - start
#Timer
def cheese():
for var in xrange(9999999):
continue
for var in xrange(100):
print cheese()
Working example, with fewer loops.

import timeit, random
def summer(myList):
result = 0
for num in myList:
result += num
return result
for i in range(10):
x = [random.randint(0, 100) for i in range(100000)]
print timeit.timeit("summer(x)", setup="from __main__ import x, summer", number = 100)
You can import the variable using from __main__ import x

I think this does the trick. It will execute the setup once per repeat and then execute stmt number=1 time. However I don't think this is that much better than the simple loop you posted.
import timeit
stmt = '[x*x*x for x in xrange(n)]' # just an example
setup = 'import random; n = random.randint(10, 100)'
r = 10000
times = timeit.repeat(stmt, setup, repeat=r, number=1)
print min(times), max(times), sum(times)/r
There is also a "cell mode" that you can use with timeit in the IPython shell, but it only returns the fasted time and there is no easy way to change that (?).
import random
%%timeit -r 10000 -n 1 n = random.randint(10,100)
var = [x*x*x for x in xrange(n)]