Develop Reference

My multiprocessing threadpool takes longer to complete tasks than a single-threaded implementation

I have written an algorithim and am trying to compare performance of diffrent versions. My benchmark function uses a threadpool, but it takes the same time or longer to benchmark than a single-core implementation.
I have used pypy and python, versions 3.11 and the result is the same.
Method to benchmark:
def main(print_results=True):
results = Queue()
start_time = time.time()
words = get_set_from_dict_file("usa.txt")
results.put(f"Total words read: {len(words)}")
results.put(f"Total time taken to read the file: {round((time.time() - start_time) * 1000)} ms")
start_time_2 = time.time()
pairs = getPairs(words)
results.put(f"Number of words that can be built with 3 letter word + letter + 3 letter word: {len(pairs)}")
results.put(f"Total time taken to find the pairs: {round((time.time() - start_time_2) * 1000)} ms")
results.put(f"Time taken: {round((time.time() - start_time) * 1000)}ms")
if print_results:
[print(x) for x in results.queue]
return (time.time() - start_time) * 1000
MultiThreaded Threadpool:
def benchmark(n=1000):
# start number of threads equal to 90% of cores running main() using multiprocessing, continue until n runs complete
core_count = os.cpu_count()
thread_num = floor(core_count * 0.9)
pool = ThreadPool(thread_num)
results = pool.map_async(main, [False] * n)
results = results.get()
pool.close()
avg_time_ms = round(sum(results) / len(results))
# Save best run time and its code as a pickle file in format (time, code)
# Currently hidden code
return avg_time_ms, -1
Test:
if __name__ == "__main__":
print("Do you want to benchmark? (y/n)")
if input().upper() == "Y":
print("Benchmark n times: (int)")
n = input()
n = int(n) if (n.isdigit() and 0 < int(n) <= 1000) else 100
start = time.time()
bench = benchmark(n)
end = time.time()
print("\n----------Multi-Thread Benchmark----------")
print(f"Average time taken: {bench[0]} ms")
print(f"Best time taken yet: {bench[1]} ms")
print(f"Total bench time: {end - start:0.5} s")
start = time.time()
non_t_results = [main(False) for _ in range(n)]
end = time.time()
print("\n----------Single-Thread Benchmark----------")
print(f"Average time taken: {round(sum(non_t_results) / len(non_t_results))} ms")
print(f"Total bench time: {end - start:0.5} s")
else:
main()
Every time I run it, no matter the number of runs or threads in the pool, the pool never completes faster. Here is an example output:
Do you want to benchmark? (y/n)
y
Benchmark n times: (int)
50
----------Multi-Thread Benchmark----------
Average time taken: 276 ms
Best time taken yet: -1 ms
Total bench time: 2.2814 s
----------Single-Thread Benchmark----------
Average time taken: 36 ms
Total bench time: 1.91 s
Process finished with exit code 0
I expect the threadpool to finish faster.

It turns out I was using threads instead of processes. Thanks to the commentators I was able to understand that ThreadPool is for concurrent processing, and Pool is for parallel processing.
Here was the changed benchmark:
def benchmark(n=1000):
# start number of threads equal to 90% of cores running main() using multiprocessing, continue until n runs complete
core_count = os.cpu_count()
process_num = floor(core_count * 0.9)
with Pool(process_num) as pool:
results = pool.map_async(main, [False] * n)
results = results.get()
avg_time_ms = round(sum(results) / len(results))
# Save best run time and its code as a pickle file in format (time, code)
"""..."""
return avg_time_ms, -1

Perfomance of concatenating strings vs joining lists? [duplicate]

This question already has answers here:
What is the most efficient string concatenation method in Python?
(12 answers)
Closed 1 year ago.
I'm doing a small python exercises for fun.
I need to print something that looks like this based on some input, and some logic:
.P....
.I....
.D....
.Z....
BANANA
.M....
.A....
And right now I'm struggling a bit with constructing the strings with the dots in them.
So what I need is a function that takes a number n a number i, and a char, like this;
def buildstring(n,i, char):
And then returns a string of length n, consisiting only of dots where the i'th char is the char given.
I currently have this attempt:
def buildprint(n,i,char):
start = "".join(['.']*(i))
mid = char
end = "".join(['.']*(n - i-1))
print(start+mid+end)
buildprint(10,3,"j")
which produces:
...j......
Which is what I want. But my solution will be graded on time, but I'm not toosure that this would be the most effecient way of concatenating the strings, since I remember something about concatenating strings often being the slow part of a program or if this would be more perfomant:
def buildprint(n,i,char):
start = ['.']*(i)
mid = [char]
end = ['.']*(n - i-1)
print("".join(start+mid+end))
So the question is really about the way string concatenation works, vs joining lists

I would use a completly different approach using string interpolation and the
f-string
def buildprint(n,i,char):
start = "".join(['.']*(i))
mid = char
end = "".join(['.']*(n - i-1))
print(start+mid+end)
def build_f_string(total_char, dots, char):
total_char -= dots + 1
print(f'{"." * dots}{char}{"." * total_char}')
test_1 = (10, 3, "j")
buildprint(*test_1)
build_f_string(*test_1)
One Liner
build_f_string = lambda total_char, dots, char : print(f'{"." * dots}{char}{"." * (total_char - (dots + 1 ))}')
build_f_string(*test_1)
Performance
Make a timer function
from time import perf_counter
from functools import wraps
def timer(runs):
def _timer(f,):
#wraps(f)
def wrapper(*args, **kwargs):
start = perf_counter()
for test in range(runs):
res = f(*args, **kwargs)
end = perf_counter()
print(f'{f.__name__}: Total Time: {end - start}')
return res
return wrapper
return _timer
Decorate our functions
TEST_RUNS = 100_000
#timer(runs=TEST_RUNS)
def buildprint(n, i, char):
start = "".join(['.'] * (i))
mid = char
end = "".join(['.'] * (n - i - 1))
return start + mid + end
#timer(runs=TEST_RUNS)
def build_f_string(total_char, dots, char):
return f'{"." * dots}{char}{"." * (total_char - dots + 1)}'
Run Tests
test_1 = (10, 3, "j")
buildprint(*test_1)
build_f_string(*test_1)
# Output
# buildprint: Total Time: 0.06150109999999999
# build_f_string: Total Time: 0.027191400000000004
Going to next level
We can use Python Deque
Deques are a generalization of stacks and queues (the name is pronounced “deck” and is short for “double-ended queue”). Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction.
def build_withdeque(n, i, char):
hyper_list = deque([char]) # initialize the deque with the char, is in the middle already
hyper_list.extendleft(["." for _ in range(i)]) # Add left side
hyper_list.extend(["." for _ in range(n - i - 1)])
return "".join(hyper_list)
Running test
from collections import deque
TEST_RUNS = 10_00_000
#timer(runs=TEST_RUNS)
def build_withdeque(n, i, char):
hyper_list = deque([char]) # initialize the deque with the char, is in the middle already
hyper_list.extendleft(["." for _ in range(i)]) # Add left side
hyper_list.extend(["." for _ in range(n - i - 1)])
return "".join(hyper_list)
test_1 = (10, 3, "j")
buildprint(*test_1)
build_f_string(*test_1)
build_withdeque(*test_1)
# buildprint: Total Time: 0.546178
# build_f_string: Total Time: 0.29445559999999993
# build_withdeque: Total Time: 1.4074019
Still not so good..
Pre build a list
#timer(runs=TEST_RUNS)
def build_list_complete(n, i, char):
return ''.join(["." * i, char, "." * (n - i - 1)])
test_1 = (10, 3, "j")
buildprint(*test_1)
build_f_string(*test_1)
build_withdeque(*test_1)
build_list_complete(*test_1)
# buildprint: Total Time: 0.5440142
# build_f_string: Total Time: 0.3002815999999999
# build_withdeque: Total Time: 1.4215970999999998
# build_list_complete: Total Time: 0.30323829999999985
Final Result
# TEST_RUNS = 10_00_000
test_1 = (10, 3, "j")
buildprint(*test_1)
build_f_string(*test_1)
build_withdeque(*test_1)
build_list_complete(*test_1)
# buildprint: Total Time: 0.6512364
# build_f_string: Total Time: 0.2695955000000001
# build_withdeque: Total Time: 14.0086889
# build_list_complete: Total Time: 3.049139399999998
Wait no so fast
As #MisterMiyagi point out, using return "." * i + char + ("." * (n - i + 1)) could be equivalent than using return f'{"." * dots}{char}{"." * (total_char - dots + 1)}'.
However it is actually Faster.
Look at this:
def SuperHyperMegaUltraFastButBasicallyLikeFStringThatMisterMiyagiSayd(n, i, char):
return "." * i + char + ("." * (n - i + 1))
Run some test with 100_00_000 repetitions.
#timer(runs=TEST_RUNS)
def SuperHyperMegaUltraFastButBasicallyLikeFStringThatMisterMiyagiSaid(n, i, char):
return "." * i + char + ("." * (n - i + 1))
test_1 = (10, 3, "j")
buildprint(*test_1)
build_f_string(*test_1)
build_withdeque(*test_1)
build_list_complete(*test_1)
buildprint: Total Time: 5.3067210000000005
build_f_string: Total Time: 2.721678
build_withdeque: Total Time: 14.302031600000001
build_list_complete: Total Time: 3.0364287999999995 SuperHyperMegaUltraFastButBasicallyLikeFStringThatMisterMiyagiSayd: Total Time: 2.440598699999999
buildprint: Total Time: 5.3067210000000005
build_f_string: Total Time: 2.721678
build_withdeque: Total Time: 14.302031600000001
build_list_complete: Total Time: 3.0364287999999995
SuperHyperMegaUltraFastButBasicallyLikeFStringThatMisterMiyagiSayd
Total Time: 2.440598699999999
The f-string string concatenation is the clear winner here, untill proven otherwise

Create clock pulse with python

I want to work with exactly 20ms sleep time. When i was using time.sleep(0.02), i am facing many problems. It is not working what i want. If I had to give an example;
import time
i = 0
end = time.time() + 10
while time.time() < end:
i += 1
time.sleep(0.02)
print(i)
We wait to see "500" in console. But it is like "320". It is a huge difference. Because sleep time is not working true and small deviations occur every sleep time. It is increasing cumulatively and we are seeing wrong result.
And then, i want to create my new project for clock pulse. Is it that possible with time.time()?
import time
first_time = time.time() * 100 #convert seconds to 10 * miliseconds
first_time = int(first_time) #convert integer
first_if = first_time
second_if = first_time + 2 #for sleep 20ms
third_if = first_time + 4 #for sleep 40ms
fourth_if = first_time + 6 #for sleep 60ms
fifth_if = first_time + 8 #for sleep 80ms
end = time.time() + 8
i = 0
while time.time() < end:
now = time.time() * 100 #convert seconds to 10 * miliseconds
now = int(now) #convert integer
if i == 0 and (now == first_if or now > first_if):
print('1_' + str(now))
i = 1
if i == 1 and (now == second_if or now > second_if):
print('2_' + str(now))
i = 2
if i == 2 and (now == third_if or now > third_if):
print('3_' + str(now))
i = 3
if i == 3 and (now == fourth_if or now > fourth_if):
print('4_' + str(now))
i = 4
if i == 4 and (now == fifth_if or now > fifth_if):
print('5_' + str(now))
break
Out >> 1_163255259009
2_163255259011
3_163255259013
4_163255259015
5_163255259017
Is this project true logic? And If it is true logic, how can finish this projects with true loops?
Because i want these sleeps to happen all the time. Thank you in advice.

Let's say you want to count in increments of 20ms. You need to sleep for the portion of the loop that's not the comparison, increment, and print. Those operations take time, probably about 10ms based on your findings.
If you want to do it in a loop, you can't hard code all the possible end times. You need to do something more general, like taking a remainder.
Start with the time before the loop:
t0 = time.time()
while time.time() < end:
i += 1
Now you need to figure out how long to sleep so that the time between t0 and the end of the sleep is a multiple of 20ms.
(time.time() - t0) % 0.02 tells you how far past a 20ms increment you are because python conveniently supports floating point modulo. The amount of time to wait is then
time.sleep(0.02 - (time.time() - t0) % 0.02)
print(i)
Using sign rules of %, you can reduce the calculation to
time.sleep(-(time.time() - t0) % 0.02)

Time measurement - function with an input inside

My task is to measure time of a function which contains an input inside (the user is to input a list).
There is the code:
import numpy
import itertools
def amount(C):
N = numpy.array(input().strip().split(" "),int)
N = list(N)
N = sorted(N)
while C < max(N):
N.remove(max(N))
res = []
for i in range(1, C):
for j in list(itertools.combinations_with_replacement(N, i)):
res.append(sum(list(j)))
m = 0
for z in range (0, len(res)):
if res[z] == C:
m += 1
if N[0] == 1:
return m + 1
else:
return m
Of course it is not optimized (but it's not the case now).
Because of the "list" standard way by which I mean:
import time
start = time.time()
amount(10)
end = time.time()
print(end - start)
does not work.
How can I measure the time? For example for:
C in range [0, 11]
N = [1, 2 , 5]
I would be grateful for any help! :)

You want somethig like this?
import time
# measure process time
t0 = time.clock()
amount(10)
print time.clock() - t0, "seconds process time"
# measure wall time
t0 = time.time()
amount(10)
print time.time() - t0, "seconds wall time"
UPDATE : The soulution maybe could be this :
times=[]
for x in range(12):
t0 = time.time()
amount(x)
times.append( time.time() - t0);
or better if you use timeit : here

What do you think about this:
import time
def time_counter(amount, n=100, M=100):
res = list(range(n))
def count_once():
start = time.perf_counter()
amount(res)
return time.perf_counter() - start
return [count_once() for m in range(M)]
It is to make M = 1000 measurements and for C in range(0, n). Is it alright?

Timing a Python program using time.clock() vs. time.time() [duplicate]

This question already has answers here:
Python's time.clock() vs. time.time() accuracy?
(16 answers)
Closed 6 years ago.
I am new to Python programming. I started working on Project Euler this morning and I wanted to find out how long it takes to execute my solution. I have searched online for a solution to my
import time
class Solution(object):
def fibonacci(self,limit):
sum = 0
current = 1
next = 2
while(current <= limit):
if current % 2==0:
sum += current
current, next = next, current + next
return str(sum)
if __name__ == "__main__":
start = time.clock()
solution = Solution().fibonacci(4000000)
elapsed = time.clock()-start
print("Solution: %s"%(solution))
print("Time: %s seconds"%(elapsed))
Output:
Solution: 4613732
Time: 2.006085436846098e-05 seconds
import time
class Solution(object):
def fibonacci(self,limit):
sum = 0
current = 1
next = 2
while(current <= limit):
if current % 2==0:
sum += current
current, next = next, current + next
return str(sum)
if __name__ == "__main__":
start = time.time()
solution = Solution().fibonacci(4000000)
elapsed = time.time()-start
print("Solution: %s"%(solution))
print("Time: %s seconds"%(elapsed))
Output:
Solution: 4613732
Time: 0.0 seconds
My question is
Is the time calculated above correct?
What is the difference between time.time() vs time.clock(). If I use time.time() I get 0.0 as time.

In the Python time module, time.clock() measures the time since the first call of the function in seconds, and time.time() measures the time since January 1st, 1970, in seconds.
time.clock() is generally more precise, so using this is what I recommend. This is the reason why you have the tiny result in the first example, rounded down to zero in the second example.