Develop Reference

Create a python code that counts prime numbers in range(3000000) using for-looped multithreading(e.g. if Nthreads is N, it is gonna divide 3000000/N)

I'm trying to run this program but it is showing me "Thread 0 has 0 prime numbers" in the console followed by "Killed" after 5 minutes. Moreover, it is very slow. Please help me develop and correct this code.
import time
Nthreads=4
maxNumber=3000000
starting_range=0
ending_range=0
division=0
lst=[]
def prime(x, y):
prime_list = []
for i in range(x, y):
if i == 0 or i == 1:
continue
else:
for j in range(2, int(i/2)+1):
if i % j == 0:
break
else:
prime_list.append(i)
return prime_list
def func_thread(x, y):
out.append(prime(x, y))
thread_list = []
results = len(lst)
for i in range(Nthreads):
devision=maxNumber//Nthreads
starting_range = (i-1)*division+1
ending_range = i*devision
lst = prime(starting_range, ending_range)
print(" Thread ", i, " has ", len(lst), " prime numbers." )
thread = threading.Thread(target=func_thread, args=(i, results))
thread_list.append(thread)
for thread in thread_list:
thread.start()
for thread in thread_list:
thread.join()```

In Python, if you use multithreading for CPU-bound tasks, it will be slower than if you don't use multithreading. You need to use multiprocessing for this problem. You can read this article for more informations: https://www.geeksforgeeks.org/difference-between-multithreading-vs-multiprocessing-in-python/

Multithreading is wholly inappropriate for CPU-intensive work such as this. However, it can be done:
from concurrent.futures import ThreadPoolExecutor
NTHREADS = 4
MAXNUMBER = 3_000_000
CHUNK = MAXNUMBER // NTHREADS
assert MAXNUMBER % NTHREADS == 0
RANGES = [(base, base+CHUNK) for base in range(0, MAXNUMBER, CHUNK)]
all_primes = []
def isprime(n):
if n <= 3:
return n > 1
if not n % 2 or not n % 3:
return False
for i in range(5, int(n**0.5)+1, 6):
if not n % i or not n % (i + 2):
return False
return True
def process(_range):
lo, hi = _range
if lo < 3:
all_primes.append(2)
lo = 3
elif lo % 2 == 0:
lo += 1
for p in range(lo, hi, 2):
if isprime(p):
all_primes.append(p)
with ThreadPoolExecutor() as executor:
executor.map(process, RANGES)
The all_primes list is unordered. Note also that this strategy will only work if MAXNUMBER is exactly divisible by NTHREADS.
Note on performance:
This takes 7.88s on my machine. A multiprocessing variant takes 2.90s

Python Concurrent.Futures ProcessPool Executor wrong output

I am trying to use the ProcessPoolExecutor() to run some functions but I cant manage to understand how to get the return of the functions out of the with.
def threaded_upload(i):
time.sleep(2)
if i == 0:
k = 10
elif i == 2:
k = i*i
else:
k = -99
return [k]
def controller():
if __name__ == "__main__":
futures = []
with ProcessPoolExecutor() as pool:
for paso in range(4):
futuro_i = pool.submit(threaded_upload,paso)
wth=[futuro_i.result()]
futures.append(futuro_i)
wait(futures, return_when=ALL_COMPLETED)
merged_list = []
for future in futures:
for valor in future.result():
merged_list.append(valor)
Lista_Final = merged_list
wait(futures, return_when=ALL_COMPLETED)
return Lista_Final
print(controller())
The output of the code is:
None
[10, -99, 4, -99]
I am not sure why?
The "wait" doesn't seem to wait until all functions are executed either.
To be honest, I have been reading and reading for a few days but the description of concurrent.futures or multiprocessing are more advanced that my current knowledge.
Any clarification will be appreciated.
Thanks in advance.

You first submit the jobs and then wait for the results. You can also return an integer instead of a list and then skip the inner loop:
test.py:
import random
import time
from concurrent.futures import ProcessPoolExecutor, wait
def worker(i):
t = random.uniform(1, 5)
print(f"START: {i} ({t:.3f}s)")
time.sleep(t)
if i == 0:
k = 10
elif i == 2:
k = i * i
else:
k = -99
print(f"END: {i}")
return k
def main():
futures = []
with ProcessPoolExecutor() as pool:
for i in range(4):
future = pool.submit(worker, i)
futures.append(future)
results = []
done, pending = wait(futures) # ALL_COMPLETED is the default value
for future in done:
results.append(future.result())
print(results)
if __name__ == "__main__":
main()
Test:
$ python test.py
START: 0 (1.608s)
START: 1 (1.718s)
START: 2 (1.545s)
START: 3 (1.588s)
END: 2
END: 3
END: 0
END: 1
[10, -99, 4, -99]

Calculate factorials with threads in Python

I have to test if n = 17 is a prime number: (n-1)! mod n = n-1. For that first I have to calculate the factorial of 16 through 4 threads, that means that the interval 1..16 has to be divided in 4 sub-intervals: 1..4, 5..8, 9..12, 14..16. I have managed to test if 17 is a prime number through 16 threads, one for each operation, but I don’t know how I could sub-divide it so that the operations are done in only 4 threads.
I would very much appreciate some ideas, thank you!
Here is my code:
import threading
n = 17
t = n-1
ts = []
num = (n-1)/t
def calcFak():
for i in range(t):
c = i*(n-1)/t+1
thread = threading.Thread(target=threads, args = (c,))
ts.append(thread)
thread.start()
thread.join()
def threads(var):
print(f"start thread {var}")
global num
num = num * var
print(f"end thread {var}")
def managerThread(thread):
calcFak()
print(num)
if num % n == t:
print(n, ' is Prime')
else:
print(n, ' is not Prime')
t2 = threading.Thread(target = managerThread, args=(ts,))
t2.start()

Right now you are calculating everything sequentially because of the thread join() statement in your calcFak() function. Thus, the execution of your function right now is actually the same as the code below:
n = 17
t = n - 1
num = t/t
for i in range(t):
c = i*t/t + 1
print(f'Start {c}')
num = num * c
print(f'End thread {c}')
print(num)
if num % n == t:
print(f'{n} is a prime number')
else:
print(f'{n} is not a prime number')
As you can see, no threading benefit. Instead, use queues and worker functions to subdivide your program's execution:
import threading
from queue import Queue
import time
n = 17
t = n - 1
num = t/t
# For threading
q = Queue()
num_lock = threading.Lock() # Lock so that no threads can overwrite our num value
threads = [] # List to store created threads
# Our worker gets items from the queue and processes them
def worker():
global num
while True:
if not q.empty():
c = q.get()
num_lock.acquire()
num = num * c
num_lock.release()
q.task_done()
for i in range(t):
q.put(i*t/t + 1)
for thread in range(4):
threads.append(threading.Thread(target=worker))
threads[-1].daemon = True
threads[-1].start()
print('Made 1 thread')
q.join()
if num % n == t:
print(f'{n} is a prime number')
else:
print(f'{n} is not a prime number')
# Prints:
# Made 1 thread
# Made 1 thread
# Made 1 thread
# Made 1 thread
# 17 is a prime number
In the code above, we create a queue that holds data for the worker functions to use. The worker functions get a lock on the num variable, and then modify it. Once the queue is empty (which happens after we join the queue with q.join()) we then access the final value of the num variable to determine if the number is prime or not.

Subprocesses are not conducted when use multiprocessing in Python

I'm now testing the code below to calculate an array in parallel, but it seems that the long time function would not be conducted. My Python version: 3.7.4, Operation system: win 10.
from multiprocessing import Pool, Lock, Array
import os, time
def long_time_task(i,array,lock):
print('Run task %s (%s)...' % (i, os.getpid()))
start = time.time()
total_count = 0
for k in range(5*10**7): total_count += 1
total_count += i
lock.acquire()
array[i] = total_count
lock.release()
end = time.time()
print('Task %s runs %0.2f seconds.' % (i, (end - start)))
def mainFunc():
print('Parent process %s.' % os.getpid())
p = Pool()
array = Array('f', 20)
lock = Lock()
for i in range(20): p.apply_async(long_time_task, args=(i,array,lock))
print('Waiting for all subprocesses done...')
p.close()
p.join()
print('All subprocesses done.')
if __name__ == '__main__':
mainFunc()

There are a few issues with your code:
The apply_async method returns a result object that you need to await for.
You can't pass normal multiprocessing Array or Lock objects to Pool methods, because they can't be pickled. You can use manager object instead.
Try this:
from multiprocessing import Pool, Lock, Array, Manager
import os, time
def long_time_task(i,array,lock):
print('Run task %s (%s)...' % (i, os.getpid()))
start = time.time()
total_count = 0
for k in range(5*10**7): total_count += 1
total_count += i
lock.acquire()
array[i] = total_count
lock.release()
end = time.time()
print('Task %s runs %0.2f seconds.' % (i, (end - start)))
def mainFunc():
print('Parent process %s.' % os.getpid())
p = Pool()
m = Manager()
array = m.Array('f', [0] * 20)
lock = m.Lock()
results = [p.apply_async(long_time_task, args=(i,array,lock)) for i in range(20)]
[result.get() for result in results]
print('Waiting for all subprocesses done...')
p.close()
p.join()
print('All subprocesses done.')
if __name__ == '__main__':
mainFunc()
You could also simplify it like this but I'm not sure if this will work in your case:
import array
import os
import time
from multiprocessing import Pool
def long_time_task(i):
print(f'Run task {i} ({os.getpid()})...')
start = time.time()
total_count = 0
for k in range(5 * 10 ** 7):
total_count += 1
total_count += i
end = time.time()
print(f'Task {i} runs {end - start:.2f} seconds.')
return total_count
def main():
print('Parent process %s.' % os.getpid())
a = array.array('d', range(20))
r = range(20)
with Pool() as pool:
for idx, result in zip(r, pool.map(long_time_task, r)):
a[idx] = result
print(a)
print(f'All subprocesses done.')
if __name__ == '__main__':
main()

Python quick-sort parallel sort slower than sequential

Need some help with code below, there have to be something wrong with it because I get better results with sequential sort than with parallel version.I'm new to python and especially parallel programming and any help would be welcome.
import random, time
from multiprocessing import Process, Pipe,cpu_count
from copy import deepcopy
def main():
create_list = [random.randint(1,1000) for x in range(25000)]
#sequential sort
sequentialsortlist=deepcopy(create_list)
start = time.time()
sorted2 =quicksort(sequentialsortlist)
elapsed = time.time() - start
print("sequential sort")
print(elapsed)
time.sleep(4)
#Parallel quicksort.
parallelsortlist = deepcopy(create_list)
start = time.time()
n = cpu_count()
pconn, cconn = Pipe()
p = Process(target=quicksortParallel,
args=(parallelsortlist, cconn, n,))
p.start()
lyst = pconn.recv()
p.join()
elapsed = time.time() - start
print("Parallels sort")
print(elapsed)
def quicksort(lyst):
less = []
pivotList = []
more = []
if len(lyst) <= 1:
return lyst
else:
pivot = lyst[0]
for i in lyst:
if i < pivot:
less.append(i)
elif i > pivot:
more.append(i)
else:
pivotList.append(i)
less = quicksort(less)
more = quicksort(more)
return less + pivotList + more
def quicksortParallel(lyst, conn, procNum):
less = []
pivotList = []
more = []
if procNum <= 0 or len(lyst) <= 1:
conn.send(quicksort(lyst))
conn.close()
return
else:
pivot = lyst[0]
for i in lyst:
if i < pivot:
less.append(i)
elif i > pivot:
more.append(i)
else:
pivotList.append(i)
pconnLeft, cconnLeft = Pipe()
leftProc = Process(target=quicksortParallel,
args=(less, cconnLeft, procNum - 1))
pconnRight, cconnRight = Pipe()
rightProc = Process(target=quicksortParallel,
args=(more, cconnRight, procNum - 1))
leftProc.start()
rightProc.start()
conn.send(pconnLeft.recv()+pivotList + pconnRight.recv())
conn.close()
leftProc.join()
rightProc.join()
if __name__ == '__main__':
main()

The simple answer is that the overhead of setting up your parallel execution environment and then re-joining it at the end is more expensive then the performance increase gained from the parallelism.
Multi-processing actually forks sub-processes. That's very expensive. It only makes sense to do this if the amount of work done in each thread is very large.
This kind of problem is actually pretty common when people neivly try and parallelize code. It's pretty common that for many 'reasonable' workloads the single-threaded implementation winds up being faster.

There is a cost associated with starting/terminating a process. Interprocess communication is not free, either. So the overhead is just too big.