I create multiple sub functions to process data and want to return the main function, once any nested function meets some condition. Then, the multiprocessing can continue the processes.
Here's a simple example:
import multiprocessing
from multiprocessing import Pool
def program1(num):
num += 1
return num
def program2(num):
num += 1
return num
def program3(num):
num += 1
return num
def main(num):
num = program1(num)
if num %2 != 0:
return
else:
num = program2(num)
if num %3 != 0:
return
else:
num = program3(num)
print(num)
return num
pool = Pool(4)
nums = range(8)
pool.map(main, nums)
Because there're multiple programs, it will make the code looks quite long ...
Is it possible to wrap the if condition in each nested function?
Then I can just use
def main(num):
num = program1(num)
num = program2(num)
num = program3(num)
Note that please don't care or simplify the data process hah. This is just an example to show the processes.
Related
I'm trying to run this program but it is showing me "Thread 0 has 0 prime numbers" in the console followed by "Killed" after 5 minutes. Moreover, it is very slow. Please help me develop and correct this code.
import time
Nthreads=4
maxNumber=3000000
starting_range=0
ending_range=0
division=0
lst=[]
def prime(x, y):
prime_list = []
for i in range(x, y):
if i == 0 or i == 1:
continue
else:
for j in range(2, int(i/2)+1):
if i % j == 0:
break
else:
prime_list.append(i)
return prime_list
def func_thread(x, y):
out.append(prime(x, y))
thread_list = []
results = len(lst)
for i in range(Nthreads):
devision=maxNumber//Nthreads
starting_range = (i-1)*division+1
ending_range = i*devision
lst = prime(starting_range, ending_range)
print(" Thread ", i, " has ", len(lst), " prime numbers." )
thread = threading.Thread(target=func_thread, args=(i, results))
thread_list.append(thread)
for thread in thread_list:
thread.start()
for thread in thread_list:
thread.join()```
In Python, if you use multithreading for CPU-bound tasks, it will be slower than if you don't use multithreading. You need to use multiprocessing for this problem. You can read this article for more informations: https://www.geeksforgeeks.org/difference-between-multithreading-vs-multiprocessing-in-python/
Multithreading is wholly inappropriate for CPU-intensive work such as this. However, it can be done:
from concurrent.futures import ThreadPoolExecutor
NTHREADS = 4
MAXNUMBER = 3_000_000
CHUNK = MAXNUMBER // NTHREADS
assert MAXNUMBER % NTHREADS == 0
RANGES = [(base, base+CHUNK) for base in range(0, MAXNUMBER, CHUNK)]
all_primes = []
def isprime(n):
if n <= 3:
return n > 1
if not n % 2 or not n % 3:
return False
for i in range(5, int(n**0.5)+1, 6):
if not n % i or not n % (i + 2):
return False
return True
def process(_range):
lo, hi = _range
if lo < 3:
all_primes.append(2)
lo = 3
elif lo % 2 == 0:
lo += 1
for p in range(lo, hi, 2):
if isprime(p):
all_primes.append(p)
with ThreadPoolExecutor() as executor:
executor.map(process, RANGES)
The all_primes list is unordered. Note also that this strategy will only work if MAXNUMBER is exactly divisible by NTHREADS.
Note on performance:
This takes 7.88s on my machine. A multiprocessing variant takes 2.90s
I have to test if n = 17 is a prime number: (n-1)! mod n = n-1. For that first I have to calculate the factorial of 16 through 4 threads, that means that the interval 1..16 has to be divided in 4 sub-intervals: 1..4, 5..8, 9..12, 14..16. I have managed to test if 17 is a prime number through 16 threads, one for each operation, but I don’t know how I could sub-divide it so that the operations are done in only 4 threads.
I would very much appreciate some ideas, thank you!
Here is my code:
import threading
n = 17
t = n-1
ts = []
num = (n-1)/t
def calcFak():
for i in range(t):
c = i*(n-1)/t+1
thread = threading.Thread(target=threads, args = (c,))
ts.append(thread)
thread.start()
thread.join()
def threads(var):
print(f"start thread {var}")
global num
num = num * var
print(f"end thread {var}")
def managerThread(thread):
calcFak()
print(num)
if num % n == t:
print(n, ' is Prime')
else:
print(n, ' is not Prime')
t2 = threading.Thread(target = managerThread, args=(ts,))
t2.start()
Right now you are calculating everything sequentially because of the thread join() statement in your calcFak() function. Thus, the execution of your function right now is actually the same as the code below:
n = 17
t = n - 1
num = t/t
for i in range(t):
c = i*t/t + 1
print(f'Start {c}')
num = num * c
print(f'End thread {c}')
print(num)
if num % n == t:
print(f'{n} is a prime number')
else:
print(f'{n} is not a prime number')
As you can see, no threading benefit. Instead, use queues and worker functions to subdivide your program's execution:
import threading
from queue import Queue
import time
n = 17
t = n - 1
num = t/t
# For threading
q = Queue()
num_lock = threading.Lock() # Lock so that no threads can overwrite our num value
threads = [] # List to store created threads
# Our worker gets items from the queue and processes them
def worker():
global num
while True:
if not q.empty():
c = q.get()
num_lock.acquire()
num = num * c
num_lock.release()
q.task_done()
for i in range(t):
q.put(i*t/t + 1)
for thread in range(4):
threads.append(threading.Thread(target=worker))
threads[-1].daemon = True
threads[-1].start()
print('Made 1 thread')
q.join()
if num % n == t:
print(f'{n} is a prime number')
else:
print(f'{n} is not a prime number')
# Prints:
# Made 1 thread
# Made 1 thread
# Made 1 thread
# Made 1 thread
# 17 is a prime number
In the code above, we create a queue that holds data for the worker functions to use. The worker functions get a lock on the num variable, and then modify it. Once the queue is empty (which happens after we join the queue with q.join()) we then access the final value of the num variable to determine if the number is prime or not.
I have to get userinputs of ints and store them in a array, and print the max number in the list. But I had to create my own max function. Im not sure what steps to take to implement it into my code.
def getInt(prompt):
n = int
done = False
while not done:
try:
n = int(input(prompt))
except ValueError:
print("I was expecting a number, please try again...")
if n == 0:
done = True
return n
def maxNum(l):
maxi = [0]
for num in l:
if maxi > num:
maxi = num
return maxi
def result():
print("The maxium value is: " + maxNum(i))
def main():
num = []
i = 0
done = False
while not done:
num = getInt("Please enter an integer < 0 to finish >: ")
if num == 0:
done = True
results = maxNum(i)
The below code does exactly what you want.
def getInt(prompt):
try:
n = int(input(prompt))
return n
except ValueError:
print("I was expecting a number, please try again...")
getInt()
def maxNum(lst):
if not lst: # if list is empty
return None
max_elem = lst[0]
for x in lst:
if x > max_elem:
max_elem = x
return max_elem
def main():
nums = []
while True:
num = getInt("Please enter an integer < 0 to finish >: ")
if num == 0:
break
nums.append(num)
result = maxNum(nums)
print("The maxium value is: " + str(result))
main()
python support built-in max function
max([1,2,3]) # return 3
and Your code is totally wrong.
if you want to input array of integers, getInt may be like this.
def getInt():
array = []
while True:
x = int(input('message'))
if x == 0: break
array.append(x)
return array
and main code will be
array = getInt()
max_value = max(array)
print (max_value)
if you want your own max function, it can be
def max_func(array):
max_val = array[0]
for val in array:
if val > max_val: max_val = val
return max_val
Here is a fixed version of your maxNum function:
def maxNum(l):
if not l:
return None # or return whatever you want if user did not input anything
maxi = l[0] # it expects 'l' to be an array!
for num in l[1:]:
if maxi > num:
maxi = num
return maxi
Let's also fix your getInt function:
def getInt(prompt):
while True:
try:
return int(input(prompt))
except ValueError:
print("I was expecting a number, please try again...")
Finally, your "main" function needs the following fix:
def main():
num = []
n = 1
while n != 0:
n = getInt("Please enter an integer < 0 to finish >: ") # store user input into n - do not overwrite num!
num.append(n) # append each user value to the list num
results = maxNum(num) # pass the entire *list* to maxNum
Program have function that may return None value, for minimize spend time this function calling in parallel workers.
In code below, result of this function have the "None" values, how to exclude this is values from the "ret"?
#!/usr/bin/python
import sys,multiprocessing,time
maxNumber = sys.maxint+2
def numGen():
cnt=0
while cnt < maxNumber:
cnt +=1
yield cnt
def oddCheck(num):
global maxNumber
# time.sleep(1)
if not bool(num%1000000):
print "[%s%%] %s" % (int((num/maxNumber)*100),num)
if num%2:
return num
pool = multiprocessing.Pool( )
if sys.maxint < maxNumber:
print "We have problem on %s"%sys.maxint
# quit()
ret = pool.imap(oddCheck, numGen())
pool.close()
pool.join()
for x in ret:
print x
oddCheck returns None when num is not a even number (you do not have a return statement in this case, so None will be returned).
If you want to avoid None results, just use a list comprehension to filter them:
ret = [x for x in pool.imap(oddCheck, numGen()) if x is not None]
It should do the trick ;-)
This question already has answers here:
Read/Write Python Closures
(8 answers)
Closed 9 years ago.
I'm performing an action many times in a loop and want to know how far along I am. I'm trying to make a progress report function that should act something like this:
def make_progress_report(n):
i = 0
def progress_report():
i = i + 1
if i % n == 0:
print i
return progress_report
pr = make_progress_report(2)
pr()
pr() # 2
pr()
pr() # 4
This code does not work. Specifically, I get an UnboundLocalError for i. How should I modify it so that it works?
Here are 3 options:
use a list for your counter:
def make_progress_report(n):
i = [0]
def progress_report():
i[0] = i[0] + 1
if i[0] % n == 0:
print i[0]
return progress_report
use itertools.count to track your counter:
from itertools import count
def make_progress_report(n):
i = count(1)
def progress_report():
cur = i.next()
if cur % n == 0:
print cur
return progress_report
Use nonlocal for your counter (Python 3+ only!):
def make_progress_report(n):
i = 0
def progress_report():
nonlocal i
i = i + 1
if i % n == 0:
print i
return progress_report
You could consider using a generator:
def progress_report(n):
i = 0
while 1:
i = i+1
if i % n == 0:
print i
yield # continue from here next time
pr = progress_report(2)
next(pr)
next(pr)
next(pr)
next(pr)
So the progress_report isn't closed over the variable i. You can check it like so...
>>> def make_progress_report(n):
... i=0
... def progress_report():
... i += 1
... if i % n == 0:
... print i
... return progress_report
...
>>> pr = make_progress_report(2)
>>> pr.__closure__
(<cell at 0x1004a5be8: int object at 0x100311ae0>,)
>>> pr.__closure__[0].cell_contents
2
You'll notice there is only one item in the closure of pr. That is the value you passed in originally for n. The i value isn't part of the closure, so outside of the function definition, i is no longer in scope.
Here is a nice discussion on closures in Python: http://www.shutupandship.com/2012/01/python-closures-explained.html
Look again at how you're defining your closure. The n should be passed in when you define the closure... take the following example:
#!/usr/env python
def progressReportGenerator(n):
def returnProgress(x):
if x%n == 0:
print "progress: %i" % x
return returnProgress
complete = False
i = 0 # counter
n = 2 # we want a progress report every 2 steps
getProgress = progressReportGenerator(n)
while not complete:
i+=1 # increment step counter
# your task code goes here..
getProgress(i) # how are we going?
if i == 20: # stop at some arbtirary point...
complete = True
print "task completed"