Suppose I have this class:
class Foo:
def __init__(self):
self.task1_dict = {}
self.task2_dict = {}
def task1(self):
for i in range(10000000):
# update self.task1_dict
def task2(self):
for i in range(10000000):
# update self.task2_dict
def run(self):
self.task1()
self.task2()
Task 1 and task 2 are both CPU intensive tasks and are non-IO. They are also independent so you can assume that running them concurrently is thread safe.
For now, my class is running the tasks sequentially and I want to change it so the tasks are run in parallel in multiple threads. I'm using the ThreadPoolExecutor from the concurrent.future package.
class Foo:
...
def run(self):
with ThreadPoolExecutor() as executor:
executor.submit(self.task1)
executor.submit(self.task2)
The problem is when I call the run method the run time does not decrease at all and even slightly increases compared to the sequential version. I'm guessing that this is because of the GIL allowing only one thread to run at a time. Is there any way that I can parallelise this program? Maybe a way to overcome the GIL and run the 2 methods on 2 threads? I have considered switching to ProcessPoolExecutor, but I cannot call the methods since class methods are not picklable. Also if I use multiprocessing, Python will create multiple instances of Foo and self.task1_dict and self.task2_dict would not be updated accordingly.
You can use multiprocessing shared memory as explained here
Related
Recently I have started using the multiprocessor pool executor in python to accelerate my processing.
So instead of doing a
list_of_res=[]
for n in range(a_number):
res=calculate_something(list_of sources[n])
list_of_res.append(res)
joint_results=pd.concat(list_of_res)
I do
with ProcessPoolExecutor(max_workers=8) as executor:
joint_results=pd.concat(executor.map(calculate_something,list_of_sources))
It works great.
However I've noticed that inside the calculate_something function I call the same function like 8 times, one after another, so I might as well apply a map to them instead of a loop
My question is, can I apply multiprocessing to a function that is already being called in multiprocess?
yes you can have a worker process spawn another pool of workers, but it is not optimal.
each time you launch a new process it takes a few hundred milliseconds to a few seconds for this new process to initialize and start executing work (OS, disk and code dependent.)
launching a worker from a worker is just wasting the overhead of spawning the first child to begin with, and you are better off extracting the loop inside calculate_something and launching it directly within your initial executor.
a better approach is to launch your initial calculate_something using a ThreadPoolExecutor and have one shared ProcessPoolExecutor that all your thread workers will push work into, this way you can limit the number of newly created processes and avoid creating and deleting much more workers than you actually need, and it takes only a few microseconds to launch a threadpool.
this is an example of how to nest threadpool and process_pool.
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
def process_worker(n):
print(n)
return n
def thread_worker(list_of_n,process_pool:ProcessPoolExecutor):
work_done = list(process_pool.map(process_worker,list_of_n))
return work_done
if __name__ == "__main__":
list_of_lists_of_n = [[1,2,3],[4,5,6]]
with ProcessPoolExecutor() as process_pool, ThreadPoolExecutor() as threadpool:
tasks = []
work_done = []
for item in list_of_lists_of_n:
tasks.append(threadpool.submit(thread_worker,item,process_pool))
for item in tasks:
work_done.append(item.result())
print(work_done)
Will the following way of using a thread pool cause a deadlock? Or is such a pattern not preferred? If so, what is the alternative.
Passing pool to a function that is run in a thread, which in turn invokes a function that is run the same pool.
from concurrent.futures import ThreadPoolExecutor
from time import sleep
def bar():
sleep(2)
return 2
def foo(pool):
sleep(2)
my_list = [pool.submit(bar) for i in range(4)]
return [i.result() for i in my_list]
pool = ThreadPoolExecutor(10)
my_list = [pool.submit(foo, pool) for i in range(2)]
for i in my_list:
print(i.result())
This would be a safe way to spawn a thread from within a thread that itself was initiated by ThreadPoolExecutor. This may not be necessary if ThreadPoolExecutor itself is thread-safe. The output shows how, in this case, there would be 10 concurrent threads.
from concurrent.futures import ThreadPoolExecutor
from time import sleep
BAR_THREADS = 4
FOO_THREADS = 2
def bar(_):
print('Running bar')
sleep(1)
def foo(_):
print('Running foo')
with ThreadPoolExecutor(max_workers=BAR_THREADS) as executor:
executor.map(bar, range(BAR_THREADS))
with ThreadPoolExecutor(max_workers=FOO_THREADS) as executor:
executor.map(foo, range(FOO_THREADS))
print('Done')
Output:
Running foo
Running foo
Running bar
Running bar
Running bar
Running bar
Running bar
Running bar
Running bar
Running bar
Done
Will the following way of using a thread pool cause a deadlock? ... If so, what is the alternative?
One alternative would be to use a thread pool that does not have a hard limit on the number of workers. Unfortunately, the concurrent.futures.ThreadPoolExecutor class is not so sophisticated. You either would have to write your own, or else find one provided by a third party. (I'm not a big-time Python programmer, so I don't know of one off-hand.)
A naive alternative thread-pool might create a new worker any time submit() was called and all of the existing workers were busy. On the other hand, that could make it easy for you to run the program out of memory by creating too many threads. A slightly more sophisticated thread pool might also kill off a worker if too many other workers were idle at the moment when the worker completed its task.
More sophisticated strategies are possible, but you might have to think more deeply about the needs and patterns-of-use of the application before writing the code.
I am currently creating a class which is supposed to execute some methods in a multi-threaded way, using the multiprocessing module. I execute the real computation using a Pool of n workers. Now I wanted to assign each of the currently n active workers an index between 0 and n for some other calculation. To do this, I wanted to use a shared Queue to assign an index in a way, that at every time no two workers have the same id. To share the same Queue inside the class between the different threads, I wanted to store it inside a Manager.Namespace(). But doing this, I got some problems with the Queue. Therefore, I created a minimal version of my problem and ended up with something like this:
from multiprocess import Process, Queue, Manager, Pool, cpu_count
class A(object):
def __init__(self):
manager = Manager()
self.ns = manager.Namespace()
self.ns.q = manager.Queue()
def foo(self):
for i in range(10):
print(i)
self.ns.q.put(i)
print(self.ns.q.get())
print(self.ns.q.qsize())
a = A()
a.foo()
In this code, the execution stops before the second print statement - therefore, I think, that no data is actually written in the Queue. When I remove the namespace related stuff the code works flawlessly. Is this the intended behaviour of the multiprocessings objects and am I doing something wrong? Or is this some kind of bug?
yes, you should not use Namespace here. when you put a Queue object into manager.Namespace(), each process will get a new Queue instance, all the writer/reader of those newly created queue objects have no connection with parent process, therefore no message will be received by worker processes. share a Queue solely instead.
by the way, you mentioned "thread" many times, but in the context of multiprocess module, a worker is a process, not a thread.
I have the following code:
class SplunkUKAnalyser(object):
def __init__
def method1
def method2
def method2
...
class SplunkDEAnalyser(SplunkUKAnalyser):
def __init__ (Over-ridden)
def method1 (Over-ridden)
def method2
def method2
...
perform_uk_analysis():
my_uk_analyser = SplunkUKAnalyser()
perform_de_analysis():
my_de_analyser = SplunkDEAnalyser()
It all works well if I just execute the below:
perform_uk_analysis()
perform_de_analysis()
How can I make it so that the two last statements are executed concurrently. (using mutliprocessing and/or multi-threading)?
From my test it seems that the second statement executes even though the first statement has not finished completely but I would like to incorporate true concurrency.
Any other additional advice is much appreciated.
Many thanks in advance.
Because of GIL (Global Interpreter Lock) you can not achieve 'true concurrency' with threading.
However, using multiprocessing to concurrently run multiple tasks is easy:
import multiprocessing
process1 = multiprocessing.Process(target=perform_uk_analysis)
process2 = multiprocessing.Process(target=perform_de_analysis)
# you can optionally daemoize the process
process2.daemon = True
# run the tasks concurrently
process1.start()
process2.start()
# you can optionally wait for a process to finish
process2.join()
For tasks that run the same function with different arguments, consider using multiprocessing.Pool, an even more convenient solution.
I have a class function in python.
I run the class in many different instances
class worker():
def__init__(x,x)
def run():
instance1 = worker(x,x)
instance1.run()
instance2 = worker(x,x)
instance2.run()
The problem is if first instance1 encounter thread.sleep() it affects the other instance2. How do i make them independent. Better if without multi-process Thank you!
Different example:
__author__ = 'user'
import time
class test():
def __init__(self, message):
self.message=message
def run(self):
while True:
print self.message
time.sleep(5)
if __name__ == '__main__':
test1 = test("PRINT-1")
test1.run()
test2 = test("PRINT-2")
test2.run()
you can use Celery for run parallel tasks. It's easy to implement.
See an example:
import time
from celery import task
#task
def make_task():
time.sleep(5)
return True
def execute_tasks():
result = group([make_task.s(), make_task.s()]).apply_async() # Execute tasks
print result.get() # Print the result
It looks like you're half-followed a tutorial on parallel code. Nothing in your current test class will cause it to run in parallel, but with just some minor tweaks you can do so with either threads or processes.
Here's a version that makes the class inherit from threading.Thread:
import threading
import time
class TestThreaded(threading.Thread):
def __init__(self, x, y):
super().__init__()
self.x = x
self.y = y
def run(self):
for i in range(self.x):
time.sleep(self.y)
print((i+1)*self.y)
You can use it like this:
t0 = TestThreaded(8, 3)
t1 = TestThreaded(6, 4)
t0.start()
t1.start()
t0.join()
t1.join()
Both threads in this example will count to 24 over a span of 24 seconds. The first thread will count by threes, the second thread will count by fours. The timings will be closely synched at 12 and 24 seconds (depending on your computer's exact timings they may get printed on the same line).
Note that we're calling the start method inherited from the Thread class, not the run method we defined above. The threading code will call run for us, in the spawned thread.
You can get an equivalent multiprocessing version by using multiprocessing.Process as the base class instead of threading.Thread. The only difference is that you'll spawn child processes instead of child threads. For CPU limited work in Python, processes are better than threads because they're not limited by the Global Interpreter Lock, which makes it impossible for two threads to run Python code at the same time. The downside is higher overhead, both during startup, and when communicating between processes.