I'm trying to call a multiprocessing function from a python thread to avoid the Global Interpreter Lock affecting my multiprocessing function.
The logic looks like this:
python --- ongoing python processing...
\
-> py thread -> py multiprocessing call -> C-code (dll/ctypes)
Does this make sense? Will the C code run on a separate core, or is this too complex to work?
Thanks.
EDIT: Thanks for the reply. I should clarify, but I need to call on a second thread because I have to first make a python array and then pass the pointer to a C function. So I can't call the multiprocessing function too early (and also the main python processing needs to start and continue seamlessly).
EDIT: Here's the code logic and why I can't call a 2nd process inline with main code:
main():
...
p = Process(target=save_c, args=(...,))
p.start()
p.join() #Main thread will lock here and wait until return;
#Other code that needs to be processed follows the multiprocess call
save_c():
''' Function which calls C-module '''
_sum = ctypes.CDLL('/home/pi/murphylab_picam/temp/libsum.so')
_sum.c_function.argtypes = (ctypes.c_int, ctypes.POINTER(ctypes.c_int))
_sum.c_function(ctypes.c_int(num_numbers), array_type(*_sum.numbers))
What am I missing here? Is there a different way to use multiprocessing inline with ongoing processing?
You don't need to join immediately after you create a process as long as you don't want to wait for that process to finish before continuing.
This is the principle of concurrent programming.
Important thing is that you eventually call join or your main process will terminate leaving the child orphan.
child_process = Process(....)
child_process.start() # the child process will start its independend execution here
do_some_stuff()
do_some_more_stuff()
child_process.join() # here you are waiting for it to finish
Related
I'm trying to use multiprocessing to run multiple scripts. At the start, I launch a loading animation, however I am unable to ever kill it. Below is an example...
Animation: foo.py
import sys
import time
import itertools
# Simple loading animation that runs infinitely.
for c in itertools.cycle(['|', '/', '-', '\\']):
sys.stdout.write('\r' + c)
sys.stdout.flush()
time.sleep(0.1)
Useful script: bar.py
from time import sleep
# Stand-in for a script that does something useful.
sleep(5)
Attempt to run them both:
import multiprocessing
from multiprocessing import Process
import subprocess
pjt_dir = "/home/solebay/path/to/project" # Setup paths..
foo_path = pjt_dir + "/foo.py" # ..
bar_path = pjt_dir + "/bar.py" # ..
def run_script(path): # Simple function that..
"""Launches python scripts.""" # ..allows me to set a..
subprocess.run(["python", path]) # ..script as a process.
foo_p = Process(target=run_script, args=(foo_path,)) # Define the processes..
bar_p = Process(target=run_script, args=(bar_path,)) # ..
foo_p.start() # start loading animation
bar_p.start() # start 'useful' script
bar_p.join() # Wait for useful script to finish executing
foo_p.kill() # Kill loading animation
I get no error messages, and (my_venv) solebay#computer:~$ comes up in my terminal, but the loading animation persists (clipping over my name and environement). How can I kill it?
I've run into a similar situation before where I couldn't terminate the program using ctrl + c. The issue is (more or less) solved by using daemonic processes/threads (see multiprocessing doc). To do this, you simply change
foo_p = Process(target=run_script, args=(foo_path,))
to
foo_p = Process(target=run_script, args=(foo_path,), daemon=True)
and similarly for other children processes that you would like to create.
With that being said, I myself am not exactly sure if this is the correct way to remedy the issue with not being able to terminate the multiprocessing program, or is it just some artifact that happens to help with this. I would suggest this thread that went into the discussion about daemon threads more. But essentially, from my understanding, daemon threads would be terminated automatically whenever their parent process is terminated, regardless of whether they are finished or not. Meanwhile, if a thread is not daemonic, then somehow you need to wait until the children processes to finish before you're able to fully terminate the program.
You are creating too many processes. These two lines:
foo_p = Process(target=run_script, args=(foo_path,)) # Define the processes..
bar_p = Process(target=run_script, args=(bar_path,)) # ..
create two new processes. Let's all them "A" and "B". Each process consists of this function:
def run_script(path): # Simple function that..
"""Launches python scripts.""" # ..allows me to set a..
subprocess.run(["python", path]) # ..script as a process.
which then creates another subprocess. Let's call those two processes "C" and "D". In all you have created 4 extra processes, instead of just the 2 that you need. It is actually process "C" that's producing the output on the terminal. This line:
bar_p.join()
waits for "B" to terminate, which implies that "D" has terminated. But this line:
foo_p.kill()
kills process "A" but orphans process "C". So the output to the terminal continues forever.
This is well documented - see the description of multiprocessing.terminate, which says:
"Note that descendant processes of the process will not be terminated – they will simply become orphaned."
The following program works as you intended, exiting gracefully from the second process after the first one has finished. (I renamed "foo.py" to useless.py and "bar.py" to useful.py, and made small changes so I could run it on my computer.)
import subprocess
import os
def run_script(name):
s = os.path.join(r"c:\pyproj310\so", name)
return subprocess.Popen(["py", s])
if __name__ == "__main__":
useless_p = run_script("useless.py")
useful_p = run_script("useful.py")
useful_p.wait() # Wait for useful script to finish executing
useless_p.kill() # Kill loading animation
You can't use subprocess.run() to launch the new processes since that function will block the main script until the process completes. So I used Popen instead. Also I placed the running code under an if __name__ == "__main__" which is good practice (and maybe necessary on Windows).
I have a program, which uses multiprocesses to execute functions from an external hardware library. The communication between the multiprocess and my program happens with JoinableQueue().
A part of the code looks like this:
# Main Code
queue_cmd.put("do_something")
queue_cmd.join() # here is my problem
# multiprocess
task = queue_cmd.get()
if task == "do_something":
external_class.do_something()
queue_cmd.task_done()
Note: external_class is the external hardware library.
This library sometimes crashes and the line queue_cmd.task_done() never gets executed. As a result, my main program hangs indefinitely in the queue_cmd.join() part, waiting for the queue_cmd.task_done() to be called. Unfortunately, there is no timeout parameter for the join() function.
How can I wait for the element in the JoinableQueue to be processed, but also deal with the event of my multiprocess terminating (due to the crash in the do_something() function)?
Ideally, the join function would have a timeout parameter (.join(timeout=30)), which I could use to restart the multiprocess - but it does not.
You can always wrap a blocking function on another thread:
queue_cmd.put("do_something")
t = Thread(target=queue_cmd.join)
t.start()
# implement a timeout
start = datetime.now()
timeout = 10 # seconds
while t.is_alive() and (datetime.now() - start).seconds < timeout:
# do something else
# waiting for the join or timeout
if t.is_alive():
# kill the subprocess that failed
pass
I think the best approach here is to start the "crashable" module in (yet) another process:
Main code
queue_cmd.put("do_something")
queue_cmd.join()
Multiprocess (You can now move this to a thread)
task = queue_cmd.get()
if task == "do_something":
subprocess.run(["python", "pleasedontcrash.py"])
queue_cmd.task_done()
pleasedontcrash.py
external_class.do_something()
As shown, I'd do it using subprocess. If you need to pass parameters (which you could with subprocess using pipes or arguments), it's easier to use multiprocessing.
I know there is something called thread, but I am confused by those complex information all over Google. myFunc() takes a little time (not computationally expensive, say play a short mp3 file).
What I want to do is call myFunc() and don't need to wait for it to return to run the following lines of code. Furthermore, I don't need to keep anything related to myFunc(arg), I only need it to be executed only.
while(True):
......
myFunc(arg)
###some
###lines
###of
###code
Sorry for my bad English. Cheers!
from threading import Thread
def myFunc(arg):
# run code here
while(True):
thread = Thread(target = myFunc, args = (arg, ))
thread.start() # starts the thread, executes the function
###some
###lines
###of
###code
thread.join() # wait for myFunc to finish
You can do similarly with processes instead of threads.
Might want take a look into pools if you want to perform a list of arguments with the same function. You can call imap and iterate the results and call the rest of the code.
I am implementing a Flask application and I'm calling a function A, based on a request. Inside the function A, there is another function called B and it is called. But, I don't need to wait until the end of the execution of function B to return the output from function A. I have done it with the following code implementation.
from threading import Thread
def functionA():
result = doSomething1()
Thread(target=functionB).start()
return result
def functionB():
# Do something after the execution of doSomething1()
Here, I am starting a new thread and do what I need to do but, I do not terminate the newly started thread. Do I need to terminate that thread by myself? If so, what is the best way to do that?
A thread will terminate on its own. To see if it is alive you can use thread.is_Alive()
If you must force termination of a thread, use thread.join() would be the closes option, as this blocks the calling thread until the thread in question has finished.
Also refer to the docs for more info on the Threading functions
https://docs.python.org/3/library/threading.html
I want to create multi process app. Here is sample:
import threading
import time
from logs import LOG
def start_first():
LOG.log("First thread has started")
time.sleep(1000)
def start_second():
LOG.log("second thread has started")
if __name__ == '__main__':
### call birhtday daemon
first_thread = threading.Thread(target=start_first())
### call billing daemon
second_thread = threading.Thread(target=start_second())
### starting all daemons
first_thread.start()
second_thread.start()
In this code second thread does not work. I guess, after calling sleep function inside first_thread main process is slept. I found this post. But here sleep was used with class. I got that(Process finished with exit code 0
) as a result when I run answer. Could anybody explain me where I made a mistake ?
I am using python 3.* on windows
When creating your thread you are actually invoking the functions when trying to set the target for the Thread instead of passing a function to it. This means when you try to create the first_thread you are actually calling start_first which includes the very long sleep. I imagine you then get frustrated that you don't see the output from the second thread and kill it, right?
Remove the parens from your target= statements and you will get what you want
first_thread = threading.Thread(target=start_first)
second_thread = threading.Thread(target=start_second)
first_thread.start()
second_thread.start()
will do what you are trying