I'm making a library that uses gevent to do some work asynchronously. I'd like to guarantee that the work is completed, even if the main module finishes execution.
class separate_library(object):
def __init__(self):
import gevent.monkey; gevent.monkey.patch_all()
def do_work(self):
from gevent import spawn
spawn(self._do)
def _do(self):
from gevent import sleep
sleep(1)
print 'Done!'
if __name__ == '__main__':
lib = separate_library()
lib.do_work()
If you run this, you'll notice the program ends immediately, and Done! doesn't get printed.
Now, the main module doesn't know, or care, how separate_library actually accomplishes the work (or even that gevent is being used), so it's unreasonable to require joining there.
Is there any way separate_library can detect certain types of program exits, and stall until the work is done? Keyboard interrupts, SIGINTs, and sys.exit() should end the program immediately, as that is probably the expected behaviour.
Thanks!
Try using a new thread that is not a daemon thread that spawns your gevent threads. Your program will not exit due to this non daemon thread.
import gevent
import threading
class separate_library(object):
def __init__(self):
import gevent.monkey; gevent.monkey.patch_all()
def do_work(self):
t = threading.Thread(target=self.spawn_gthreads)
t.setDaemon(False)
t.start()
def spawn_gthreads(self):
from gevent import spawn
gthreads = [spawn(self._do,x) for x in range(10)]
gevent.joinall(gthreads)
def _do(self,sec):
from gevent import sleep
sleep(sec)
print 'Done!'
if __name__ == '__main__':
lib = separate_library()
lib.do_work()
Related
I am using concurent.futures.ProcessPoolExecutor has a high level API to multiprocessing.
I want to identify the current process in the worker functions.
With the low level API multiprocessing I can do it like this.
import multiprocessing
def worker():
print(multiprocessing.current_process())
Is there a current_process() pendant when using workers with the ProcessPoolExecutor()?
Since each execution happens in a separate process, you can simply do
import os
def worker():
# Get the process ID of the current process
pid = os.getpid()
..
.. do something with pid
For example,
from concurrent.futures import ProcessPoolExecutor
import os
import time
def task():
time.sleep(1)
print("Executing on Process {}".format(os.getpid()))
def main():
with ProcessPoolExecutor(max_workers=3) as executor:
for i in range(3):
executor.submit(task)
if __name__ == '__main__':
main()
➜ python3.9 so.py
Executing on Process 71137
Executing on Process 71136
Executing on Process 71138
Note that if the task in hand is small and executed fast enough, your pid might stay the same. Try it out by removing the time.sleep call from my example.
import os
import sys
from multiprocessing import Process, Queue
import threading
class Test:
def __init__(self):
print '__init__ is called'
def say_hello_again_and_again(self):
print 'Hello :D'
threading.Timer(1, self.say_hello_again_and_again).start()
test = Test()
#test.say_hello_again_and_again()
process = Process(target=test.say_hello_again_and_again)
process.start()
this is test code.
the result:
pi#raspberrypi:~/Plant2 $ python test2.py
__init__ is called
Hello :D
If I use test.say_hello_again_and_again() , "Hello :D" is printed repeatedly.
But, process is not working as I expected. Why is "Hello :D" not being printed in my process?
What's happening in my process?
There are two problems with your code:
First: You start a process with start(). This is doing a fork, that means now you have two processes, the parent and the child running side by side. Now, the parent process immediately exits, because after start() it's the end of the program. To wait until the child has finished (which in your case is never), you have to add process.join().
I did test your suggestion, but it not works
Indeed. There is a second issue: You start a new thread with threading.Timer(1, ...).start() but then immediately end the process. Now, you don't wait until your thread started because the underlying process immediately dies. You'd need to also wait until the thread has stopped with join().
Now that's how your program would look like:
from multiprocessing import Process
import threading
class Test:
def __init__(self):
print '__init__ is called'
def say_hello_again_and_again(self):
print 'Hello :D'
timer = threading.Timer(1, self.say_hello_again_and_again)
timer.start()
timer.join()
test = Test()
process = Process(target=test.say_hello_again_and_again)
process.start()
process.join()
But this is suboptimal at best because you mix multiprocessing (which is using fork to start independent processes) and threading (which starts a thread within the process). While this is not really a problem it makes debugging a lot harder (one problem e.g. with the code above is that you can't stop it with ctrl-c because of some reason your spawned process is inherited by the OS and kept running). Why don't you just do this?
from multiprocessing import Process, Queue
import time
class Test:
def __init__(self):
print '__init__ is called'
def say_hello_again_and_again(self):
while True:
print 'Hello :D'
time.sleep(1)
test = Test()
process = Process(target=test.say_hello_again_and_again)
process.start()
process.join()
I'm trying to close child process(which is doing while loop) when parent process is exited (Whenever parent process is clean-exit, forced-exit or exited because of exception) not to make child process a zombie process.
I'm making a game that communicates with Arduino (using serial), and main process is running Panda3D's ShowBase instance(Game engine, do render and another many things) so main must not be stopped.
So, I created subprocess using multiprocessing module so that main process is safe from stopping to wait serial in.
But the problem is, when i close Panda3D window, call sys.exit() or exited because of exception, main process exits immediately, and can't join or give false to subprocess, so subprocess becomes zombie.
I have no idea how to solve this. What should i do to make it work as i expected?
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from multiprocessing import Process, Queue
from panda3d.core import *
class HW_support:
def hardware_event_handler(self, process_status):
self.process_alive = True
while self.process_alive:
print('Working!')
self.process_alive = process_status.get()
return
if __name__ == '__main__':
from direct.showbase.ShowBase import ShowBase
import sys
class TestApp(ShowBase):
def __init__(self):
ShowBase.__init__(self)
self.process_status_argv = Queue()
self.HW_sub_process = Process(target = HW_support().hardware_event_handler, args=(self.process_status_argv,))
self.HW_sub_process.start()
base.messenger.toggleVerbose()
taskMgr.add(self.task_make_alive, 'task_make_alive')
base.accept('escape', self.exit_taskloop)
def exit_taskloop(self, task=None):
taskMgr.stop()
def task_make_alive(self, task=None):
self.process_status_argv.put(True)
return task.cont
app = TestApp()
app.run()
#app.HW_sub_process.join()
app.process_status_argv.put(False)
in the main program add this at the top (well below import multiprocessing)
if multiprocessing.current_process().name == 'MainProcess':
import atexit
atexit.register(lambda *a : os.remove("running.txt"))
open("running.txt","wb").close()
in the subprocces change your while True loop to while os.path.exists("running.txt"):
alternatively you could have atexit place a message in the queue or do whatever to signal to the subprocess that it should exit.
Multiple processes makes things a lot more complicated.
To shut down the HW_support process cleanly, you need to send it the message via your Queue object, then the parent needs to join() it (wait for it to exit) before exiting itself.
Anything that could make the parent exit unexpectedly (console interrupt, thrown exception, sys.exit, etc etc) will need to be carefully caught and managed so that you can still cleanly shut down the child before exiting.
I'm new to Python and I'm writing a script that
includes some timed routines.
My current approach is to instantiate a class
that includes those Timers (from: threading.Timer),
but I don't want the script to return when it gets to the
end of the function:
import mytimer
timer = mytimer()
Suppose I have a imple script like that one. All it
does is instantiate a mytimer object which performs a series
of timed activities.
In order for the application not to exit, I could use Qt like this:
from PyQt4.QtCore import QCoreApplication
import mytimer
import sys
def main():
app = QCoreApplication(sys.argv)
timer = mytimer()
sys.exit(app.exec_())
if __name__ == '__main__':
main()
This way, the sys.exit() call won't return immediately, and the
timer would just keep doing its thing 'forever' in background.
Although this is a solution I've used before, using Qt just for this doesn't
fell right to me.
So my question is, Is there any way to accomplish this using standard Python?
Thanks
Create a function in your script which tests a select or poll object to terminate a loop. Check out serve_forever in SocketServer.py from the standard library as an example.
A Google search for "python timer" finds:
http://docs.python.org/library/sched.html
http://docs.python.org/release/2.5.2/lib/timer-objects.html
The sched module seems to be exactly what you need.
Example:
>>> import sched, time
>>> s = sched.scheduler(time.time, time.sleep)
>>> def print_time(): print "From print_time", time.time()
...
>>> def print_some_times():
... print time.time()
... s.enter(5, 1, print_time, ())
... s.enter(10, 1, print_time, ())
... s.run()
... print time.time()
...
>>> print_some_times()
930343690.257
From print_time 930343695.274
From print_time 930343700.273
930343700.276
Once you have built your queue of times for things to happen, you just call the .run() method on your sched instance, and it will automatically wait until the queue is emptied, then will complete. So you can just put s.run() as the last thing in your script, and it will automatically exit only when the timed tasks are all done.
import mytimer
import sys
from threading import Lock
lock = Lock()
lock.acquire() # put lock into locked state
def main():
timer = mytimer()
lock.acquire() # blocks until someone calls lock.release()
if __name__ == '__main__':
main()
If you want a clean exit, you can just make mytimer() call lock.release() at some point.
I am using gevent and I am monkey patching everything.
It seems like the monkey patching causes the threading to work serially.
My code:
import threading
from gevent import monkey; monkey.patch_all()
class ExampleThread(threading.Thread):
def run(self):
do_stuff() # takes a few minutes to finish
print 'finished working'
if __name__ == '__main__':
worker = ExampleThread()
worker.start()
print 'this should be printed before the worker finished'
So the thread is not working as expected.
But if I remove the monkey.patch_all() it is working fine.
The problem is that I need the monkey.patch_all() for using gevent (now shown in the code above)
My solution:
I changed the
monkey.patch_all()
to
monkey.patch_all(thread=False)
so I am not patching the thread.
When threads are monkey patched in gevent, they behave as coroutines. This means that you have to explicitly yield control to make it possible for other coroutines to execute.
The way to do this is call a blocking operation that has been patched (this will yield automatically) or gevent.sleep:
#!/usr/bin/env python
from gevent import monkey, sleep
monkey.patch_all()
import threading
class ExampleThread(threading.Thread):
def run(self):
for i in xrange(10):
print 'working'
sleep()
if __name__ == '__main__':
worker = ExampleThread()
worker.start()
print 'this will be printed after the first call to sleep'
You can leave your Thread-based class in place if you substitute the Thread with Greenlet, for example like this:
from gevent import monkey
from gevent import Greenlet
from threading import Thread
class ThreadLikeGreenlet(Greenlet):
def __init__(self, name=None, target=None, args=(), kwargs=()):
super().__init__(target, *args, **dict(kwargs))
self.name = name
def is_gevent_patched():
return monkey.is_module_patched('threading')
if is_gevent_patched():
Thread = ThreadLikeGreenlet # substitute Thread with Greenlet
class ExampleThread(Thread):
...
it will work as you wish then.