I've been writing a few Twisted servers and have created a WatchDog timer that runs periodically. It's default behavior is to check if it was called within some delta of time from it's schedule, which helps to report if the program is being blocked unduly. It also provides a way for a user defined callback function to the WatchDog that could be used to check the health of other parts of the system. The WatchDog timer is implemented using the twisted.internet.task.LoopingCall. I'm concerned if the user defined function creates an exception the WatchDog timer will stop being called. I have Exception handling in the code, but I'd like to have a way to restart the WatchDog timer if it should still manage to crash. However, I don't understand how to use the deferred returned by the LoopingCall().start() method. Here's some sample code to show what I mean:
import sys
from twisted.internet import reactor, defer, task
from twisted.python import log
def periodic_task():
log.msg("periodic task running")
x = 10 / 0
def periodic_task_crashed():
log.msg("periodic_task broken")
log.startLogging(sys.stdout)
my_task = task.LoopingCall(periodic_task)
d = my_task.start(1)
d.addErrback(periodic_task_crashed)
reactor.run()
When I run this code I get one "periodic task running" message from the periodic_task() function and that's it. The deferred returned by my_task.start(1) never has it's errback called, which by my reading of the documentation is what's supposed to happen.
Can someone help me out and point me to what I'm doing wrong?
Thanks in advance!
Doug
The signature of periodic_task_crashed is wrong. It is an error callback on a Deferred, so it will be called with an argument, the Failure representing the error result the Deferred got. Since it is defined to take no arguments, calling it produces a TypeError which becomes the new error result of the Deferred.
Redefine it like this:
def periodic_task_crashed(reason):
log.err(reason, "periodic_task broken")
Related
I have a function in a Tkinter script that utilizes subprocess.Popen.wait() that keeps freezing my GUI. After going through some docs (1 and 2), I found that I need to use asyncio to solve this via asynchronous waiting but I am rather confused:
Note: The function is implemented using a busy loop (non-blocking call
and short sleeps). Use the asyncio module for an asynchronous wait:
see asyncio.create_subprocess_exec.
Does this mean that I have to create a subprocess for my wait() function?
Also, here's what I have so far:
def foo(self)
try:
self.proc = subprocess.Popen(["python", "someModule.py"])
try:
self.proc.wait(11) # <-- freezes GUI
except Exception as l:
someFunc()
except Exception as e:
print(e)
Note: the TimeoutException catch on wait() serves as an indicator for another function to begin execution, whereas successful execution of wait() (i.e., a return value of 0) is for the opposite. Further, my goal is to have a timer in my Tkinter script activated upon successful execution of the child process, which as aforementioned, is indicated by a TimeoutException. The Tkinter interface will always be running whereas the child process will only start on user input and, similarly, end on user input (or if it unexpectedly crashes).
Edit: someModule.py is a script that activates an external data collecting bluetooth device. It must (1) establish a connection to the device, and if (1) is successful then (2) begin collecting data. Function (1) will wait 10 seconds for a connection to the bluetooth device to be established. If after 10 seconds a connection is not made, someModule.py (i.e., the child process) prints an error and terminates. This is why wait() executes for 11 seconds.
How do I implement asynchronous waiting?
Workaround: executing foo() via a thread solved the GUI freezing problem.
import threading
def thread(self):
t1 = threading.Thread(target=foo)
t1.start() # will terminate independently upon completion of foo()
I have a PySide2 application that runs mostly well. However, from time to time, it exits itself without any error (stdout, stderr are empty, even when run from terminal - tried everything) while a worker thread sends updates to the GUI thread via signals.
To emit these signals however, I'm doing it in a way such that my libraries can just take callbacks and not even know they are actually emitting signals, and I was wondering whether this could be a potential source of crashes:
Gui code:
from mylib import do_threaded_work
MyClass(QObject):
sig = Signal(int)
# ... proper initialization and connection of self.sig to a QProgressBar setValue slot
def on_some_button_pressed(self):
threading.Thread(target=do_threaded_work, args=(self.sig.emit,), daemon=True).start()
mylib.py dummy code:
do_threaded_work(on_progress):
i = 0
while (True):
# ... do some work
on_progress(i)
i = min(100, i + 1)
As you see, I'm directly passing the emit function of my signal instance to the library, that calls it and thus should emit the signal. Is it OK to do that?
Sometimes, I pass the signal object self.sig as argument instead of self.sig.emit, then call argument.emit(...) in the library code. I assume it has the same effect?
The reason I ask is because I didn't find any counter argument stating not to do this in PySide2, and the official documentation on signal and slots is quite light here.
You guys have any input on this? Thanks!
I am trying to execute a time-consuming back-end job, executed by a front-end call. This back-end job should execute a callback method when it is completed, which will release a semaphore. The front end shouldn't have to wait for the long process to finish in order to get a response from the call to kick off the job.
I'm trying to use the Pool class from the multiprocessing library to solve this issue, but I'm running into some issues. Namely that it seems like the only way to actually execute the method passed into apply_async is to call the .get() method in the ApplyResult object that is returned by the apply_async call.
In order to solve this, I thought to create a Process object with the target being apply_result.get. But this doesn't seem to work.
Is there a basic understanding that I'm missing here? What would you folks suggest to solve this issue.
Here is a snippet example of what I have right now:
p = Pool(1)
result = p.apply_async(long_process, args=(config, requester), callback=complete_long_process)
Process(target=result.get).start()
response = {'status': 'success', 'message': 'Job started for {0}'.format(requester)}
return jsonify(response)
Thanks for the help in advance!
I don't quite understand why you would need a Process object here. Look at this snippet:
#!/usr/bin/python
from multiprocessing import Pool
from multiprocessing.managers import BaseManager
from itertools import repeat
from time import sleep
def complete_long_process(foo):
print "completed", foo
def long_process(a,b):
print a,b
sleep(10)
p = Pool(1)
result = p.apply_async(long_process, args=(1, 42),
callback=complete_long_process)
print "submitted"
sleep(20)
If I understand what you are trying to achieve, this does exactly that. As soon as you call apply_async, it launches long_process function and execution of the main program continues. As soon as it completes, complete_long_process is called. There is no need to use get method to execute long_process, and the code does not block and wait anything.
If your long_process does not appear to run, I assume your problem is somewhere within long_process.
Hannu
I have a web2py application that basically serves as a browser interface for a Python script. This script usually returns pretty quickly, but can occasionally take a long time. I want to provide a way for the user to stop the script's execution if it takes too long.
I am currently calling the function like this:
def myView(): # this function is called from ajax
session.model = myFunc() # myFunc is from a module which i have complete control over
return dict(model=session.model)
myFunc, when called with certain options, uses multiprocessing but still ends up taking a long time. I need some way to terminate the function, or at the very least the thread's children.
The first thing i tried was to run myFunc in a new process, and roll my own simple event system to kill it:
# in the controller
def myView():
p_conn, c_conn = multiprocessing.Pipe()
events = multiprocessing.Manager().dict()
proc = multiprocessing.Process(target=_fit, args=(options, events c_conn))
proc.start()
sleep(0.01)
session.events = events
proc.join()
session.model = p_conn.recv()
return dict(model=session.model)
def _fit(options, events pipe):
pipe.send(fitting.logistic_fit(options=options, events=events))
pipe.close()
def stop():
try:
session.events['kill']()
except SystemExit:
pass # because it raises that error intentionally
return dict()
# in the module
def kill():
print multiprocessing.active_children()
for p in multiprocessing.active_children():
p.terminate()
raise SystemExit
def myFunc(options, events):
events['kill'] = kill
I ran into a few major problems with this.
The session in stop() wasn't always the same as the session in myView(), so session.events was None.
Even when the session was the same, kill() wasn't properly killing the children.
The long-running function would hang the web2py thread, so stop() wasn't even processed until the function finished.
I considered not calling join() and using AJAX to pick up the result of the function at a later time, but I wasn't able to save the process object in session for later use. The pipe seemed to be able to be pickled, but then I had the problem with not being able to access the same session from another view.
How can I implement this feature?
For long running tasks, you are better off queuing them via the built-in scheduler. If you want to allow the user to manually stop a task that is taking too long, you can use the scheduler.stop_task(ref) method (where ref is the task id or uuid). Alternatively, when you queue a task, you can specify a timeout, so it will automatically stop if not completed within the timeout period.
You can do simple Ajax polling to notify the client when the task has completed (or implement something more sophisticated with websockets or SSE).
I'm building an application using gevent. My app is getting rather big now as there are a lot of jobs being spawned and destroyed. Now I've noticed that when one of these jobs crashes my entire application just keeps running (if the exception came from a non main greenlet) which is fine. But the problem is that I have to look at my console to see the error. So some part of my application can "die" and I'm not instantly aware of that and the app keeps running.
Jittering my app with try catch stuff does not seem to be a clean solution.
Maybe a custom spawn function which does some error reporting?
What is the proper way to monitor gevent jobs/greenlets? catch exceptions?
In my case I listen for events of a few different sources and I should deal with each different.
There are like 5 jobs extremely important. The webserver greenlet, websocket greenlet,
database greenlet, alarms greenlet, and zmq greenlet. If any of those 'dies' my application should completely die. Other jobs which die are not that important. For example, It is possible that websocket greenlet dies due to some exception raised and the rest of the applications keeps running fine like nothing happened. It is completely useless and dangerous now and should just crash hard.
I think the cleanest way would be to catch the exception you consider fatal and do sys.exit() (you'll need gevent 1.0 since before that SystemExit did not exit the process).
Another way is to use link_exception, which would be called if the greenlet died with an exception.
spawn(important_greenlet).link_exception(lambda *args: sys.exit("important_greenlet died"))
Note, that you also need gevent 1.0 for this to work.
If on 0.13.6, do something like this to kill the process:
gevent.get_hub().parent.throw(SystemExit())
You want to greenlet.link_exception() all of your greenlets to a to janitor function.
The janitor function will be passed any greenlet that dies, from which it can inspect its greenlet.exception to see what happened, and if necessary do something about it.
As #Denis and #lvo said, link_exception is OK, but I think there would be a better way for that, without change your current code to spawn greenlet.
Generally, whenever an exception is thrown in a greenlet, _report_error method (in gevent.greenlet.Greenlet) will be called for that greenlet. It will do some stuff like call all the link functions and finally, call self.parent.handle_error with exc_info from current stack. The self.parent here is the global Hub object, this means, all the exceptions happened in each greenlet will always be centralize to one method for handling. By default Hub.handle_error distinguish the exception type, ignore some type and print the others (which is what we always saw in the console).
By patching Hub.handle_error method, we can easily register our own error handlers and never lose an error anymore. I wrote a helper function to make it happen:
from gevent.hub import Hub
IGNORE_ERROR = Hub.SYSTEM_ERROR + Hub.NOT_ERROR
def register_error_handler(error_handler):
Hub._origin_handle_error = Hub.handle_error
def custom_handle_error(self, context, type, value, tb):
if not issubclass(type, IGNORE_ERROR):
# print 'Got error from greenlet:', context, type, value, tb
error_handler(context, (type, value, tb))
self._origin_handle_error(context, type, value, tb)
Hub.handle_error = custom_handle_error
To use it, just call it before the event loop is initialized:
def gevent_error_handler(context, exc_info):
"""Here goes your custom error handling logics"""
e = exc_info[1]
if isinstance(e, SomeError):
# do some notify things
pass
sentry_client.captureException(exc_info=exc_info)
register_error_handler(gevent_error_handler)
This solution has been tested under gevent 1.0.2 and 1.1b3, we use it to send greenlet error information to sentry (a exception tracking system), it works pretty well so far.
The main issue with greenlet.link_exception() is that it does not give any information on traceback which can be really important to log.
For logging with traceback, I use a decorator to spwan jobs which indirect job call into a simple logging function:
from functools import wraps
import gevent
def async(wrapped):
def log_exc(func):
#wraps(wrapped)
def wrapper(*args, **kwargs):
try:
func(*args, **kwargs)
except Exception:
log.exception('%s', func)
return wrapper
#wraps(wrapped)
def wrapper(*args, **kwargs):
greenlet = gevent.spawn(log_exc(wrapped), *args, **kwargs)
return wrapper
Of course, you can add the link_exception call to manage jobs (which I did not need)