I need to monitor a function. I want to create a class Logger that will monitor the run() function, then automatically stream the monitoring data to a specified logfile. For simplicity, let's say I only want to log the start and stop time of the run() function. I want to be able to call an instance of class Logger() within the run() function to automatically monitor run(), without any further code (i.e., without start and stop commands).
def run(protocol=protocolContext):
Logger = Logger(protocol=protocol)
#a bunch of instructions that take a long time
I am really a beginner, and I am having trouble understanding how I need to structure the class in order to do this. Do I need to use threading? So far all I have is:
from datetime import datetime
class Logger:
protocol = None
def __init__(self, protocol):
self.start = None
self.stop = None
self.protocol = protocol
def start(self):
self.start = datetime.datetime.now()
def stop(self):
self.stop = datetime.datetime.now()
def print_to_logfile(self):
#code that opens the logfile and prints the start and stop time
Upon looking up event handler modules, I came across pydispatcher, which seemed beginner friendly. My use case for the library is that I want to send a signal if my queue size is over a threshold. The handler function can then start processing and removing items from the queue (and subsequently do a bulk insert into the database).
I would like the handler function to run in the background. I am aware that I can simply overwrite the queue.append() method checking for the queue size and calling the handler function asynchronously, but I would like to implement the listener-dispatcher model to keep the logic clean and separated.
Does pydispatcher do this out of the box? If not, is there another module that can help me do this? Would I need to manage the access to the queue, since there might be multiple threads processing and appending to the queue at the same time?
Note that in my use case there is only a single dispatcher and event handler.
I've recently released the Akuanduba module, that may help you with this task. There's a single example on the repository that may help you understand how it works and it seems similar to what you want.
Anyway, I'll try to explain here a way of implementing your code with Akuanduba:
First you could make a data frame that would hold your queue:
# Mandatory imports
from Akuanduba.core.messenger.macros import *
from Akuanduba.core.constants import *
from Akuanduba.core import NotSet, AkuandubaDataframe
# Your imports go here:
from queue import Queue
class MyQueue (AkuandubaDataframe):
def __init__(self, name):
# Mandatory stuff
AkuandubaDataframe.__init__(self, name)
self.__queue = Queue ()
def getQueue (self):
return self.__queue
def putQueue (self, val):
self.__queue.put(val)
def getQueueSize (self):
return self.__queue.qsize()
#
# "toRawObj" method is a mandatory method that delivers a dict with the desired data
# for file saving
#
def toRawObj(self):
d = {
"Queue" : self.getQueue(),
}
return d
Then you could make a TriggerCondition that would check the queue size:
from Akuanduba.core import StatusCode, NotSet, StatusTrigger
from Akuanduba.core.messenger.macros import *
from Akuanduba.core import TriggerCondition
import time
class CheckQueueSize (TriggerCondition):
def __init__(self, name, maxSize):
TriggerCondition.__init__(self, name)
self._name = name
self._maxSize = maxSize
def initialize(self):
return StatusCode.SUCCESS
def execute (self):
size = self.getContext().getHandler("MyQueue").getQueueSize()
if (size > SIZE_THRESHOLD):
return StatusTrigger.TRIGGERED
else:
return StatusTrigger.NOT_TRIGGERED
def finalize(self):
return StatusCode.SUCCESS
Make a tool that would be your handler function:
# Mandatory imports
from Akuanduba.core import AkuandubaTool, StatusCode, NotSet, retrieve_kw
# Your imports go here:
class SampleTool(AkuandubaTool):
def __init__(self, name, **kw):
# Mandatory stuff
AkuandubaTool.__init__(self, name)
def initialize(self):
# Lock the initialization. After that, this tool can not be initialized once again
self.init_lock()
return StatusCode.SUCCESS
def execute(self,context):
#
# DO SOMETHING HERE
#
# Always return SUCCESS
return StatusCode.SUCCESS
def finalize(self):
self.fina_lock()
return StatusCode.SUCCESS
And finally, make a main script in order to make it all work together:
# Akuanduba imports
from Akuanduba.core import Akuanduba, LoggingLevel, AkuandubaTrigger
from Akuanduba import ServiceManager, ToolManager, DataframeManager
# This sample's imports
import MyQueue, CheckQueueSize, SampleTool
# Creating your handler
your_handler = SampleTool ("Your Handler's name")
# Creating dataframes
queue = MyQueue ("MyQueue")
# Creating trigger
trigger = AkuandubaTrigger("Sample Trigger Name", triggerType = 'or')
# Append conditions and tools to trigger just adding them
# Tools appended to the trigger will only run when trigger is StatusTrigger.TRIGGERED,
# and will run in the order they've been appended
trigger += CheckQueueSize( "CheckQueueSize condition", MAX_QUEUE_SIZE )
trigger += your_handler
# Creating Akuanduba
manager = Akuanduba("Akuanduba", level=LoggingLevel.INFO)
# Appending tools
#
# ToolManager += TOOL_1
# ToolManager += TOOL_2
#
ToolManager += trigger
# Apprending dataframes
DataframeManager += sampleDataframe
# Initializing
manager.initialize()
manager.execute()
manager.finalize()
That way, you'd have clean and separated code.
In pymodbus library in server.sync, SocketServer.BaseRequestHandler is used, and defines as follow:
class ModbusBaseRequestHandler(socketserver.BaseRequestHandler):
""" Implements the modbus server protocol
This uses the socketserver.BaseRequestHandler to implement
the client handler.
"""
running = False
framer = None
def setup(self):
""" Callback for when a client connects
"""
_logger.debug("Client Connected [%s:%s]" % self.client_address)
self.running = True
self.framer = self.server.framer(self.server.decoder, client=None)
self.server.threads.append(self)
def finish(self):
""" Callback for when a client disconnects
"""
_logger.debug("Client Disconnected [%s:%s]" % self.client_address)
self.server.threads.remove(self)
def execute(self, request):
""" The callback to call with the resulting message
:param request: The decoded request message
"""
try:
context = self.server.context[request.unit_id]
response = request.execute(context)
except NoSuchSlaveException as ex:
_logger.debug("requested slave does not exist: %s" % request.unit_id )
if self.server.ignore_missing_slaves:
return # the client will simply timeout waiting for a response
response = request.doException(merror.GatewayNoResponse)
except Exception as ex:
_logger.debug("Datastore unable to fulfill request: %s; %s", ex, traceback.format_exc() )
response = request.doException(merror.SlaveFailure)
response.transaction_id = request.transaction_id
response.unit_id = request.unit_id
self.send(response)
# ----------------------------------------------------------------------- #
# Base class implementations
# ----------------------------------------------------------------------- #
def handle(self):
""" Callback when we receive any data
"""
raise NotImplementedException("Method not implemented by derived class")
def send(self, message):
""" Send a request (string) to the network
:param message: The unencoded modbus response
"""
raise NotImplementedException("Method not implemented by derived class")
setup() is called when a client is connected to the server, and finish() is called when a client is disconnected. I want to manipulate these methods (setup() and finish()) in another class in another file which use the library (pymodbus) and add some code to setup and finish functions. I do not intend to modify the library, since it may cause strange behavior in specific situation.
---Edited ----
To clarify, I want setup function in ModbusBaseRequestHandler class to work as before and remain untouched, but add sth else to it, but this modification should be done in my code not in the library.
The simplest, and usually best, thing to do is to not manipulate the methods of ModbusBaseRequestHandler, but instead inherit from it and override those methods in your subclass, then just use the subclass wherever you would have used the base class:
class SoupedUpModbusBaseRequestHandler(ModbusBaseRequestHandler):
def setup(self):
# do different stuff
# call super().setup() if you want
# or call socketserver.BaseRequestHandler.setup() to skip over it
# or call neither
Notice that a class statement is just a normal statement, and can go anywhere any other statement can, even in the middle of a method. So, even if you need to dynamically create the subclass because you won't know what you want setup to do until runtime, that's not a problem.
If you actually need to monkeypatch the class, that isn't very hard—although it is easy to screw things up if you aren't careful.
def setup(self):
# do different stuff
ModbusBaseRequestHandler.setup = setup
If you want to be able to call the normal implementation, you have to stash it somewhere:
_setup = ModbusBaseRequestHandler.setup
def setup(self):
# do different stuff
# call _setup whenever you want
ModbusBaseRequestHandler.setup = setup
If you want to make sure you copy over the name, docstring, etc., you can use `wraps:
#functools.wraps(ModbusBaseRequestHandler.setup)
def setup(self):
# do different stuff
ModbusBaseRequestHandler.setup = setup
Again, you can do this anywhere in your code, even in the middle of a method.
If you need to monkeypatch one instance of ModbusBaseRequestHandler while leaving any other instances untouched, you can even do that. You just have to manually bind the method:
def setup(self):
# do different stuff
myModbusBaseRequestHandler.setup = setup.__get__(myModbusBaseRequestHandler)
If you want to call the original method, or wraps it, or do this in the middle of some other method, etc., it's otherwise basically the same as the last version.
It can be done by Interceptor
from functools import wraps
def iterceptor(func):
print('this is executed at function definition time (def my_func)')
#wraps(func)
def wrapper(*args, **kwargs):
print('this is executed before function call')
result = func(*args, **kwargs)
print('this is executed after function call')
return result
return wrapper
#iterceptor
def my_func(n):
print('this is my_func')
print('n =', n)
my_func(4)
more explanation can be found here
So I have written a wrapper WebApiSession for a Web API. When an instance is created a login etc is done and a session is created. The session needs to be kept alive so the constructor launches a separate process handling this. The method close() logs out of the session as well as stops the process. Now ideally I would not want to have to call close(). Instead I want this to happen when the instance is not needed anymore, i.e. I would like to be able to remove the session.close() call below. Is this possible?
import time
from multiprocessing import Process
class WebApiSession:
def __init__(self):
# start session, login etc
# ...
# start touch loop
self.touchLoop = Process(target = self.runTouchLoop)
self.touchLoop.start()
def runTouchLoop(self):
self.touch()
time.sleep(1)
self.runTouchLoop()
def touch(self):
# touch session
pass
def close(self):
# logout etc
# ...
self.touchLoop.terminate()
def doSomething(self):
pass
if __name__ == '__main__':
session = WebApiSession()
session.doSomething()
session.close()
It sounds like you could benefit from implementing WebApiSession as a context manager. You can then treat your session like any other "context" that has special methods that must be called when it's opened and closed, like a file or other connection. It would also would give you the added bonus of neatly wrapping up errors and so on.
class WebApiSession(object):
def __init__(self):
pass # other init stuff here, but don't connect yet.
def __enter__(self): # entering the context.
# start session, login, start touch loop
self.touchLoop = Process(target = self.runTouchLoop)
self.touchLoop.start()
return self
def __exit__(self, exc_type, exc_val, traceback): # leaving the context.
# Bonus feature: handle exception info here as needed!
self.close()
if __name__ == '__main__':
with WebApiSession() as session:
session.doSomething()
A friend and I have been playing around with pygame some and came across this tutorial for building games using pygame. We really liked how it broke out the game into a model-view-controller system with events as a go-between, but the code makes heavy use of isinstance checks for the event system.
Example:
class CPUSpinnerController:
...
def Notify(self, event):
if isinstance( event, QuitEvent ):
self.keepGoing = 0
This results in some extremely unpythonic code. Does anyone have any suggestions on how this could be improved? Or an alternative methodology for implementing MVC?
This is a bit of code I wrote based on #Mark-Hildreth answer (how do I link users?) Does anyone else have any good suggestions? I'm going to leave this open for another day or so before picking a solution.
class EventManager:
def __init__(self):
from weakref import WeakKeyDictionary
self.listeners = WeakKeyDictionary()
def add(self, listener):
self.listeners[ listener ] = 1
def remove(self, listener):
del self.listeners[ listener ]
def post(self, event):
print "post event %s" % event.name
for listener in self.listeners.keys():
listener.notify(event)
class Listener:
def __init__(self, event_mgr=None):
if event_mgr is not None:
event_mgr.add(self)
def notify(self, event):
event(self)
class Event:
def __init__(self, name="Generic Event"):
self.name = name
def __call__(self, controller):
pass
class QuitEvent(Event):
def __init__(self):
Event.__init__(self, "Quit")
def __call__(self, listener):
listener.exit(self)
class RunController(Listener):
def __init__(self, event_mgr):
Listener.__init__(self, event_mgr)
self.running = True
self.event_mgr = event_mgr
def exit(self, event):
print "exit called"
self.running = False
def run(self):
print "run called"
while self.running:
event = QuitEvent()
self.event_mgr.post(event)
em = EventManager()
run = RunController(em)
run.run()
This is another build using the examples from #Paul - impressively simple!
class WeakBoundMethod:
def __init__(self, meth):
import weakref
self._self = weakref.ref(meth.__self__)
self._func = meth.__func__
def __call__(self, *args, **kwargs):
self._func(self._self(), *args, **kwargs)
class EventManager:
def __init__(self):
# does this actually do anything?
self._listeners = { None : [ None ] }
def add(self, eventClass, listener):
print "add %s" % eventClass.__name__
key = eventClass.__name__
if (hasattr(listener, '__self__') and
hasattr(listener, '__func__')):
listener = WeakBoundMethod(listener)
try:
self._listeners[key].append(listener)
except KeyError:
# why did you not need this in your code?
self._listeners[key] = [listener]
print "add count %s" % len(self._listeners[key])
def remove(self, eventClass, listener):
key = eventClass.__name__
self._listeners[key].remove(listener)
def post(self, event):
eventClass = event.__class__
key = eventClass.__name__
print "post event %s (keys %s)" % (
key, len(self._listeners[key]))
for listener in self._listeners[key]:
listener(event)
class Event:
pass
class QuitEvent(Event):
pass
class RunController:
def __init__(self, event_mgr):
event_mgr.add(QuitEvent, self.exit)
self.running = True
self.event_mgr = event_mgr
def exit(self, event):
print "exit called"
self.running = False
def run(self):
print "run called"
while self.running:
event = QuitEvent()
self.event_mgr.post(event)
em = EventManager()
run = RunController(em)
run.run()
A cleaner way of handling events (and also a lot faster, but possibly consumes a bit more memory) is to have multiple event handler functions in your code. Something along these lines:
The Desired Interface
class KeyboardEvent:
pass
class MouseEvent:
pass
class NotifyThisClass:
def __init__(self, event_dispatcher):
self.ed = event_dispatcher
self.ed.add(KeyboardEvent, self.on_keyboard_event)
self.ed.add(MouseEvent, self.on_mouse_event)
def __del__(self):
self.ed.remove(KeyboardEvent, self.on_keyboard_event)
self.ed.remove(MouseEvent, self.on_mouse_event)
def on_keyboard_event(self, event):
pass
def on_mouse_event(self, event):
pass
Here, the __init__ method receives an EventDispatcher as an argument. The EventDispatcher.add function now takes the type of the event you are interested in, and the listener.
This has benefits for efficiency since the listener only ever gets called for events that it is interested in. It also results in more generic code inside the EventDispatcher itself:
EventDispatcher Implementation
class EventDispatcher:
def __init__(self):
# Dict that maps event types to lists of listeners
self._listeners = dict()
def add(self, eventcls, listener):
self._listeners.setdefault(eventcls, list()).append(listener)
def post(self, event):
try:
for listener in self._listeners[event.__class__]:
listener(event)
except KeyError:
pass # No listener interested in this event
But there is a problem with this implementation. Inside NotifyThisClass you do this:
self.ed.add(KeyboardEvent, self.on_keyboard_event)
The problem is with self.on_keyboard_event: it is a bound method which you passed to the EventDispatcher. Bound methods hold a reference to self; this means that as long as the EventDispatcher has the bound method, self will not be deleted.
WeakBoundMethod
You will need to create a WeakBoundMethod class that holds only a weak reference to self (I see you already know about weak references) so that the EventDispatcher does not prevent the deletion of self.
An alternative would be to have a NotifyThisClass.remove_listeners function that you call before deleting the object, but that's not really the cleanest solution and I find it very error prone (easy to forget to do).
The implementation of WeakBoundMethod would look something like this:
class WeakBoundMethod:
def __init__(self, meth):
self._self = weakref.ref(meth.__self__)
self._func = meth.__func__
def __call__(self, *args, **kwargs):
self._func(self._self(), *args, **kwargs)
Here's a more robust implementation I posted on CodeReview, and here's an example of how you'd use the class:
from weak_bound_method import WeakBoundMethod as Wbm
class NotifyThisClass:
def __init__(self, event_dispatcher):
self.ed = event_dispatcher
self.ed.add(KeyboardEvent, Wbm(self.on_keyboard_event))
self.ed.add(MouseEvent, Wbm(self.on_mouse_event))
Connection Objects (Optional)
When removing listeners from the manager/ dispatcher, instead of making the EventDispatcher needlessly search through the listeners until it finds the right event type, then search through the list until it finds the right listener, you could have something like this:
class NotifyThisClass:
def __init__(self, event_dispatcher):
self.ed = event_dispatcher
self._connections = [
self.ed.add(KeyboardEvent, Wbm(self.on_keyboard_event)),
self.ed.add(MouseEvent, Wbm(self.on_mouse_event))
]
Here EventDispatcher.add returns a Connection object that knows where in the EventDispatcher's dict of lists it resides. When a NotifyThisClass object is deleted, so is self._connections, which will call Connection.__del__, which will remove the listener from the EventDispatcher.
This could make your code both faster and easier to use because you only have to explicitly add the functions, they are removed automatically, but it's up to you to decide if you want to do this. If you do it, note that EventDispatcher.remove shouldn't exist anymore.
I stumbled upon SJ Brown's tutorial on making games in the past. It's a great page, one of the best I've read. However, like you, I didn't like the calls to isinstance, or the fact that all the listeners receive all the events.
First, isinstance is slower than checking that two strings are equals, so I ended up storing a name on my events and test for the name rather than the class. But still, the notify function with its battery of if was itching me because it felt like a waste of time. We can do two optimizations here:
Most listeners are interested in only a few types of events. For performance reasons, when QuitEvent is posted, only the listeners interested in it should be notified. The event manager keeps track of which listener wants to listen to which event.
Then, to avoid going through a tons of if statements in a single notify method, we will have one method per type of event.
Example:
class GameLoopController(...):
...
def onQuitEvent(self, event):
# Directly called by the event manager when a QuitEvent is posted.
# I call this an event handler.
self._running = False
Because I want the developer to type as little as possible, I made the following thing:
When a listener is registered to an event manager, the event manager scans all the methods of the listener. When one method starts with 'on' (or any prefix you like), then it looks at the rest ("QuitEvent") and binds this name to this method. Later, when the event manager pumps its event list, it looks at the event class name: "QuitEvent". It knows that name, and therefore can directly call all the corresponding event handlers directly. The developer has nothing to do but adding onWhateverEvent methods to have them working.
It has some drawbacks:
If I make a typo in the name of the handler ("onRunPhysicsEvent"
instead of "onPhysicsRanEvent" for example") then my handler will
never be called and I'll wonder why. But I know the trick so I
don't wonder why very long.
I cannot add an event handler after the listener has been
registered. I must un-register and re-register. Indeed, the
events handlers are scanned only during the registration. Then
again, I never had to do that anyway so I don't miss it.
Despite these drawbacks I like it much more than having the constructor of the listener explicitly explain the event manager that it wants to stay tuned of this, this, this and this event. And it's the same execution speed anyway.
Second point:
When designing our event manager, we want to be careful. Very often, a listener will respond to an event by creating-registering or unregistering-destroying listeners. This happens all the time. If we don't think about it then our game may break with RuntimeError: dictionary changed size during iteration. The code that you propose iterates over a copy of the dictionary so you're protected against explosions; but it has consequences to be aware of:
- Listeners registered because of an event will not receive that event.
- Listeners unregistered because of an event will still receive that
event.
I never found it to be a problem though.
I implemented that myself for the game I am developing. I can link you to two articles and a half I wrote on the subject:
http://niriel.wordpress.com/2011/08/06/who-controls-the-controllers/
http://niriel.wordpress.com/2011/08/08/the-event-management-is-in-place/
http://niriel.wordpress.com/2011/08/11/the-first-screenshot-of-infiniworld/
The links to my github account will bring you directly to the source code of the relevant parts. If you cannot wait, here's the thing: https://github.com/Niriel/Infiniworld/blob/v0.0.2/src/evtman.py . In there you'll see that the code for my event class is a bit big, but that every inherited event is declared in 2 lines: the base Event class is making your life easy.
So, this all works using python's introspection mechanism, and using the fact that methods are objects like any other that can be put in dictionaries. I think it's quite pythony :).
Give each event a method (possibly even using __call__), and pass in the Controller object as an argument. The "call" method should then call the controller object. For example...
class QuitEvent:
...
def __call__(self, controller):
controller.on_quit(self) # or possibly... controller.on_quit(self.val1, self.val2)
class CPUSpinnerController:
...
def on_quit(self, event):
...
Whatever code you're using to route your events to your controllers will call the __call__ method with the correct controller.
I stumbled upon the same issue (almost a decade later!), and here is an implementation I've been using so that the EventManager would notify only a subset of listeners.
It is based on defaultdict: the _listeners attribute of EventManager is a defaultdict of WeakKeyDictionary().
Event are all inherited from an empty abstract Event class, so listeners can focus only on some classes of events they want to listen to.
Here is a minimalist code to get the idea behind it:
from collections import defaultdict
from weakref import WeakKeyDictionary
class Event:
def __init__(self):
pass
class KeyboardEvent(Event): # for instance, a keyboard event class with the key pressed
def __init__(self, key):
self._key = key
class EventManager:
def __init__(self):
self._listeners = defaultdict(lambda: WeakKeyDictionary())
def register_listener(self, event_types, listener):
for event_type in event_types:
self._listeners[event_type][listener] = 1
def unregister_listener(self, listener):
for event_type in self._listeners:
self._listeners[event_type].pop(listener, None)
def post_event(self, event):
for listener in self._listeners[event.__class__]:
listener.notify(event)
When registering, a listener tells the event manager which event type it wants to be notified of.
When posting events, the event manager will only notify the listeners which registered to be notified for that type of event.
This piece of code has, of course, much less scope than the very general (and very elegant) solution proposed by #Paul Manta, but in my case, it helped remove some repetitive calls to isinstance and other checks while keeping things as simple as I could.
One of the drawbacks of this is that all type of events have to be objects of some class, but in OO python, this is supposed to be the way to go.