Develop Reference

Detach COM events using pywin32

Is it possible to detach a specific event after attaching it to a COM object?
For example, how to deregister the ClassOfHandlers in the following snippet:
from win32com.client import WithEvents
# ...
class ClassOfHandlers():
def OnStart(self):
print("Start observed")
class AnotherClassOfHandlers():
def OnStart(self):
print("Start observed from another")
WithEvents(client, ClassOfHandlers)
# ...
WithEvents(client, AnotherClassOfHandlers)
# ...
# Deregister `ClassOfHandlers`

As a variation on the OP's answer, which avoids a static member variable, it is worth remembering that WithEvents() returns an instance of the handler class.
from win32com.client import WithEvents
def MyOnStart():
print("Start observed")
def MySecondOnStart():
print("Start observed from another")
class ClassOfHandlers():
def __init__(self):
self._fn = MyOnStart
def setStartFunction(self,fn):
self._fn = fn
def OnStart(self):
self._fn()
handler = WithEvents(client, ClassOfHandlers)
#then later
handler.setStartFunction(MySecondOnStart)
Hence you can re-use the handler class for a different client.
Alternatively you could try opening an issue here and maybe the developers can advise on whether they expose the IConnectionPoint::Unadvise() function which would be needed behind the scenes to switch event handlers (I think).

Edit
Based on DS_London's answer we could benefit from WithEvents return, thus the combined solution would look like
from win32com.client import WithEvents
def MyOnStart():
print("Start observed")
def MySecondOnStart():
print("Start observed from another")
class ClassOfHandlers():
def __init__(self):
self._onStarts = []
# self._onStops = []
# ... add a list of functions for each event type
# the following 3 methods are implemented for each event type
def attachStart(self, fn):
self._onStarts.append(fn)
def detachStart(self, fn):
self._onStarts.remove(fn)
def OnStart(self):
for fn in self._onStarts:
fn()
# Always at the beginning
handler = WithEvents(client, ClassOfHandlers)
handler.attachStart(MyOnStart)
# ...
handler.attachStart(MySecondOnStart)
# ...
handler.detachStart(MyOnStart)
Limitation
If support for multiple clients is needed and thus threading is used, this edit won't work, and it would be needed to use the original answer's approach.
The cause: one needs to pass the ClassOfHandlers to the thread runnable*, however the thread runnable would PumpWaitingMessages() till interrupted, thus it won't be able to return the client handler back, preventing us from being able to detach/attach further functions while waiting for messages.
* PumpWaitingMessages() requires that it runs on the same thread that connected the ClassOfHandlers to the client, thus we can't create the client handler out of the thread then send it into the thread runnable.
Following is a snippet that shows this scenario:
def threadRunnable(client, eventsClass, controller):
pythoncom.CoInitializeEx(pythoncom.COINIT_MULTITHREADED)
# Connect the custom events
# The connection needs to be done inside the same thread for PumpWaitingMessages
handler = WithEvents(client, eventsClass)
if controller == None:
print("no control was provided")
controller = { "sleep_time": 1, "running_flag": True}
# With this while we won't be able to return the handler
while controller["running_flag"]:
pythoncom.PumpWaitingMessages()
time.sleep(controller["sleep_time"])
pythoncom.CoUninitialize()
def connectEvents(client, eventsClass, controller=None, runnable=threadRunnable):
flusher = Thread(target=runnable, args=(client,eventsClass,controller))
flusher.daemon = True
flusher.start()
def main():
controller = { "sleep_time": 1, "running_flag": True}
connectEvents(client, ClassOfHandlers, controller)
Original
I'm now able to achieve the desired behavior, by attaching a single permanent observer class and managing the events myself.
For example:
from win32com.client import WithEvents
# ...
class ClassOfHandlers():
OnStarts = []
def OnStart(self):
for handler in ClassOfHandlers.OnStarts:
handler()
def MyOnStart():
print("Start observed")
def MySecondOnStart():
print("Start observed from another")
# Always at the beginning
WithEvents(client, ClassOfHandlers)
ClassOfHandlers.OnStarts.append(MyOnStart)
# ...
ClassOfHandlers.OnStarts.append(MySecondOnStart)
# ...
ClassOfHandlers.OnStarts.remove(MyOnStart)
Hint:
The class variable OnStarts shall be changed to an instance variable if the class represents an instantiable COM object, to allow having an instance of the ClassOfHandlers (each instance having a different handler list) for each instantiated COM object.
One also needs to ensure that WithEvents is called only once for each COM object instance.

How to call "process" function using ALAudioDevice "subscribe" method

I'm new at NAO programming and I'm having some trouble regarding the ALAudioDevice API.
My problem is the following one: I wrote a python module that should record raw data from the front microphone.
The documentation of the ALAudioDevice API says that the method "subscribe(...)" calls the function "process" automatically
and regularly with raw data from microphones as inputs. I wrote a code to execute this process (see bellow), and it runs without raising
the error flag. However, the subscribe is bypassing the function "process" and the module doesn't get any audio at all.
Has someone had the same problem?
import qi
class AudioModule(object):
def __init__(self):
super(AudioModule, self).__init__()
self.moduleName = "AudioModule"
try :
self.ALAudioDevice = ALProxy("ALAudioDevice")
except Exception, e:
self.logger.error("Error when creating proxy on ALAudioDevice:")
self.logger.error(e)
def begin_stream(self):
self.ALAudioDevice.setClientPreferences(self.moduleName, 16000, 3, 0)
self.ALAudioDevice.subscribe(self.moduleName)
def end_stream(self):
self.ALAudioDevice.unsubscribe(self.moduleName)
def processRemote( self, nbOfChannels, samplesByChannel, altimestamp, buffer ):
nbOfChannels = nbOfChannels
mylogger = qi.Logger("data")
mylogger.info("It works !" + str(nbOfChannels))
class MyClass(GeneratedClass):
def __init__(self):
GeneratedClass.__init__(self, False)
self.audio = AudioModule()
def onLoad(self):
self.serviceId = self.session().registerService("AudioModule", self.audio)
pass
def onUnload(self):
if self.serviceId != -1:
self.session().unregisterService(self.serviceId)
self.serviceId = -1
pass
def onInput_onStart(self):
self.audio.begin_stream()
self.onInput_onStop()
pass
def onInput_onStop(self):
self.audio.end_stream()
self.onUnload
self.onStopped()
pass

It appears you are subscribing to the audio from a Choregraphe box. I'm not sure it is supposed to work.
But in this configuration the Python code is executed from within the same process as the ALAudioDevice service. So probably you should name your callback "process" instead of "processRemote".
Otherwise, you can still do this from a separate Python script.

Python DBus remove signal match (remove signal receiver)

Community!
I have a little Problem with the Python-DBus-API. I create a signal-receiver and it does its job. However if I try to remove the signal, it won't be removed and the signalhandler (sigHandler) is happily called every time the signal matches.
class A(threading.Thread)
bus = None
mainloop = None
systemBusMainLoop = None
signalReceiver = None
def __init__(self,dbusMainLoop):
log("Hello.")
super(A, self).__init__()
gobject.threads_init()
self.mainloop = gobject.MainLoop()
self.systemBusMainLoop = dbusMainLoop
self.bus = dbus.SystemBus(mainloop=dbusMainLoop)
self.signalReceiver = self.bus.add_signal_receiver(self.sigHandler,
bus_name="org.bluez",
dbus_interface="org.freedesktop.DBus.Properties",
signal_name="PropertiesChanged",
path_keyword="path")
def run(self):
self.mainloop.run()
log("Running.")
def end(self):
log("Shutting down...")
self.bus.remove_signal_receiver(self.sigHandler,
self.signalReceiver,
dbus_interface="org.freedesktop.DBus.Properties")
#self.signalReceiver.remove() #tried this also
if (self.mainloop):
self.mainloop.quit()
del self.signalReceiver
log("Bye.")
def sigHandler(self, interface, changed, invalidated, path)
print interface
print changed
print invalidated
print path
Called:
dbusA = A(dbusMainLoop=dbus.mainloop.glib.DBusGMainLoop())
dbusA.run()
#doing something unrelated
dbusA.end() #remove the Signal
del dbusA
Do i miss something? Why ist my sigHandler not removed (or why is my match not removed).
Thanks in advance!

The line you commented out (self.signalReceiver.remove()) works.
add_signal_receiver() returns a SignalMatch instance on which you can call remove() method to unregister your callback.
The problem in your code snippet is that dbusA.run() is a blocking call because it calls self.mainloop.run() which itself is a blocking call.
This said, program execution will never reach dbusA.end(), thus signal unregistering and quitting the run loop will not happen. You can however call this method from a different thread, or from any event handler of your dbus connection.

Python Queues memory leaks when called inside thread

I have python TCP client and need to send media(.mpg) file in a loop to a 'C' TCP server.
I have following code, where in separate thread I am reading the 10K blocks of file and sending it and doing it all over again in loop, I think it is because of my implementation of thread module, or tcp send. I am using Queues to print the logs on my GUI ( Tkinter ) but after some times it goes out of memory..
UPDATE 1 - Added more code as requested
Thread class "Sendmpgthread" used to create thread to send data
.
.
def __init__ ( self, otherparams,MainGUI):
.
.
self.MainGUI = MainGUI
self.lock = threading.Lock()
Thread.__init__(self)
#This is the one causing leak, this is called inside loop
def pushlog(self,msg):
self.MainGUI.queuelog.put(msg)
def send(self, mysocket, block):
size = len(block)
pos = 0;
while size > 0:
try:
curpos = mysocket.send(block[pos:])
except socket.timeout, msg:
if self.over:
self.pushlog(Exit Send)
return False
except socket.error, msg:
print 'Exception'
return False
pos = pos + curpos
size = size - curpos
return True
def run(self):
media_file = None
mysocket = None
try:
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.connect((self.ip, string.atoi(self.port)))
media_file = open(self.file, 'rb')
while not self.over:
chunk = media_file.read(10000)
if not chunk: # EOF Reset it
print 'resetting stream'
media_file.seek(0, 0)
continue
if not self.send(mysocket, chunk): # If some error or thread is killed
break;
#disabling this solves the issue
self.pushlog('print how much data sent')
except socket.error, msg:
print 'print exception'
except Exception, msg:
print 'print exception'
try:
if media_file is not None:
media_file.close()
media_file = None
if mysocket is not None:
mysocket.close()
mysocket = None
finally:
print 'some cleaning'
def kill(self):
self.over = True
I figured out that it is because of wrong implementation of Queue as commenting that piece resolves the issue
UPDATE 2 - MainGUI class which is called from above Thread class
class MainGUI(Frame):
def __init__(self, other args):
#some code
.
.
#from the above thread class used to send data
self.send_mpg_status = Sendmpgthread(params)
self.send_mpg_status.start()
self.after(100, self.updatelog)
self.queuelog = Queue.Queue()
def updatelog(self):
try:
msg = self.queuelog.get_nowait()
while msg is not None:
self.printlog(msg)
msg = self.queuelog.get_nowait()
except Queue.Empty:
pass
if self.send_mpg_status: # only continue when sending
self.after(100, self.updatelog)
def printlog(self,msg):
#print in GUI

Since printlog is adding to a tkinter text control, the memory occupied by that control will grow with each message (it has to store all the log messages in order to display them).
Unless storing all the logs is critical, a common solution is to limit the maximum number of log lines displayed.
A naive implementation is to eliminate extra lines from the begining after the control reaches a maximum number of messages. Add a function to get the number of lines in the control and then, in printlog something similar to:
while getnumlines(self.edit) > self.maxloglines:
self.edit.delete('1.0', '1.end')
(above code not tested)
update: some general guidelines
Keep in mind that what might look like a memory leak does not always mean that a function is wrong, or that the memory is no longer accessible. Many times there is missing cleanup code for a container that is accumulating elements.
A basic general approach for this kind of problems:
form an opinion on what part of the code might be causing the problem
check it by commenting that code out (or keep commenting code until you find a candidate)
look for containers in the responsible code, add code to print their size
decide what elements can be safely removed from that container, and when to do it
test the result

I can't see anything obviously wrong with your code snippet.
To reduce memory usage a bit under Python 2.7, I'd use buffer(block, pos) instead of block[pos:]. Also I'd use mysocket.sendall(block) instead of your send method.
If the ideas above don't solve your problem, then the bug is most probably elsewhere in your code. Could you please post the shortest possible version of the full Python script which still grows out-of-memory (http://sscce.org/)? That increases your change of getting useful help.

Out of memory errors are indicative of data being generated but not consumed or released. Looking through your code I would guess these two areas:
Messages are being pushed onto a Queue.Queue() instance in the pushlog method. Are they being consumed?
The MainGui printlog method may be writing text somewhere. eg. Is it continually writing to some kind of GUI widget without any pruning of messages?
From the code you've posted, here's what I would try:
Put a print statement in updatelog. If this is not being continually called for some reason such as a failed after() call, then the queuelog will continue to grow without bound.
If updatelog is continually being called, then turn your focus to printlog. Comment the contents of this function to see if out of memory errors still occur. If they don't, then something in printlog may be holding on to the logged data, you'll need to dig deeper to find out what.
Apart from this, the code could be cleaned up a bit. self.queuelog is not created until after the thread is started which gives rise to a race condition where the thread may try to write into the queue before it has been created. Creation of queuelog should be moved to somewhere before the thread is started.
updatelog could also be refactored to remove redundancy:
def updatelog(self):
try:
while True:
msg = self.queuelog.get_nowait()
self.printlog(msg)
except Queue.Empty:
pass
And I assume the the kill function is called from the GUI thread. To avoid thread race conditions, the self.over should be a thread safe variable such as a threading.Event object.
def __init__(...):
self.over = threading.Event()
def kill(self):
self.over.set()

There is no data piling up in your TCP sending loop.
Memory error is probably caused by logging queue, as you have not posted complete code try using following class for logging:
from threading import Thread, Event, Lock
from time import sleep, time as now
class LogRecord(object):
__slots__ = ["txt", "params"]
def __init__(self, txt, params):
self.txt, self.params = txt, params
class AsyncLog(Thread):
DEBUGGING_EMULATE_SLOW_IO = True
def __init__(self, queue_max_size=15, queue_min_size=5):
Thread.__init__(self)
self.queue_max_size, self.queue_min_size = queue_max_size, queue_min_size
self._queuelock = Lock()
self._queue = [] # protected by _queuelock
self._discarded_count = 0 # protected by _queuelock
self._pushed_event = Event()
self.setDaemon(True)
self.start()
def log(self, message, **params):
with self._queuelock:
self._queue.append(LogRecord(message, params))
if len(self._queue) > self.queue_max_size:
# empty the queue:
self._discarded_count += len(self._queue) - self.queue_min_size
del self._queue[self.queue_min_size:] # empty the queue instead of creating new list (= [])
self._pushed_event.set()
def run(self):
while 1: # no reason for exit condition here
logs, discarded_count = None, 0
with self._queuelock:
if len(self._queue) > 0:
# select buffered messages for printing, releasing lock ASAP
logs = self._queue[:]
del self._queue[:]
self._pushed_event.clear()
discarded_count = self._discarded_count
self._discarded_count = 0
if not logs:
self._pushed_event.wait()
self._pushed_event.clear()
continue
else:
# print logs
if discarded_count:
print ".. {0} log records missing ..".format(discarded_count)
for log_record in logs:
self.write_line(log_record)
if self.DEBUGGING_EMULATE_SLOW_IO:
sleep(0.5)
def write_line(self, log_record):
print log_record.txt, " ".join(["{0}={1}".format(name, value) for name, value in log_record.params.items()])
if __name__ == "__main__":
class MainGUI:
def __init__(self):
self._async_log = AsyncLog()
self.log = self._async_log.log # stored as bound method
def do_this_test(self):
print "I am about to log 100 times per sec, while text output frequency is 2Hz (twice per second)"
def log_100_records_in_one_second(itteration_index):
for i in xrange(100):
self.log("something happened", timestamp=now(), session=3.1415, itteration=itteration_index)
sleep(0.01)
for iter_index in range(3):
log_100_records_in_one_second(iter_index)
test = MainGUI()
test.do_this_test()
I have noticed that you do not sleep() anywhere in the sending loop, this means data is read as fast as it can and is sent as fast as it can. Note that this is not desirable behavior when playing media files - container time-stamps are there to dictate data-rate.

libvlc and dbus interface

I'm trying a to create a basic media player using libvlc which will be controlled through dbus. I'm using the gtk and libvlc bindings for python. The code is based on the official example from the vlc website
The only thing I modified is to add the dbus interface to the vlc instance
# Create a single vlc.Instance() to be shared by (possible) multiple players.
instance = vlc.Instance()
print vlc.libvlc_add_intf(instance, "dbus"); // this is what i added. // returns 0 which is ok
All is well, the demo works and plays any video files. but for some reason the dbus control module doesn't work (I can't believe I just said the dreaded "doesn't work" words):
I already have the working client dbus code which binds to the MPRIS 2 interface. I can control a normal instance of a VLC media player - that works just fine, but with the above example nothing happens. The dbus control module is loaded properly, since libvlc_add_intf doesn't return an error and i can see the MPRIS 2 service in D-Feet (org.mpris.MediaPlayer2.vlc).
Even in D-Feet, trying to call any of the methods of the dbus vlc object returns no error but nothing happens.
Do I need to configure something else in order to make the dbus module control the libvlc player?
Thanks
UPDATE
It seems that creating the vlc Instance and setting a higher verbosity, shows that the DBus calls are received but they have no effect whatsoever on the player itself.
Also, adding the RC interface to the instance instead of DBus, has some problems too: When I run the example from the command line it drops me to the RC interface console where i can type the control commands, but it has the same behaviour as DBus - nothing happens, no error, nada, absolutely nothing. It ignores the commands completely.
Any thoughts?
UPDATE 2
Here is the code that uses libvlc to create a basic player:
from dbus.mainloop.glib import DBusGMainLoop
import gtk
import gobject
import sys
import vlc
from gettext import gettext as _
# Create a single vlc.Instance() to be shared by (possible) multiple players.
instance = vlc.Instance("--one-instance --verbose 2")
class VLCWidget(gtk.DrawingArea):
"""Simple VLC widget.
Its player can be controlled through the 'player' attribute, which
is a vlc.MediaPlayer() instance.
"""
def __init__(self, *p):
gtk.DrawingArea.__init__(self)
self.player = instance.media_player_new()
def handle_embed(*args):
if sys.platform == 'win32':
self.player.set_hwnd(self.window.handle)
else:
self.player.set_xwindow(self.window.xid)
return True
self.connect("map", handle_embed)
self.set_size_request(640, 480)
class VideoPlayer:
"""Example simple video player.
"""
def __init__(self):
self.vlc = VLCWidget()
def main(self, fname):
self.vlc.player.set_media(instance.media_new(fname))
w = gtk.Window()
w.add(self.vlc)
w.show_all()
w.connect("destroy", gtk.main_quit)
self.vlc.player.play()
DBusGMainLoop(set_as_default = True)
gtk.gdk.threads_init()
gobject.MainLoop().run()
if __name__ == '__main__':
if not sys.argv[1:]:
print "You must provide at least 1 movie filename"
sys.exit(1)
if len(sys.argv[1:]) == 1:
# Only 1 file. Simple interface
p=VideoPlayer()
p.main(sys.argv[1])
the script can be run from the command line like:
python example_vlc.py file.avi
The client code which connects to the vlc dbus object is too long to post so instead pretend that i'm using D-Feet to get the bus connection and post messages to it.
Once the example is running, i can see the players dbus interface in d-feet, but i am unable to control it. Is there anything else that i should add to the code above to make it work?

I can't see your implementation of your event loop, so it's hard to tell what might be causing commands to not be recognized or to be dropped. Is it possible your threads are losing the stacktrace information and are actually throwing exceptions?
You might get more responses if you added either a psuedo-code version of your event loop and DBus command parsing or a simplified version?

The working programs found on nullege.com use ctypes. One which acted as a server used rpyc. Ignoring that one.
The advantages of ctypes over dbus is a huge speed advantage (calling the C library code, not interacting using python) as well as not requiring the library to implement the dbus interface.
Didn't find any examples using gtk or dbus ;-(
Notable examples
PyNuvo vlc.py
Milonga Tango DJing program
Using dbus / gtk
dbus uses gobject mainloop, not gtk mainloop. Totally different beasts. Don't cross the streams! Some fixes:
Don't need this. Threads are evil.
gtk.gdk.threads_init()
gtk.main_quit() shouldn't work when using gobject Mainloop. gobject mainloop can't live within ur class.
if __name__ == '__main__':
loop = gobject.MainLoop()
loop.run()
Pass in loop into ur class. Then call to quit the app
loop.quit()
dbus (notify) / gtk working example
Not going to write ur vlc app for u. But here is a working example of using dbus / gtk. Just adapt to vlc. Assumed u took my advise on gtk above. As u know any instance of DesktopNotify must be called while using gobject.Mainloop . But u can place it anywhere within ur main class.
desktop_notify.py
from __future__ import print_function
import gobject
import time, dbus
from dbus.exceptions import DBusException
from dbus.mainloop.glib import DBusGMainLoop
class DesktopNotify(object):
""" Notify-OSD ubuntu's implementation has a 20 message limit. U've been warned. When queue is full, delete old message before adding new messages."""
#Static variables
dbus_loop = None
dbus_proxy = None
dbus_interface = None
loop = None
#property
def dbus_name(self):
return ("org.freedesktop.Notifications")
#property
def dbus_path(self):
return ("/org/freedesktop/Notifications")
#property
def dbus_interface(self):
return self.dbus_name
def __init__(self, strInit="initializing passive notification messaging")
strProxyInterface = "<class 'dbus.proxies.Interface'>"
""" Reinitializing dbus when making a 2nd class instance would be bad"""
if str(type(DesktopNotify.dbus_interface)) != strProxyInterface:
DesktopNotify.dbus_loop = DBusGMainLoop(set_as_default=True)
bus = dbus.SessionBus(mainloop=DesktopNotify.dbus_loop)
DesktopNotify.dbus_proxy = bus.get_object(self.dbus_name, self.dbus_path)
DesktopNotify.dbus_interface = dbus.Interface(DesktopNotify.dbus_proxy, self.dbus_interface )
DesktopNotify.dbus_proxy.connect_to_signal("NotificationClosed", self.handle_closed)
def handle_closed(self, *arg, **kwargs):
""" Notification closed by user or by code. Print message or not"""
lngNotificationId = int(arg[0])
lngReason = int(arg[1])
def pop(self, lngID):
""" ID stored in database, but i'm going to skip this and keep it simple"""
try:
DesktopNotify.dbus_interface.CloseNotification(lngID)
except DBusException as why:
print(self.__class__.__name__ + ".pop probably no message with id, lngID, why)
finally:
pass
def push(self, strMsgTitle, strMsg, dictField):
""" Create a new passive notification (took out retrying and handling full queues)"""
now = time.localtime( time.time() )
strMsgTime = strMsg + " " + time.asctime(now)
del now
strMsgTime = strMsgTime % dictField
app_name="[your app name]"
app_icon = ''
actions = ''
hint = ''
expire_timeout = 10000 #Use seconds * 1000
summary = strMsgTitle
body = strMsgTime
lngNotificationID = None
try:
lngNotificationID = DesktopNotify.dbus_interfacec.Notify(app_name, 0, app_icon, summary, body, actions, hint, expire_timeout)
except DBusException as why:
#Excellent spot to delete oldest notification and then retry
print(self.__class__.__name__ + ".push Being lazy. Posting passive notification was unsuccessful.", why)
finally:
#Excellent spot to add to database upon success
pass