I have a class Indexer which is instantiated from the main thread, the instance of this class is stored in a variable, say, indexer. watchdog.observers.Observer() watches directories for changes and these happen in another thread. I tried passing this indexer variable from main thread through my handler Vigilante which was passed to ob.schedule(Vigilante(indexer)) alongside some other variables from main thread. I can't access the indexer variable in the Vigilante class, because of being in different threads. I know I could use a Queue but I don't know how I'd pass the Queue to watchdog's thread.
Here is the code from main thread:
if watch:
import watchdog.observers
from .utils import vigilante
class callbacks:
def __init__(self):
pass
#staticmethod
def build(filename, response):
return _build(filename, response)
#staticmethod
def renderer(src, mode):
return render(src, mode)
handler = vigilante.Vigilante(_filter, ignore, Indexer, callbacks, Mode)
path_to_watch = os.path.normpath(os.path.join(workspace, '..'))
ob = watchdog.observers.Observer()
ob.schedule(handler, path=path_to_watch, recursive=True)
ob.start()
import time
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
ob.stop()
Indexer.close()
ob.join()
The Indexer class is meant to write to a database from another part of the code where the Indexer was instantiated.
Here is the code from Vigilante class running in watchdog's thread:
class Vigilante(PatternMatchingEventHandler):
"""Helps to watch files, directories for changes"""
def __init__(self, pattern, ignore, indexer, callback, mode):
pattern.append("*.yml")
self.Callback = callback
self.Mode = mode
self.Indexer = indexer
super(Vigilante, self).__init__(patterns=pattern, ignore_directories=ignore)
def vigil(self, event):
print(event.src_path, 'modified')
IndexReader = self.Indexer.get_index_on(event.src_path)
dep = IndexReader.read_index()
print(dep.next(), 'dependency')
feedout = self.Callback.build(
os.path.basename(event.src_path)
,self.Callback.renderer(event.src_path, self.Mode.FILE_MODE)
)
def on_modified(self, event):
self.vigil(event)
def on_created(self, event):
self.vigil(event)
All I need is a way to pass those variables from the main thread to watchdog's thread, through the Vigilante class
I finally found a way to do it without crossing threads too much as before, with an idea derived from #EvertW 's answer. I passed a Queue from main thread to the Vigilante class which was in another thread, so every modified file would be put in the Queue and then, from the main thread, I got the file modified from the queue, read from the Indexer database, and every other task which the Vigilante.vigil method needed to perform was moved to the main thread since those tasks depends on the modified file and what is read from the Indexer database.
This error disappeared:
SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 9788 and this is thread id 4288.
Here is a snippet from what I did:
....
q = Queue.LifoQueue(10)
handler = vigilante.Vigilante(q, _filter, ignore)
path_to_watch = os.path.normpath(os.path.join(workspace, '..'))
ob = watchdog.observers.Observer()
ob.schedule(handler, path=path_to_watch, recursive=True)
ob.start()
import time
try:
while True:
if not q.empty():
modified = q.get()
IndexReader = Indexer.get_index_on(modified)
deps = IndexReader.read_index()
print(deps.next(), 'dependency')
# TODO
else:
print('q is empty')
time.sleep(1)
except KeyboardInterrupt:
ob.stop()
Indexer.close()
ob.join()
Vigilante class:
class Vigilante(PatternMatchingEventHandler):
"""Helps to watch files, directories for changes"""
def __init__(self, q, pattern, ignore):
self.q = q
super(Vigilante, self).__init__(
patterns=pattern,
ignore_patterns=ignore,
ignore_directories=True
)
def vigil(self, event):
print(event.src_path, 'modified')
self.q.put(event.src_path)
def on_modified(self, event):
self.vigil(event)
def on_created(self, event):
self.vigil(event)
....
PS: A word of advice : my word of advice to whoever comes across this kind of problem with threading in watchdog is; "Don't trust the watchdog's thread to do tasks on modified files, just get the modified files out and do whatever you like with them, except the task is a simple one."
You could try the Observer pattern (no pun intended), i.e. let the Observer class have a list of listeners that it will inform of any changes it sees. Then let the indexer announce its interest to the Observer.
In my example, the Observer expects subscribers to be callables that receive the changes. Then you can do:
from queue import Queue
class Observable:
def __init__(self):
self.listeners = []
def subscribe(listener):
self.listeners.append(listener)
def onNotify(change):
for listener in self.listeners:
listener(change)
class Indexer(Thread):
def __init__(self, observer):
Thread.__init__(self)
self.q = Queue()
observer.subscribe(self.q.put)
def run(self):
while True:
change = self.q.get()
Because the standard Queue is completely thread-safe, this will work fine.
Related
Is it possible to detach a specific event after attaching it to a COM object?
For example, how to deregister the ClassOfHandlers in the following snippet:
from win32com.client import WithEvents
# ...
class ClassOfHandlers():
def OnStart(self):
print("Start observed")
class AnotherClassOfHandlers():
def OnStart(self):
print("Start observed from another")
WithEvents(client, ClassOfHandlers)
# ...
WithEvents(client, AnotherClassOfHandlers)
# ...
# Deregister `ClassOfHandlers`
As a variation on the OP's answer, which avoids a static member variable, it is worth remembering that WithEvents() returns an instance of the handler class.
from win32com.client import WithEvents
def MyOnStart():
print("Start observed")
def MySecondOnStart():
print("Start observed from another")
class ClassOfHandlers():
def __init__(self):
self._fn = MyOnStart
def setStartFunction(self,fn):
self._fn = fn
def OnStart(self):
self._fn()
handler = WithEvents(client, ClassOfHandlers)
#then later
handler.setStartFunction(MySecondOnStart)
Hence you can re-use the handler class for a different client.
Alternatively you could try opening an issue here and maybe the developers can advise on whether they expose the IConnectionPoint::Unadvise() function which would be needed behind the scenes to switch event handlers (I think).
Edit
Based on DS_London's answer we could benefit from WithEvents return, thus the combined solution would look like
from win32com.client import WithEvents
def MyOnStart():
print("Start observed")
def MySecondOnStart():
print("Start observed from another")
class ClassOfHandlers():
def __init__(self):
self._onStarts = []
# self._onStops = []
# ... add a list of functions for each event type
# the following 3 methods are implemented for each event type
def attachStart(self, fn):
self._onStarts.append(fn)
def detachStart(self, fn):
self._onStarts.remove(fn)
def OnStart(self):
for fn in self._onStarts:
fn()
# Always at the beginning
handler = WithEvents(client, ClassOfHandlers)
handler.attachStart(MyOnStart)
# ...
handler.attachStart(MySecondOnStart)
# ...
handler.detachStart(MyOnStart)
Limitation
If support for multiple clients is needed and thus threading is used, this edit won't work, and it would be needed to use the original answer's approach.
The cause: one needs to pass the ClassOfHandlers to the thread runnable*, however the thread runnable would PumpWaitingMessages() till interrupted, thus it won't be able to return the client handler back, preventing us from being able to detach/attach further functions while waiting for messages.
* PumpWaitingMessages() requires that it runs on the same thread that connected the ClassOfHandlers to the client, thus we can't create the client handler out of the thread then send it into the thread runnable.
Following is a snippet that shows this scenario:
def threadRunnable(client, eventsClass, controller):
pythoncom.CoInitializeEx(pythoncom.COINIT_MULTITHREADED)
# Connect the custom events
# The connection needs to be done inside the same thread for PumpWaitingMessages
handler = WithEvents(client, eventsClass)
if controller == None:
print("no control was provided")
controller = { "sleep_time": 1, "running_flag": True}
# With this while we won't be able to return the handler
while controller["running_flag"]:
pythoncom.PumpWaitingMessages()
time.sleep(controller["sleep_time"])
pythoncom.CoUninitialize()
def connectEvents(client, eventsClass, controller=None, runnable=threadRunnable):
flusher = Thread(target=runnable, args=(client,eventsClass,controller))
flusher.daemon = True
flusher.start()
def main():
controller = { "sleep_time": 1, "running_flag": True}
connectEvents(client, ClassOfHandlers, controller)
Original
I'm now able to achieve the desired behavior, by attaching a single permanent observer class and managing the events myself.
For example:
from win32com.client import WithEvents
# ...
class ClassOfHandlers():
OnStarts = []
def OnStart(self):
for handler in ClassOfHandlers.OnStarts:
handler()
def MyOnStart():
print("Start observed")
def MySecondOnStart():
print("Start observed from another")
# Always at the beginning
WithEvents(client, ClassOfHandlers)
ClassOfHandlers.OnStarts.append(MyOnStart)
# ...
ClassOfHandlers.OnStarts.append(MySecondOnStart)
# ...
ClassOfHandlers.OnStarts.remove(MyOnStart)
Hint:
The class variable OnStarts shall be changed to an instance variable if the class represents an instantiable COM object, to allow having an instance of the ClassOfHandlers (each instance having a different handler list) for each instantiated COM object.
One also needs to ensure that WithEvents is called only once for each COM object instance.
I need to pause and resume thread, which continuously executes some task. Execution begins when start() is called, it should not be interrupted and must continue from the point when pause() is called.
How can I do this?
Please remember that using threads in Python will not grant you a parallel processing, except for the case of IO blocking operations. For more information on this, take a look at this and this
You cannot pause a Thread arbitrarily in Python (please keep that in mind before reading further). I am neither sure you have a way to do that at an OS level (e.g. by using pure-C). What you can do is allow the thread to be paused at specific points you consider beforehand. I will give you an example:
class MyThread(threading.Thread):
def __init__(self, *args, **kwargs):
super(MyThread, self).__init__(*args, **kwargs)
self._event = threading.Event()
def run(self):
while True:
self.foo() # please, implement this.
self._event.wait()
self.bar() # please, implement this.
self._event.wait()
self.baz() # please, implement this.
self._event.wait()
def pause(self):
self._event.clear()
def resume(self):
self._event.set()
This approach will work but:
Threading is usually a bad idea, based on the links I gave you.
You have to code the run method by yourself, with this approach. This is because you need to have control over the exact points you'd like to check for pause, and this implies accessing the Thread object (perhaps you'd like to create an additional method instead of calling self._event.wait()).
The former point makes clear that you cannot pause arbitrarily, but just when you specified you could pause. Avoid having long operations between pause points.
Edit I did not test this one, but perhaps this will work without so much subclassing if you need more than one thread like this:
class MyPausableThread(threading.Thread):
def __init__(self, group=None, target=None, name=None, args=(), kwargs={}):
self._event = threading.Event()
if target:
args = (self,) + args
super(MyPausableThread, self).__init__(group, target, name, args, kwargs)
def pause(self):
self._event.clear()
def resume(self):
self._event.set()
def _wait_if_paused(self):
self._event.wait()
This should allow you to create a custom thread without more subclassing, by calling MyPausableThread(target=myfunc).start(), and your callable's first parameter will receive the thread object, from which you can call self._wait_if_paused() when you need to pause-check.
Or even better, if you want to isolate the target from accessing the thread object:
class MyPausableThread(threading.Thread):
def __init__(self, group=None, target=None, name=None, args=(), kwargs={}):
self._event = threading.Event()
if target:
args = ((lambda: self._event.wait()),) + args
super(MyPausableThread, self).__init__(group, target, name, args, kwargs)
def pause(self):
self._event.clear()
def resume(self):
self._event.set()
And your target callable will receive in the first parameter a function that can be called like this: pause_checker() (provided the first param in the target callable is named pause_checker).
You can do this by attaching a trace function that causes all other threads to wait for a signal:
import sys
import threading
import contextlib
# needed to enable tracing
if not sys.gettrace():
sys.settrace(lambda *args: None)
def _thread_frames(thread):
for thread_id, frame in sys._current_frames().items():
if thread_id == thread.ident:
break
else:
raise ValueError("No thread found")
# walk up to the root
while frame:
yield frame
frame = frame.f_back
#contextlib.contextmanager
def thread_paused(thread):
""" Context manager that pauses a thread for its duration """
# signal for the thread to wait on
e = threading.Event()
for frame in _thread_frames(thread):
# attach a new temporary trace handler that pauses the thread
def new(frame, event, arg, old = frame.f_trace):
e.wait()
# call the old one, to keep debuggers working
if old is not None:
return old(frame, event, arg)
frame.f_trace = new
try:
yield
finally:
# wake the other thread
e.set()
Which you can use as:
import time
def run_after_delay(func, delay):
""" Simple helper spawning a thread that runs a function in the future """
def wrapped():
time.sleep(delay)
func()
threading.Thread(target=wrapped).start()
main_thread = threading.current_thread()
def interrupt():
with thread_paused(main_thread):
print("interrupting")
time.sleep(2)
print("done")
run_after_delay(interrupt, 1)
start = time.time()
def actual_time(): return time.time() - start
print("{:.1f} == {:.1f}".format(0.0, actual_time()))
time.sleep(0.5)
print("{:.1f} == {:.1f}".format(0.5, actual_time()))
time.sleep(2)
print("{:.1f} != {:.1f}".format(2.5, actual_time()))
Giving
0.0 0.0
0.5 0.5
interrupting
done
2.5 3.0
Note how the interrupt causes the sleep on the main thread to wait longer
You can do this using Process class from psutil library.
Example:
>>> import psutil
>>> pid = 7012
>>> p = psutil.Process(pid)
>>> p.suspend()
>>> p.resume()
See this answer: https://stackoverflow.com/a/14053933
Edit: This method will suspend the whole process, not only one thread. ( I don't delete this answer, so others can know this method won't work.)
while(int(any) < 2000):
sleep(20)
print(waiting any...)
I'm trying to create a subclass of threading.Thread whose methods are threaded. I'm using it for video, but I suspect that a working example will be generally useful for folks.
I realized here that I never instantiated a thread and never called the start() method, but I don't know where to call it from or how. I also want to save the thread handle so I can stop it if I receive a stop() signal.
import threading
class VideoThread(threading.Thread):
"""Thread class with a stop() method. The thread itself checks
regularly for the stopped() condition."""
def __init__(self, playlist=None):
super(VideoThread, self).__init__()
self._stop = threading.Event()
self._player_pgid_list = []
if playlist:
self.start_sequence(playlist)
def stop(self):
self._stop.set()
def stopped(self):
return self._stop.isSet()
def start_sequence(self, playlist):
if not isinstance(playlist, list):
raise ValueError("Expecting a list")
for video in playlist:
if not self.stopped():
self.__start_video__(video)
def __start_video__(self, video):
if not isinstance(video, dict):
raise ValueError("Expecting a dictionary of video data")
# start the video
# store the video pgid so we can kill it if we have to
# tight wait loop to check for stopped condition
# kill all video(s) if necessary using the stored pgids
The class works as far as it goes, but of course, none of the methods are actually threaded.
start_sequence() is public so I can start a threaded sequence of videos like this:
video = VideoThread()
video.start_sequence([films[1], films[3], films[2]])
Or when I instantiate the class like this:
video = VideoThread([films[1], films[3], films[2]])
Later, if I need to stop it, I can:
video.stop()
What am I missing?
You should rename the start_sequence method to run and delete the playlist parameter (use self.playlist instead). Also, delete those two last lines in __init__ method. I mean:
class VideoThread(threading.Thread):
def __init__(self, playlist=None):
super().__init__()
self._stop = threading.Event()
self._player_pgid_list = []
self.playlist = playlist
def run(self):
if not isinstance(self.playlist, list):
raise ValueError("Expecting a list")
for video in self.playlist:
if not self.stopped():
self.__start_video__(video)
...
Then, to use your class just do:
playlist = VideoThread(films)
playlist.start()
And you can stop it using:
playlist.stop()
Note that when you call .start, it invokes the run method in a separate thread of control, check the official documentation for more information.
I have the following code in python:
class gateWay:
def __init__(self):
self.var1 = []
self.var2 = {}
self.currentThread = None
def stateProcess(self, file):
# some irrelevant code
self.currentThread = saltGatWayThread(self, file).start()
return self.var1
def stopRunning(self):
self.currentThread.proc.stop()
In addition, here the source code of the saltGatWayThread:
class saltGatWayThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
# some irrelevant code
self.proc = src.proc.Process1()
In addition, I have the following code in src/proc/__init__.py:
class Process1:
def stop(self):
# code to stop operation
In the console, I notice that self.currentThread is null.
My purpose is to save the thread in local variable, when start it. If I get an abort request, I apply
stopRunning function. This function, would take the saved thread and will do "clean" exit (finish the process of the tread and exit).
Why can't I save the thread, and use the structure of it later on?
invoke currentThread = saltGatWayThread() and then call .start(). currentThread does not contains thread instance because starts() method always returns nothing according to the threading.py source code. See source of C:\Python27\Lib\threading.py
def start(self):
"""Start the thread's activity.
It must be called at most once per thread object. It arranges for the
object's run() method to be invoked in a separate thread of control.
This method will raise a RuntimeError if called more than once on the
same thread object.
"""
if not self.__initialized:
raise RuntimeError("thread.__init__() not called")
if self.__started.is_set():
raise RuntimeError("threads can only be started once")
if __debug__:
self._note("%s.start(): starting thread", self)
with _active_limbo_lock:
_limbo[self] = self
try:
_start_new_thread(self.__bootstrap, ())
except Exception:
with _active_limbo_lock:
del _limbo[self]
raise
self.__started.wait()
I have a Manager (main thread), that creates other Threads to handle various operations.
I would like my Manager to be notified when a Thread it created ends (when run() method execution is finished).
I know I could do it by checking the status of all my threads with the Thread.isActive() method, but polling sucks, so I wanted to have notifications.
I was thinking of giving a callback method to the Threads, and call this function at the end of the run() method:
class Manager():
...
MyThread(self.on_thread_finished).start() # How do I pass the callback
def on_thread_finished(self, data):
pass
...
class MyThread(Thread):
...
def run(self):
....
self.callback(data) # How do I call the callback?
...
Thanks!
The thread can't call the manager unless it has a reference to the manager. The easiest way for that to happen is for the manager to give it to the thread at instantiation.
class Manager(object):
def new_thread(self):
return MyThread(parent=self)
def on_thread_finished(self, thread, data):
print thread, data
class MyThread(Thread):
def __init__(self, parent=None):
self.parent = parent
super(MyThread, self).__init__()
def run(self):
# ...
self.parent and self.parent.on_thread_finished(self, 42)
mgr = Manager()
thread = mgr.new_thread()
thread.start()
If you want to be able to assign an arbitrary function or method as a callback, rather than storing a reference to the manager object, this becomes a bit problematic because of method wrappers and such. It's hard to design the callback so it gets a reference to both the manager and the thread, which is what you will want. I worked on that for a while and did not come up with anything I'd consider useful or elegant.
Anything wrong with doing it this way?
from threading import Thread
class Manager():
def Test(self):
MyThread(self.on_thread_finished).start()
def on_thread_finished(self, data):
print "on_thread_finished:", data
class MyThread(Thread):
def __init__(self, callback):
Thread.__init__(self)
self.callback = callback
def run(self):
data = "hello"
self.callback(data)
m = Manager()
m.Test() # prints "on_thread_finished: hello"
If you want the main thread to wait for children threads to finish execution, you are probably better off using some kind of synchronization mechanism. If simply being notified when one or more threads has finished executing, a Condition is enough:
import threading
class MyThread(threading.Thread):
def __init__(self, condition):
threading.Thread.__init__(self)
self.condition = condition
def run(self):
print "%s done" % threading.current_thread()
with self.condition:
self.condition.notify()
condition = threading.Condition()
condition.acquire()
thread = MyThread(condition)
thread.start()
condition.wait()
However, using a Queue is probably better, as it makes handling multiple worker threads a bit easier.