I've added a module to mopidy core that uses gobject.timeout_add() for a repeating function. It works fine when running normally, however when running tests it appears that the handler function never gets called.
The module under test has a method that starts a regular process that emits events roughly every half a second. The test calls the method, then call time.sleep(2). The events from the timer function don't occur, and neither does some debug logging in the timer function. Other events and debug logging (outside the timer function) work fine.
What do I need to do to get gobject.timeout_add() to work in nose tests? Or do I need to use something other than time.sleep() in the test in order to allow the other code to run? It's calling gobject.threads_init() in the test setup, is there anything else I need to do?
You need to be running an event loop. As the documentation for g_timeout_add explains, that function (and other similar functions in glib) will create a new timeout source which is then attached to the event loop (well, the GMainContext, but you don't really need to worry about that). It is not the equivalent of spawning a new thread and having it sleep for whatever you specified as the duration of the timeout, which seems to be the behavior you're expecting—using the main loop allows everything to happen in a single thread.
Related
I want to run a timer in a tornado based web app such that the it runs in background and is non blocking.
Once the timer finishes, a specific task has to be called so it is very important that the timer completes exactly on time.
What should be the ideal way to do it ?
I read Tornado IOLoop.spawn_callback in the documentation but I am not very clear that it would behave correctly.
I don't quite understand the statement in the doc
Unlike all other callback-related methods on IOLoop, spawn_callback does not associate the callback with its caller’s stack_context
If you want to run a function after a specific time, you can use IOLoop.call_later. Use it like this:
IOLoop.current().call_later(5, my_func) # will call my_func 5 seconds later
def my_func():
# do something
IOLoop.spawn_callback is used for running a callback/function in the next iteration of the IOLoop, that is - almost instantly. You can't add a time out to spawn_callback. Since you want to schedule a callback after a timeout, IOLoop.call_later is what you need.
In your comment you asked
Why according to you IOLoop.spawn_callback is not to be used?
Well, I never said to not use it. You can use it if you need it. In this case, you don't.
So, when do you need it? When you'll need to run a callback almost instantly, without a timeout, that's when you can use spawn_callback. But even then, there's IOLoop.add_callback which is used much more widely than spawn_callback.
I'm seeing an issue in my test suite in trial where everything works fine until I get a timeout. If a test fails due to a timeout, the tearDown function never gets called, leaving the reactor unclean which in turn causes the rest of the tests to fail. I believe tearDown should be called after a timeout, does anyone know why this might happen?
You are correct that tearDown() should be called regardless of what happens in your test. From the documentation for tearDown():
This is called even if the test method raised an exception
However, there is a catch. From the same documentation:
This method will only be called if the setUp() succeeds, regardless of the outcome of the test method.
So it sounds like you perhaps start the reactor in setUp() and when it times out, this is preventing your tearDown() from running - the idea being that whatever you were trying to "set up" in setUp() was not successfully set up, so you do not want to try to tear it down. However, it would be hard to diagnose with certainty unless you provide the code of your setUp and tearDown methods, along with the code of any relevant tests.
It's rather strange because on my box, the teardown executes even if a timeout occurs. The tests should stop running if the reactor is not in a clean state, unless you use the --unclean-warnings flag. Does the test runner stop after the timeout for you? What version of Python and Twisted are you running?
As a side note, if you need to run a unique teardown for a specific test function, there's a very convenient addCleanup() callback. It comes in handy if you need to cancel callback, LoopingCall, or callLater functions so that the reactor isn't in a dirty state. addCleanup returns a Deferred so you can just chain callbacks that perform an adhoc teardown. It might be a good option to try if the class teardown isn't working for you.
PS
I've been so used to writing "well behaved" Twisted code, I don't even recall how to get into an unclean reactor state :D I swear I'm not bragging. Could you provide me a brief summary of what you're doing so that I could test it out on my end?
I found the problem, I'll put this here in case it's helpful to anyone else in the future.
I was returning a deferred from the test that had already been called (as in, deferred.callback had been called), but it still had an unfinished callback chain. From what I can see in the trial code here https://github.com/twisted/twisted/blob/twisted-16.5.0/src/twisted/trial/_asynctest.py#L92, the reactor is crashed when this happens, which explains why the tearDown doesn't get called. The solution for me was to return a deferred from the offending tests that does not have a callback chain that lives for a long time (it's callbacks do not return deferreds themselves).
I'm using Tornado as a coroutine engine for a periodic process, where the repeating coroutine calls ioloop.call_later() on itself at the end of each execution. I'm now trying to drive this with unit tests (using Tornado's gen.test) where I'm mocking the ioloop's time with a local variable t:
DUT.ioloop.time = mock.Mock(side_effect= lambda: t)
(DUT <==> Device Under Test)
Then in the test, I manually increment t, and yield gen.moment to kick the ioloop. The idea is to trigger the repeating coroutine after various intervals so I can verify its behaviour.
But the coroutine doesn't always trigger - or perhaps it yields back to the testing code before completing execution, causing failures.
I think should be using stop() and wait() to synchronise the test code, but I can't see concretely how to use them in this situation. And how does this whole testing strategy work if the DUT runs in its own ioloop?
In general, using yield gen.moment to trigger specific events is dicey; there are no guarantees about how many "moments" you must wait, or in what order the triggered events occur. It's better to make sure that the function being tested has some effect that can be asynchronously waited for (if it doesn't have such an effect naturally, you can use a tornado.locks.Condition).
There are also subtleties to patching IOLoop.time. I think it will work with the default Tornado IOLoops (where it is possible without the use of mock: pass a time_func argument when constructing the loop), but it won't have the desired effect with e.g. AsyncIOLoop.
I don't think you want to use AsyncTestCase.stop and .wait, but it's not clear how your test is set up.
So I have this library that I use and within one of my functions I call a function from that library, which happens to take a really long time. Now, at the same time I have another thread running where I check for different conditions, what I want is that if a condition is met, I want to cancel the execution of the library function.
Right now I'm checking the conditions at the start of the function, but if the conditions happen to change while the library function is running, I don't need its results, and want to return from it.
Basically this is what I have now.
def my_function():
if condition_checker.condition_met():
return
library.long_running_function()
Is there a way to run the condition check every second or so and return from my_function when the condition is met?
I've thought about decorators, coroutines, I'm using 2.7 but if this can only be done in 3.x I'd consider switching, it's just that I can't figure out how.
You cannot terminate a thread. Either the library supports cancellation by design, where it internally would have to check for a condition every once in a while to abort if requested, or you have to wait for it to finish.
What you can do is call the library in a subprocess rather than a thread, since processes can be terminated through signals. Python's multiprocessing module provides a threading-like API for spawning forks and handling IPC, including synchronization.
Or spawn a separate subprocess via subprocess.Popen if forking is too heavy on your resources (e.g. memory footprint through copying of the parent process).
I can't think of any other way, unfortunately.
Generally, I think you want to run your long_running_function in a separate thread, and have it occasionally report its information to the main thread.
This post gives a similar example within a wxpython program.
Presuming you are doing this outside of wxpython, you should be able to replace the wx.CallAfter and wx.Publisher with threading.Thread and PubSub.
It would look something like this:
import threading
import time
def myfunction():
# subscribe to the long_running_function
while True:
# subscribe to the long_running_function and get the published data
if condition_met:
# publish a stop command
break
time.sleep(1)
def long_running_function():
for loop in loops:
# subscribe to main thread and check for stop command, if so, break
# do an iteration
# publish some data
threading.Thread(group=None, target=long_running_function, args=()) # launches your long_running_function but doesn't block flow
myfunction()
I haven't used pubsub a ton so I can't quickly whip up the code but it should get you there.
As an alternative, do you know the stop criteria before you launch the long_running_function? If so, you can just pass it as an argument and check whether it is met internally.
I'm writing a multi-threaded application that utilizes QThreads. I know that, in order to start a thread, I need to override the run() method and call that method using the thread.start() somewhere (in my case in my GUI thread).
I was wondering, however, is it required to call the .wait() method anywhere and also am I supposed to call the .quit() once the thread finishes, or is this done automatically?
I am using PySide.
Thanks
Both answers depend on what your code is doing and what you expect from the thread.
If your logic which uses the thread needs to wait synchronously for the moment QThread finishes, then yes, you need to call wait(). However such requirement is a sign of sloppy threading model, except very specific situations like application startup and shutdown. Usage of QThread::wait() suggests creeping sequential operation, which means that you are effectively not using threads concurrently.
quit() exits QThread-internal event loop, which is not mandatory to use. A long-running thread (as opposed to one-task worker) must have an event loop of some sort - this is a generic statement, not specific to QThread. You either do it yourself (in form of some while(keepRunning) { } cycle) or use Qt-provided event loop, which you fire off by calling exec() in your run() method. The former implementation is finishable by you, because you did provide the keepRunning condition. The Qt-provided implementation is hidden from you and here goes the quit() call - which internally does nothing more than setting some sort of similar flag inside Qt.