Am on a project using txrdq, am testing (using trial) for a case where a queued job may fail, trial marks the testcase as failed whenever it hits a failure in a errback ..
The errback is a normal behaviour, since a queued job may fail to launch, how to test this case using trial without failing the test ?
here's an example of the test case:
from twisted.trial.unittest import TestCase
from txrdq.rdq import ResizableDispatchQueue
from twisted.python.failure import Failure
class myTestCase(TestCase):
def aFailingJob(self, a):
return Failure("This is a failure")
def setUp(self):
self.queue = ResizableDispatchQueue(self.aFailingJob, 1)
def tearDown(self):
pass
def test_txrdq(self):
self.queue.put("Some argument", 1)
It seems likely that the exception is being logged, since the error handler just raises it. I'm not exactly sure what the error handling code in txrdq looks like, so this is just a guess, but I think it's a pretty good one based on your observations.
Trial fails any unit test that logs an exception, unless the test cleans that exception up after it's logged. Use TestCase.flushLoggedErrors(exceptionType) to deal with this:
def test_txrdq(self):
self.queue.put("Some argument", 1)
self.assertEqual(1, len(self.flushLoggedErrors(SomeException)))
Also notice that you should never do Failure("string"). This is analogous to raise "string". String exceptions are deprecated in Python since a looooong time ago. Always construct a Failure with an exception instance:
class JobError(Exception):
pass
def aFailingJob(self, a):
return Failure(JobError("This is a failure"))
This makes JobError the exception type you'd pass to flushLoggedErrors.
Make sure that you understand whether queue processing is synchronous or asynchronous. If it is synchronous, your test (with the flushLoggedErrors call added) is fine. If it is asynchronous, your error handler may not have run by the time your test method returns. In that case, you're not going to be testing anything useful, and the errors might be logged after the call to flush them (making the flush useless).
Finally, if you're not writing unit tests '''for''' txrdq, then you might not want to write tests like this. You can probably unit test txrdq-using code without using an actual txrdq. A normal Queue object (or perhaps another more specialized test double) will let you more precisely target the units in your application, making your tests faster, more reliable, and easier to debug.
This issue has now (finally!) been solved, by L. Daniel Burr. There's a new version (0.2.14) of txRDQ on PyPI.
By the way, in your test you should add from txrdq.job import Job, and then do something like this:
d = self.queue.put("Some argument", 1)
return self.assertFailure(d, Job)
Trial will make sure that d fails with a Job instance. There are a couple of new tests at the bottom of txrdq/test/test_rdq.py that illustrate this kind of assertion.
I'm sorry this problem caused so much head scratching for you - it was entirely my fault.
Sorry to see you're still having a problem. I don't know what's going on here, but I have been playing with it for over an hour trying to...
The queue.put method returns a Deferred. You can attach an errback to it to do the flush as #exarkun describes, and then return the Deferred from the test. I expected that to fix things (having read #exarkun's reply and gotten a comment from #idnar in #twisted). But it doesn't help.
Here's a bit of the recent IRC conversation, mentioning what I think could be happening: https://gist.github.com/2177560
As far as I can see, txRDQ is doing the right thing. The job fails and the deferred that is returned by queue.put is errbacked.
If you look in _trial_temp/test.log after you run the test, what do you see? I see an error that says Unhandled error in Deferred and the error is a Failure with a Job in it. So it seems likely to me that the error is somewhere in txRDQ. That there is a deferred that's failing and it's passing on the failure just fine to whoever needs it, but also returning the failure - causing trial to complain. But I don't know where that is. I put a print into the init of the Deferred class just out of curiousity to see how many deferreds were made during the running of the test. The answer: 12!
Sorry not to have better news. If you want to press on, go look at every deferred made by the txRDQ code. Is one of them failing with an errback that returns the failure? I don't see it, and I've put print statements in all over the place to check that things are right. I guess I must be missing something.
Thanks, and thanks too #exarkun.
Related
I have a test that may crash if the server is currently unavailable.
Can you tell me how I can immediately restart the same test in case it crashes (Without waiting for the completion of other tests)? And to recognize it as really failed only in case of a second failure.
I saw a solution using the pytest-rerunfailures library, but it needs to be additionally installed, and this does not quite suit me
Also, I present a solution with the try-except construction, but it seems to me that there should be a more convenient solution
With the help of try-except I present the solution like this:
def test_first():
try:
assert 1 == 0 # The main part of the function
except:
assert 1 == 0 # In case of failure, just repeat the same code
I am relatively new to Python, and I am having difficulty understanding the behavior of this pool. I have attempted to view similar questions on stackoverflow, but I am still having difficulties on comprehending this problem.
For example if we have use of an Executor:
with ThreadPoolExecutor(8) as pool:
pool.map(doSomething,test_list)
All errors that occur within the pool are suppressed and not shown.
Is there any parameter available so the pool stops execution and shows the exceptions?
Thanks in advance!
I have found a possibility for myself which can circumvent this problem indirectly. It's actually pretty simple so I'd like to publish it here if someone else encounters this problem. I do not know if it's very clever, but for me this solution was fine.
At first thanks to #artiom, as he already said, it is best to catch the error directly in the parallelized function.
Now the "bypass":
Put a try except block inside the function around the whole code and catch all exceptions with
except Exception as e:
.
The exception is simply logged in the except block, for example with sentry or another logging framework. With that, you can at least look at the thrown exceptions again and fix your mistakes in your code.
Simple example:
def for_parallel(chunk):
for one in chunk:
try:
a = 1
foo(one['bla'])
doSomething(one['blu'])
except Exception as e:
sentry.capture_exception(e)
def main():
with ThreadPoolExecutor(8) as pool:
pool.map(for_parallel, my_list)
Is it bad form to exit() from within function?
def respond_OK():
sys.stdout.write('action=OK\n\n')
sys.stdout.flush() # redundant when followed by exit()
sys.exit(0)
Rather than setting an exit code and exit()ing from the __main__ name space?
def respond_OK():
global exit_status
sys.stdout.write('action=OK\n\n')
sys.stdout.flush()
exit_status = 0
sys.exit(exit_status)
The difference is negligible from a function perspective, just wondered what the consensus is on form. If you found the prior in someone else's code, would you look at it twice?
I would prefer to see an exception raised and handled from a main entry point, the type of which is translated into the exit code. Subclassing exceptions is so simple in python it's almost fun.
As posted in this answer's comments: Using sys.exit also means that the point of termination needs to know the actual status code, as opposed to the kind of error it encountered. Though that could be solved by an set of constants, of course. Using exceptions has other advantages, though: if one method fails, you could try another without re-entry, or print some post-mortem debugging info.
It makes no difference in terms of functionality, but it will likely make your code harder to follow, unless you take appropriate steps, e.g. commenting each of the calls from the main namespace which could lead to an exit.
Update: Note #mgilson's answer re the effect of catching an exception [It is possible to catch the exception that system.exit raises, and thus prevent exit]. You could make your code even more confusing that way.
Update 2: Note #sapht's suggestion to use an exception to orchestrate an exit. This is good advice, if you really want to do a non-local exit. Much better than setting a global.
There are a few cases where it's reasonably idiomatic.
If the user gives you bad command-line arguments, instead of this:
def usage(arg0):
print ... % (arg0,)
return 2
if __name__ == '__main__':
if ...:
sys.exit(usage(sys.argv[0]))
You often see this:
def usage():
print ... % (sys.argv[0],)
sys.exit(2)
if __name__ == '__main__':
if ...:
usage()
The only other common case I can think of is where initializing some library (via ctypes or a low-level C extension module) fails unexpectedly and leaves you in a state you can't reason about, so you just want to get out as soon as possible (e.g., to reduce the chance of segfaulting or printing garbage) For example:
if libfoo.initialize() != 0:
sys.exit(1)
Some might object to that because sys.exit doesn't actually bail out of the interpreter as soon as possible (it throws and catches an exception), so it's a false sense of safety. But you still see it reasonably often.
I have some Twisted code which creates multiple chains of Deferreds. Some of these may fail without having an errback which puts them back on the callback chain. I haven't been able to write a unit test for this code - the failing Deferred causes the test to fail after the test code has completed. How can I write a passing unit test for this code? Is it expected that every Deferred which could fail in normal operation should have an errback at the end of the chain which puts it back on the callback chain?
The same thing happens when there's a failed Deferred in a DeferredList, unless I create the DeferredList with consumeErrors. This is the case even when the DeferredList is created with fireOnOneErrback and is given an errback which puts it back on the callback chain. Are there any implications for consumeErrors besides suppressing test failures and error logging? Should every Deferred which may fail without an errback be put a DeferredList?
Example tests of example code:
from twisted.trial import unittest
from twisted.internet import defer
def get_dl(**kwargs):
"Return a DeferredList with a failure and any kwargs given."
return defer.DeferredList(
[defer.succeed(True), defer.fail(ValueError()), defer.succeed(True)],
**kwargs)
def two_deferreds():
"Create a failing Deferred, and create and return a succeeding Deferred."
d = defer.fail(ValueError())
return defer.succeed(True)
class DeferredChainTest(unittest.TestCase):
def check_success(self, result):
"If we're called, we're on the callback chain."
self.fail()
def check_error(self, failure):
"""
If we're called, we're on the errback chain.
Return to put us back on the callback chain.
"""
return True
def check_error_fail(self, failure):
"""
If we're called, we're on the errback chain.
"""
self.fail()
# This fails after all callbacks and errbacks have been run, with the
# ValueError from the failed defer, even though we're
# not on the errback chain.
def test_plain(self):
"""
Test that a DeferredList without arguments is on the callback chain.
"""
# check_error_fail asserts that we are on the callback chain.
return get_dl().addErrback(self.check_error_fail)
# This fails after all callbacks and errbacks have been run, with the
# ValueError from the failed defer, even though we're
# not on the errback chain.
def test_fire(self):
"""
Test that a DeferredList with fireOnOneErrback errbacks on failure,
and that an errback puts it back on the callback chain.
"""
# check_success asserts that we don't callback.
# check_error_fail asserts that we are on the callback chain.
return get_dl(fireOnOneErrback=True).addCallbacks(
self.check_success, self.check_error).addErrback(
self.check_error_fail)
# This succeeds.
def test_consume(self):
"""
Test that a DeferredList with consumeErrors errbacks on failure,
and that an errback puts it back on the callback chain.
"""
# check_error_fail asserts that we are on the callback chain.
return get_dl(consumeErrors=True).addErrback(self.check_error_fail)
# This succeeds.
def test_fire_consume(self):
"""
Test that a DeferredList with fireOnOneCallback and consumeErrors
errbacks on failure, and that an errback puts it back on the
callback chain.
"""
# check_success asserts that we don't callback.
# check_error_fail asserts that we are on the callback chain.
return get_dl(fireOnOneErrback=True, consumeErrors=True).addCallbacks(
self.check_success, self.check_error).addErrback(
self.check_error_fail)
# This fails after all callbacks and errbacks have been run, with the
# ValueError from the failed defer, even though we're
# not on the errback chain.
def test_two_deferreds(self):
# check_error_fail asserts that we are on the callback chain.
return two_deferreds().addErrback(self.check_error_fail)
There are two important things about trial related to this question.
First, a test method will not pass if a Failure is logged while it is running. Deferreds which are garbage collected with a Failure result cause the Failure to be logged.
Second, a test method which returns a Deferred will not pass if the Deferred fires with a Failure.
This means that neither of these tests can pass:
def test_logit(self):
defer.fail(Exception("oh no"))
def test_returnit(self):
return defer.fail(Exception("oh no"))
This is important because the first case, the case of a Deferred being garbage collected with a Failure result, means that an error happened that no one handled. It's sort of similar to the way Python will report a stack trace if an exception reaches the top level of your program.
Likewise, the second case is a safety net provided by trial. If a synchronous test method raises an exception, the test doesn't pass. So if a trial test method returns a Deferred, the Deferred must have a success result for the test to pass.
There are tools for dealing with each of these cases though. After all, if you couldn't have a passing test for an API that returned a Deferred that fired with a Failure sometimes, then you could never test your error code. That would be a pretty sad situation. :)
So, the more useful of the two tools for dealing with this is TestCase.assertFailure. This is a helper for tests that want to return a Deferred that's going to fire with a Failure:
def test_returnit(self):
d = defer.fail(ValueError("6 is a bad value"))
return self.assertFailure(d, ValueError)
This test will pass because d does fire with a Failure wrapping a ValueError. If d had fired with a success result or with a Failure wrapping some other exception type, then the test would still fail.
Next, there's TestCase.flushLoggedErrors. This is for when you're testing an API that's supposed to log an error. After all, sometimes you do want to inform an administrator that there's a problem.
def test_logit(self):
defer.fail(ValueError("6 is a bad value"))
gc.collect()
self.assertEquals(self.flushLoggedErrors(ValueError), 1)
This lets you inspect the failures which got logged to make sure your logging code is working properly. It also tells trial not to worry about the things you flushed, so they'll no longer cause the test to fail. (The gc.collect() call is there because the error isn't logged until the Deferred is garbage collected. On CPython, it'll be garbage collected right away due to the reference counting GC behavior. However, on Jython or PyPy or any other Python runtime without reference counting, you can't rely on that.)
Also, since garbage collection can happen at pretty much any time, you might sometimes find that one of your tests fails because an error is logged by a Deferred created by an earlier test being garbage collected during the execution of the later test. This pretty much always means your error handling code is incomplete in some way - you're missing an errback, or you failed to chain two Deferreds together somewhere, or you're letting your test method finish before the task it started actually finishes - but the way the error is reported sometimes makes it hard to track down the offending code. Trial's --force-gc option can help with this. It causes trial to invoke the garbage collector between each test method. This will slow down your tests significantly, but it should cause the error to be logged against the test which is actually triggering it, not an arbitrary later test.
in the C world, a function can return error code to represent the exit status, and use INOUT/OUT parameter to carry the actual fruit of the process. when it comes to xmlrpc, no INOUT/OUT parameter, is there any best practice/conventions to represent the exit status and actual result?
the context is i am trying to write an agent/daemon (python SimpleXMLRPCServer) running on the Server, and want to design the "protocol" to interact with it.
any advice is appreciated.
EDIT:
per S.Lott's comment, make the problem more clear.
it is more about os convention rather
than C convention. I agree with that.
the job of the agent is more or less run some cmd on the server, inherently with an exit code/result idiom
.
One simple way to implement this in Python is with a tuple. Have your function return a tuple of: (status, result) where the status can be numeric or a string, and the result can be any Python data structure you fancy.
Here's an example, adapted from the module documentation. Server code:
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
# Restrict to a particular path.
class RequestHandler(SimpleXMLRPCRequestHandler):
rpc_paths = ('/RPC2',)
# Create server
server = SimpleXMLRPCServer(("localhost", 8000),
requestHandler=RequestHandler)
def myfunction(x, y):
status = 1
result = [5, 6, [4, 5]]
return (status, result)
server.register_function(myfunction)
# Run the server's main loop
server.serve_forever()
Client code:
import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:8000')
print s.myfunction(2, 4)
The server function returns a tuple
"in the C world, a function can return error code to represent the exit status, and use INOUT/OUT parameter to carry the actual fruit of the process"
Consider an exit status to be a hack. It's not a C-ism, it's a Linux-ism. C functions return exactly one value. C doesn't have exceptions, so there are several ways to indicate failure, all pretty bad.
Exception handling is what's needed. Python and Java have this, and they don't need exit status.
OS's however, still depend on exit status because shell scripting is still very primitive and some languages (like C) can't produce exceptions.
Consider in/out variables also to be a hack. This is a terrible hack because the function has multiple side-effects in addition to returning a value.
Both of these "features" aren't really the best design patterns to follow.
Ideally, a function is "idempotent" -- no matter how many times you call it, you get the same results. In/Out variables break idempotency in obscure, hard-to-debug ways.
You don't really need either of these features, that's why you don't see many best practices for implementing them.
The best practice is to return a value or raise an exception. If you need to return multiple values you return a tuple. If things didn't work, you don't return an exit status, you raise an exception.
Update. Since the remote process is basically RSH to run a remote command, you should do what remctl does.
You need to mimic: http://linux.die.net/man/1/remctl precisely. You have to write a Python client and server. The server returns a message with a status code (and any other summary, like run-time). The client exits with that same status code.