We have a complex multiservice that needs to do some fairly intricate accounting when shutting down in order to implement a "graceful" shutdown.
I'm trying to write tests for this under trial. The issue is that the reactor is effectively a process-global resource, and shutting down my service means that trial's reactor is also stopped, which (of course) makes it explode.
This is documented to be a no-no in trial, but I need some kind of workaround that allows me to write my tests. My first thought was to use a mock.Mock, but this means we're not really using a reactor that's shutting down, which isn't going to give me behavior that's faithful to the actual shutdown process.
I believe what I need is a way to separate trial's reactor from the reactor of my service-under-test. Sharing a mutable resource between the test system and the system under test is surely an anti-pattern.
There's a difference between shutting down a service and stopping a reactor. You should be able to test most of the desired behavior with myservice.stopService. To test the code that actually initiates the shutdown, just mock out reactor.stop with self.patch(reactor, 'stop', mock.Mock()), and later assert that it was called. If you want to link the two, then have your mock stop call your service's stopService.
Related
I've tried lately to write my own Socket-Server in python.
While i was writing a thread to handle server commands (sort of command line in the server), I've tried to implement a code that will restart the server when the raw_input() receives specific command.
Basically, i want to restart the server as soon as the "Running" variable changes its state from True to False, and when it does, i would like to stop the function (The function that called the thread) from running (get back to main function) and then run it again. Is there a way to do it?
Thank you very much, and i hope i was clear about my problem,
Idan :)
Communication between threads can be done with Events, Queues, Semaphores, etc. Check them out and choose the one, that fits your problem best.
You can't abort a thread, or raise an exception into it asynchronously, in Python.
The standard Unix solution to this problem is to use a non-blocking socket, create a pipe with pipe, replace all your blocking sock.recv calls with a blocking r, _, _ = select.select([sock, pipe], [], []), and then the other thread can write to the pipe to wake up the other thread.
To make this portable to Windows you'll need to create a UDP localhost socket instead of a pipe, which makes things slightly more complicated, but it's still not hard.
Or, of course, you can use a higher-level framework, like asyncio in 3.4+, or twisted or another third-party lib, which will wrap this up for you. (Most of them are already running the equivalent of a loop around select to service lots of clients in one thread or a small thread pool, so it's trivial to toss in a stop pipe.)
Are there other alternatives? Yes, but all less portable and less good in a variety of other ways.
Most platforms have a way to asynchronously kill or signal another thread, which you can access via, e.g., ctypes. But this is a bad idea, because it will prevent Python from doing any normal cleanup. Even if you don't get a segfault, this could mean files never get flushed and end up with incomplete/garbage data, locks are left acquired to deadlock your program somewhere completely unrelated a short time later, memory gets leaked, etc.
If you're specifically trying to interrupt the main thread, and you only care about CPython on Unix, you can use a signal handler and the kill function. The signal will take effect on the next Python bytecode, and if the interpreter is blocked on any kind of I/O (or most other syscalls, e.g., inside a sleep), the system will return to the interpreter with an EINTR, allowing it to interrupt immediately. If the interpreter is blocked on something else, like a call to a C library that blocks signals or just does nothing but CPU work for 30 seconds, then you'll have to wait 30 seconds (although that doesn't come up that often, and you should know if it will in your case). Also, threads and signals don't play nice on some older *nix platforms. And signals don't work the same way on Windows, or in some other Python implementations like Jython.
On some platforms (including Windows--but not most modern *nix plafforms), you can wake up a blocking socket call just by closing the socket out from under the waiting thread. On other platforms, this will not unblock the thread, or will do it sometimes but not other times (and theoretically it could even segfault your program or leave the socket library in an unusable state, although I don't think either of those will happen on any modern platform).
As far as I understand the documentation, and some experiments I've over the last weeks, there is no way to really force another thread to 'stop' or 'abort'. Unless the function is aware of the possibility of being stopped and has a foolproof method of avoiding getting stuck in some of the I/O functions. Then you can use some communication method such as semaphores. The only exception is the specialized Timer function, which has a Cancel method.
So, if you really want to stop the server thread forcefully, you might want to think about running it in a separate process, not a thread.
EDIT: I'm not sure why you want to restart the server - I just thought it was in case of a failure. Normal procedure in a server is to loop waiting for connections on the socket, and when a connection appears, attend it and return to that loop.
A better way, is to use the GIO library (part of glib), and connect methods to the connection event, to attend the connection even asynchronously. This avoids the loop completely. I don't have any real code for this in Python, but here's an example of a client in Python (which uses GIO for reception events) and a server in C, which uses GIO for connections.
Use of GIO makes life so much easier...
To achieve something similar to google app engines 'deferred calls' (i.e., the request is handled, and afterwards the deferred task is handled), i experimented a little and came up with the solution to spawn a thread in which my deferred call is handled.
I am now trying to determine if this is an acceptable way.
Is it possible (according to the WSGI specification) that the process is terminated by the webserver after the actual request is handled, but before all threads run out?
(if there's a better way, that would be also fine)
WSGI does not specify the lifetime of an application process (as WSGI application is a Python callable object). You can run it in a way that is completely independent of the web server, in which case, only you control the lifetime.
There is also nothing in the WSGI that would prohibit you from spawning threads, or processes, or doing whatever the hell you want.
FWIW, also have a read of:
http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode
The hooking of actions to close() of iterable is the only way within context of the WSGI specification itself for doing deferred work. That isn't in a separate thread though and would occur within the context of the actual request, albeit after the response is supposed to have been flushed back to the client. Thus your deferred action will consume that request thread until the work is complete and so that request thread would not be able to handle other requests until then.
In general, if you do use background threads, there is no guarantee that any hosting mechanism would wait until those background threads complete before shutting process down. In fact, can't even think of any standard deployment mechanism which does wait. There isn't really even a guarantee that atexit handlers will be called on process shutdown, something that the referenced documentation also briefly talks about.
hello to you all :)
i have a program that have a n number of threads(could be a lot) and they do a pretty extensive job. My problem is that sometimes some people turn off or reboot the server(the program runs all day in the company servers) i know that there is a way to make a handler for the linux signals i want to know what i should do to interact with all threads making them to use run a function and then stop working. There is a way to do that?
sorry the bad english :P
The best way of handling this is not requiring any shutdown actions at all.
For example, your signal handler for (e.g.) SIGTERM or SIGQUIT can just call _exit and quit the process with no clean-up.
Under Linux (with non-ancient threads) when one thread calls _exit (or exit if you really want) other threads get stopped too - whatever they were in the middle of doing.
This would be good as it implements a crash-only design.
Crash-only design for a server is based on the principle that the machine may crash at any point, so you need to be able to recover from such a failure anyway, so just make it the normal way of quitting. No extra code should be required as your server should be robust enough anyway.
About the only thing you can do is set a global variable from your signal handler, and have your threads check its value periodically.
As others have already mentioned, signal handlers can get messy (due to the restrictions, particularly in multi-threaded programs), so it's better to chose another option:
have a dedicated thread for handling signals via sigwaitinfo - the bad news, though, is that python doesn't appear to support that out of the box.
use the Linux-specific signalfd to handle signals (either in a separate thread or integrated into some event loop) - at least there is a python-signalfd module you can use.
As there is no need to install signal handlers here, there is no restriction on what you can do when you are notified of a signal and it should be easy to shut down the others threads in your program cleanly.
Is it somehow possible to "detect" that the reactor is overloaded and start dropping connections, or refuse new connections? How can we avoid the reactor being completely overloaded and not being able to catch up?
If I understand Twisted Reactors correctly, they don't parallelize everything. Whatever operations have been queued is scheduled and is done one by one.
One way out for you is to have a custom addCallback which checks for how many callbacks have been registered already and drop if necessary.
No easy way, but here's some suggestions: http://www.mail-archive.com/twisted-python#twistedmatrix.com/msg00389.html
I would approach this per protocol. Throttle when the actual service requires it, not when you think it will. Rather than worrying about how many callbacks are waiting for a reactor tick, I'd worry about how long the HTTP requests (for example) are taking to complete. The number of operations waiting for the reactor could be an implementation detail - for example, if one access pattern ended up with callbacks on long DeferredLists, and another had a more linear chain of callbacks, the time to respond might not be different even though the number of callbacks would be.
This could be done by keeping metrics of the time to complete a logical operation (such as servicing a HTTP request). An advantage of this is that it gives you important information before a problem happens.
UPDATE: For ease of reading, here is how to add a callback before the reactor gets shutdown:
reactor.addSystemEventTrigger('before', 'shutdown', callable)
Original question follows.
If I have a client connected to a server, and it's chilling in the reactor main loop waiting for events, when I hit CTRL-C, I get a "Connection to the other side was lost in a non-clean fashion: Connection lost." How can I set it up so that I know when a KeyboardInterrupt happens, so that I can do proper clean-up and disconnect cleanly? Or how can I implement a cleaner way to shutdown that doesn't involve CTRL-C, if possible?
If you really, really want to catch C-c specifically, then you can do this in the usual way for a Python application - use signal.signal to install a handler for SIGINT that does whatever you want to do. If you invoke any Twisted APIs from the handler, make sure you use reactor.callFromThread since almost all other Twisted APIs are unsafe for invocation from signal handlers.
However, if you're really just interested in inserting some shutdown-time cleanup code, then you probably want to use IService.stopService (or the mechanism in terms of which it is implemented,reactor.addSystemEventTrigger) instead.
If you're using twistd, then using IService.stopService is easy. You already have an Application object with at least one service attached to it. You can add another one with a custom stopService method that does your shutdown work. The method is allowed to return a Deferred. If it does, then the shutdown process is paused until that Deferred fires. This lets you clean up your connections nicely, even if that involves some more network (or any other asynchronous) operations.
If you're not using twistd, then using reactor.addSystemEventTrigger directly is probably easier. You can install a before shutdown trigger which will get called in the same circumstance IService.stopService would have been called. This trigger (just any callable object) can also return a Deferred to delay shutdown. This is done with a call to reactor.addSystemEventTrigger('before', 'shutdown', callable) (sometime before shutdown is initiated, so that it's already registered whenever shutdown does happen).
service.tac gives an example of creating and using a custom service.
wxacceptance.py gives an example of using addSystemEventTrigger and delaying shutdown by (an arbitrary) three seconds.
Both of these mechanisms will give you notification whenever the reactor is stopping. This may be due to a C-c keystroke, or it may be because someone used kill -INT ..., or it may be because somewhere reactor.stop() was called. They all lead to reactor shutdown, and reactor shutdown always processes shutdown event triggers.
I'm not sure whether you talking about a client or a server that you've written.
Anyway, nothing wrong with 'CTRL-C'.
If you're writing a server as an Application. Subclass from twisted.application.service.Service and define startService and stopService. Maintain a list of active protocol instances. Use stopService to go through them and close them gracefully.
If you've got a client, you could also subclass Service, but it could be simpler to use reactor.addSystemEventTrigger('before','shutdown',myCleanUpFunction), and close connection(s) gracefully in this function.