I wrote a twisted program that handling request from TCP sockets and raw sockets.
As the twisted doesn't support raw thread, I write the raw-socket select poll loop in a function named 'raw_socket_loop'. The main reactor program create a separate thread to run this loop by reactor.callInThread() function.
My problem is, I click control-C in the console but the reactor can not stop. I think the reactor's main thread receives this signal and handles well but the spawned thread doesn't receive this break signal. Is there is graceful shutdown suggestion for multi-thread reactor program?
Thanks a lot,
Threads aren't interruptable. You have to build a mechanism into the code running in a thread to receive shutdown notification and exit in response to it.
If you're using select(2) in the thread, then you can use the self-pipe trick (which is how Twisted itself does this for its own thread-control needs).
However, if you're using select(2) in a thread, then maybe you should consider not using a thread and instead implementing IFileDescriptor and using it with the reactor's IReactorFDSet implementation to get readiness events on it. This way you avoid threads, you let the reactor actually implement the event loop, and you still get your raw sockets.
Related
I am implementing a MQTT worker in python with paho-mqtt.
Are all the on_message() multi threaded in different threads, so that if one of the task is time consuming, other messages can still be processed?
If not, how to achieve this behaviour?
The python client doesn't actually start any threads, that's why you have to call the loop function to handle network events.
In Java you would use the onMessage callback to put the incoming message on to a local queue that a separate pool of threads will handle.
Python doesn't have native threading support but does have support for spawning processes to act like threads. Details of the multiprocessing can be found here:
https://docs.python.org/2.7/library/multiprocessing.html
EDIT:
On looking closer at the paho python code a little closer it appears it can actually start a new thread (using the loop_start() function) to handle the network side of things previously requiring the loop functions. This does not change the fact the all calls to the on_message callback will happen on this thread. If you need to do large amounts of work in this callback you should definitely look spinning up a pool of new threads to do this work.
http://www.tutorialspoint.com/python/python_multithreading.htm
I'm using a client to connect a socket via UDP in python. I have two threads. After a KeyboardInterrupt, the first thread still is waiting for a connection via recvfrom.
(...)
udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
(...)
udp.shutdown(socket.SHUT_RDWR) # Does not work
udp.close() # Does not work
I have a global variable that all threads share and I update this value after a KeyboardInterrupt. But the thread don't close.
How can I exit from this thread?
Thanks in advance.
Some clarification might be needed on your part, but this is what I understand from your question. If your KeyboardInterrupt kills the first thread and you need paired threads to close when this occurs, then you should consider using a daemon thread. This will guarantee the end of these threads once the parent thread finishes. However, as mentioned in the docs:
Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
Events are nothing other than signals meant to be utilized between threads. Just like a status flag, you can check whether an event has been 'set' and take proper precautions to handle resources before finishing execution within the thread.
I would like to know why doesn't python2.7 drop blocking operations when ctrl+c is pressed, I am unable to kill my threaded application, there are several socket waits, semaphore waits and so on. In python3 ctrl+c dropped every blocking operation and garbage-collected everything, released all the sockets and whatsoever ... Is there (I am convinced there is, I just yet don't know how) a way to acomplish this? Signal handle? Thanks guys
I guess you are launching the threads and then the main thread is waiting to join them on termination.
You should catch the exception generated by Ctrl-C in the main thread, in order to signal the spawned threads to terminate (changing a flag in each thread, for instance). In this manner, all the children thread will terminate and the main thread will complete the join call, reaching the bottom of your main.
I am implementing an app consuming a few http streams at the same time.
All threads (a pycurl object each) are spawned in the same loop.
The trick is how to build a proper architecture for handling reconnects.
Is it a good practice to create a separate controller thread that somehow
checks which connections are not alive or need forced reconnect?
Or may be such task should be done inside separate processes?
I would suggest to have one controling thread which spawns http streaming threads, and such a streaming thread implements the proper handling for a connection loss or timeout (e.g. either terminating itself or telling to controling thread that a new streaming thread should be spawned for a reconnect). Depending on your http serving peer you could also try to continue an interrupted stream by using the http Content-Range feature.
UPDATE: For ease of reading, here is how to add a callback before the reactor gets shutdown:
reactor.addSystemEventTrigger('before', 'shutdown', callable)
Original question follows.
If I have a client connected to a server, and it's chilling in the reactor main loop waiting for events, when I hit CTRL-C, I get a "Connection to the other side was lost in a non-clean fashion: Connection lost." How can I set it up so that I know when a KeyboardInterrupt happens, so that I can do proper clean-up and disconnect cleanly? Or how can I implement a cleaner way to shutdown that doesn't involve CTRL-C, if possible?
If you really, really want to catch C-c specifically, then you can do this in the usual way for a Python application - use signal.signal to install a handler for SIGINT that does whatever you want to do. If you invoke any Twisted APIs from the handler, make sure you use reactor.callFromThread since almost all other Twisted APIs are unsafe for invocation from signal handlers.
However, if you're really just interested in inserting some shutdown-time cleanup code, then you probably want to use IService.stopService (or the mechanism in terms of which it is implemented,reactor.addSystemEventTrigger) instead.
If you're using twistd, then using IService.stopService is easy. You already have an Application object with at least one service attached to it. You can add another one with a custom stopService method that does your shutdown work. The method is allowed to return a Deferred. If it does, then the shutdown process is paused until that Deferred fires. This lets you clean up your connections nicely, even if that involves some more network (or any other asynchronous) operations.
If you're not using twistd, then using reactor.addSystemEventTrigger directly is probably easier. You can install a before shutdown trigger which will get called in the same circumstance IService.stopService would have been called. This trigger (just any callable object) can also return a Deferred to delay shutdown. This is done with a call to reactor.addSystemEventTrigger('before', 'shutdown', callable) (sometime before shutdown is initiated, so that it's already registered whenever shutdown does happen).
service.tac gives an example of creating and using a custom service.
wxacceptance.py gives an example of using addSystemEventTrigger and delaying shutdown by (an arbitrary) three seconds.
Both of these mechanisms will give you notification whenever the reactor is stopping. This may be due to a C-c keystroke, or it may be because someone used kill -INT ..., or it may be because somewhere reactor.stop() was called. They all lead to reactor shutdown, and reactor shutdown always processes shutdown event triggers.
I'm not sure whether you talking about a client or a server that you've written.
Anyway, nothing wrong with 'CTRL-C'.
If you're writing a server as an Application. Subclass from twisted.application.service.Service and define startService and stopService. Maintain a list of active protocol instances. Use stopService to go through them and close them gracefully.
If you've got a client, you could also subclass Service, but it could be simpler to use reactor.addSystemEventTrigger('before','shutdown',myCleanUpFunction), and close connection(s) gracefully in this function.