I've just rewritten something akin to a basic python server
( https://docs.python.org/3/library/socketserver.html ) because I thought I needed to.
My question is, did I?
What I wanted to do is break out of the handler and out of the server loop if a certain request is received (a stop-the-server request, if you will).
Originally, I tried to break out of the server loop by throwing an exception, but it turns out the way the socketserver handlers are run is inside of a "try catch-all expect" block, which means exceptions thrown inside of a handler won't ever propagate beyond the handler invoking function (the one with the catch-all exception block).
So does python has a longjump mechanism that can pierce a try-catch_all-expect block or could I run the serve_forever_loop inside a thread and then, from the handler, do something like Thread.current.kill() (how can I do this?).
As far as I know, there is no way to skip stack frames when you raise an exception.
But if you really need this functionality, you can find other ways for one part of your code to send messages to another part. If both the handler and server are running in the same interpreter instance (i.e. not in separate threads), you can have the handler change some variable accessible to the main server loop, which the server loop checks for. If you're in different interpreters, you could have the handler write to a log file that the server loop watches. The log file idea is kind of hackish, but logging is a good thing to have for servers anyway.
Related
I know about this python bug, which does not allow sys.excepthook to be used from inside a thread. A recommended workaround is to surround the thread's run method with try/except. However, since I am using Django, I am already locked in in Django main thread.
So: Is there any way to globally catch exceptions in Django?
I'm using Python http.client.HTTPResponse.read() to read data from a stream. That is, the server keeps the connection open forever and sends data periodically as it becomes available. There is no expected length of response. In particular, I'm getting Tweets through the Twitter Streaming API.
To accomplish this, I repeatedly call http.client.HTTPResponse.read(1) to get the response, one byte at a time. The problem is that the program will hang on that line if there is no data to read, which there isn't for large periods of time (when no Tweets are coming in).
I'm looking for a method that will get a single byte of the HTTP response, if available, but that will fail instantly if there is no data to read.
I've read that you can set a timeout when the connection is created, but setting a timeout on the connection defeats the whole purpose of leaving it open for a long time waiting for data to come in. I don't want to set a timeout, I want to read data if there is data to be read, or fail if there is not, without waiting at all.
I'd like to do this with what I have now (using http.client), but if it's absolutely necessary that I use a different library to do this, then so be it. I'm trying to write this entirely myself, so suggesting that I use someone else's already-written Twitter API for Python is not what I'm looking for.
This code gets the response, it runs in a separate thread from the main one:
while True:
try:
readByte = dc.request.read(1)
except:
readByte = []
if len(byte) != 0:
dc.responseLock.acquire()
dc.response = dc.response + chr(byte[0])
dc.responseLock.release()
Note that the request is stored in dc.request and the response in dc.response, these are created elsewhere. dc.responseLock is a Lock that prevents dc.response from being accessed by multiple threads at once.
With this running on a separate thread, the main thread can then get dc.response, which contains the entire response received so far. New data is added to dc.response as it comes in without blocking the main thread.
This works perfectly when it's running, but I run into a problem when I want it to stop. I changed my while statement to while not dc.twitterAbort, so that when I want to abort this thread I just set dc.twitterAbort to True, and the thread will stop.
But it doesn't. This thread remains for a very long time afterward, stuck on the dc.request.read(1) part. There must be some sort of timeout, because it does eventually get back to the while statement and stop the thread, but it takes around 10 seconds for that to happen.
How can I get my thread to stop immediately when I want it to, if it's stuck on the call to read()?
Again, this method is working to get Tweets, the problem is only in getting it to stop. If I'm going about this entirely the wrong way, feel free to point me in the right direction. I'm new to Python, so I may be overlooking some easier way of going about this.
Your idea is not new, there are OS mechanisms(*) for making sure that an application is only calling I/O-related system calls when they are guaranteed to be not blocking . These mechanisms are usually used by async I/O frameworks, such as tornado or gevent. Use one of those, and you will find it very easy to run code "while" your application is waiting for an I/O event, such as waiting for incoming data on a socket.
If you use gevent's monkey-patching method, you can proceed using http.client, as requested. You just need to get used to the cooperative scheduling paradigm introduced by gevent/greenlets, in which your execution flow "jumps" between sub-routines.
Of course you can also perform blocking I/O in another thread (like you did), so that it does not affect the responsiveness of your main thread. Regarding your "How can I get my thread to stop immediately" problem:
Forcing a thread that's blocking in a system call to stop is usually not a clean or even valid process (also see Is there any way to kill a Thread in Python?). Either -- if your application has finished its jobs -- you take down the entire process, which also affects all contained threads, or you just leave the thread be and give it as much time to terminate as required (these 10 seconds you were referring to are not a problem -- are they?)
If you do not want to have such long-blocking system calls anywhere in your application (be it in the main thread or not), then use above-mentioned techniques to prevent blocking system calls.
(*) see e.g. O_NONBLOCK option in http://man7.org/linux/man-pages/man2/open.2.html
My setting is the following - I have a Tornado based HTTP server which is pretty much the "hello world" example:
server = tornado.httpserver.HTTPServer(app)
server.bind(8888)
server.start(2)
tornado.ioloop.IOLoop.instance().start()
Now in this setting, I also have another process (let's call it control process) spawned by the root process. Thus this control process is a sibling to the two Tornado handler processes. Naturally I can communicate between the handler processes and the control process through a pipe created by the root process. I however, am more interested in calling a method of the control process and getting it's output.
What is the best approach to do such a thing? If I use the pipe for sending the request from a handler to the control and return the result, should I use a lock to implement process-safety?
You don't need a lock with pipes. The pipe is its own synchronization. Or, put a different way, the two sides of the pipe are separate objects.
(Of course the control process may need a lock internally, e.g., if it's handling the pipe from a different thread than the main event loop and needs to share any data with code that runs in the main loop, but that's not related to inter-process safety.)
Anyway, if you step back and think about this from a higher level, what you're implementing is the exact definition of an RPC mechanism. If what you're doing is simple enough, implementing it from scratch this way is fine, but if not, you may want to have add another protocol to control and let Tornado manage it along with your existing protocol(s).
I'm trying to build a Python application using pyGTK, treads, and sockets. I'm having this weird error, but given all the modules involved, I'm not entirely sure where the error is. I did a little debugging with some print statements to narrow things down a bit and I think the error is somewhere in this snippet of code:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect(("localhost", 5005))
self.collectingThread = threading.Thread(target=self.callCollect)
self.collectingThread.daemon = True
self.collectingThread.start()
def callCollect(self):
gobject.timeout_add(500, self.collectData)
def collectData(self):
print "hello"
try:
print self.sock.recv(1024)
except:
print "except"
print "return"
return True
So basically what I'm trying to do is setup a socket, connect to a "server" script (which is really just another python script running locally), and create a separate thread to collect all incoming data from the server script. This thread is set to run the collectData method every 500 milliseconds.
After inserting the print statements into the collectData method here is what I notice when running the program:
-Initially the GUI is fully functional
-then the following is printed in the terminal:
hello
**all data received from server script and printed here**
return
hello
-after the text is printed in the terminal, the GUI becomes completely nonfunctional (buttons cant be pressed, etc.) and I have to force quit to close the application
What seems to be happening is that the thread prints "hello", prints the data from the server, and prints "return". 500 milliseconds later, it runs the collectData method again, prints "hello", then tries to print data from the server. However, because there is no data left it raises an exception, but for some unknown reason it doesn't execute the code in the exception block and everything just hangs from there.
Any idea on what is going wrong?
timeout_add is scheduling the action to happen on the main thread -- so the recv just blocks the main thread (when it's just waiting for data) and therefore the GUI, so, no exception unless you put a timeout or set the socket to non-blocking.
You need to delegate the receiving to the thread from the scheduled action rather than viceversa to get the effect you're after: have the thread e.g. wait on an event object, and the scheduled action signal that event every 500 milliseconds.
No, obviously the sock.recv call blocks because the socket wasn't closed yet and socket receives are blocking by default. Make sure you close the connection at some point.
It would make more sense to run the receive call in a new thread, or else it might block the GUI because your current implementation runs the recv call in the GUI thread (using timeout_add). The way you're currently doing it only makes sense if the answer is received very fast and/or you have to access widgets.
By the way, creating a new thread for calling gobject.timeout_add is totally unnecessary. timeout_add() and idle_add() register the specified callback function and return immediately. The GTK event loop then automatically executes the callback after the timeout (or on idle status for idle_add).
UPDATE: For ease of reading, here is how to add a callback before the reactor gets shutdown:
reactor.addSystemEventTrigger('before', 'shutdown', callable)
Original question follows.
If I have a client connected to a server, and it's chilling in the reactor main loop waiting for events, when I hit CTRL-C, I get a "Connection to the other side was lost in a non-clean fashion: Connection lost." How can I set it up so that I know when a KeyboardInterrupt happens, so that I can do proper clean-up and disconnect cleanly? Or how can I implement a cleaner way to shutdown that doesn't involve CTRL-C, if possible?
If you really, really want to catch C-c specifically, then you can do this in the usual way for a Python application - use signal.signal to install a handler for SIGINT that does whatever you want to do. If you invoke any Twisted APIs from the handler, make sure you use reactor.callFromThread since almost all other Twisted APIs are unsafe for invocation from signal handlers.
However, if you're really just interested in inserting some shutdown-time cleanup code, then you probably want to use IService.stopService (or the mechanism in terms of which it is implemented,reactor.addSystemEventTrigger) instead.
If you're using twistd, then using IService.stopService is easy. You already have an Application object with at least one service attached to it. You can add another one with a custom stopService method that does your shutdown work. The method is allowed to return a Deferred. If it does, then the shutdown process is paused until that Deferred fires. This lets you clean up your connections nicely, even if that involves some more network (or any other asynchronous) operations.
If you're not using twistd, then using reactor.addSystemEventTrigger directly is probably easier. You can install a before shutdown trigger which will get called in the same circumstance IService.stopService would have been called. This trigger (just any callable object) can also return a Deferred to delay shutdown. This is done with a call to reactor.addSystemEventTrigger('before', 'shutdown', callable) (sometime before shutdown is initiated, so that it's already registered whenever shutdown does happen).
service.tac gives an example of creating and using a custom service.
wxacceptance.py gives an example of using addSystemEventTrigger and delaying shutdown by (an arbitrary) three seconds.
Both of these mechanisms will give you notification whenever the reactor is stopping. This may be due to a C-c keystroke, or it may be because someone used kill -INT ..., or it may be because somewhere reactor.stop() was called. They all lead to reactor shutdown, and reactor shutdown always processes shutdown event triggers.
I'm not sure whether you talking about a client or a server that you've written.
Anyway, nothing wrong with 'CTRL-C'.
If you're writing a server as an Application. Subclass from twisted.application.service.Service and define startService and stopService. Maintain a list of active protocol instances. Use stopService to go through them and close them gracefully.
If you've got a client, you could also subclass Service, but it could be simpler to use reactor.addSystemEventTrigger('before','shutdown',myCleanUpFunction), and close connection(s) gracefully in this function.