WSGI is extremely useful for building highly concurrent HTTP servers to support e.g. long polling, however, typically, the long running HTTP request will at some point be ended by the client side; to clean up any resources and open handles, the WSGI server backend should be notified of any such events, however, it doesn't currently seem to be possible to catch those events in the WSGI handler:
# pseudocode example
def application(env, start_response):
start_response(...)
q = Queue()
ev_handle = register_event_handler(lambda event, arg: q.put((event, arg)))
# ??? need to call e.g. ev_handle.unregister() when the HTTP request is terminated
return iter(lambda: render(q.get()), None)
For example, when using gevent.pywsgi, the corresponding exception (error: [Errno 32] Broken pipe) is thrown somewhere inside gevent and never even seems to surface anywhere the handler could potentially see it:
Traceback (most recent call last):
File "/Users/erik.allik/.virtualenvs/myproj/lib/python2.7/site-packages/gevent/pywsgi.py", line 508, in handle_one_response
self.run_application()
File "/Users/erik.allik/.virtualenvs/myproj/lib/python2.7/site-packages/gevent/pywsgi.py", line 495, in run_application
self.process_result()
File "/Users/erik.allik/.virtualenvs/myproj/lib/python2.7/site-packages/gevent/pywsgi.py", line 486, in process_result
self.write(data)
File "/Users/erik.allik/.virtualenvs/myproj/lib/python2.7/site-packages/gevent/pywsgi.py", line 376, in write
self._write(data)
File "/Users/erik.allik/.virtualenvs/myproj/lib/python2.7/site-packages/gevent/pywsgi.py", line 369, in _write
self._sendall(data)
File "/Users/erik.allik/.virtualenvs/myproj/lib/python2.7/site-packages/gevent/pywsgi.py", line 355, in _sendall
self.socket.sendall(data)
File "/Users/erik.allik/.virtualenvs/myproj/lib/python2.7/site-packages/gevent/socket.py", line 458, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/Users/erik.allik/.virtualenvs/myproj/lib/python2.7/site-packages/gevent/socket.py", line 435, in send
return sock.send(data, flags)
Looks like what happens when a request gets terminated is that, in addition to the (seemingly uncatchable) exception traceback, the iterator that was returned from the WSGI handler is .close()-d. It is thus possible to determine when the any workers/resources/handles associated with the response should be closed. This is basically what werkzeug.wsgi.ClosingIterator does:
class ClosingIterator(object):
def __init__(self, iterable, on_close):
iterator = iter(iterable)
self.close = on_close
def __iter__(self):
return self
def __next__(self):
return self._next()
def application(env, start_response):
start_response(...)
q = Queue()
ev_handle = register_event_handler(lambda event, arg: q.put((event, arg)))
return ClosingIterator(
iter(lambda: render(q.get()), None),
on_close=ev_handle.unregister
)
This does not however silence the error message/traceback, but this seems tolerable unless somebody can come up with a solution that can fix even that.
Related
I have a textbased interface (asciimatics module) for my program that uses asyncio and discord.py module and occasionally when my wifi adapter goes down I get an exception like so:
Task exception was never retrieved
future: <Task finished coro=<WebSocketCommonProtocol.run() done, defined at /home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py:428> exception=ConnectionResetError(104, 'Connection reset by peer')>
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 434, in run
msg = yield from self.read_message()
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 456, in read_message
frame = yield from self.read_data_frame(max_size=self.max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 511, in read_data_frame
frame = yield from self.read_frame(max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 546, in read_frame
self.reader.readexactly, is_masked, max_size=max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/framing.py", line 86, in read_frame
data = yield from reader(2)
File "/usr/lib/python3.5/asyncio/streams.py", line 670, in readexactly
block = yield from self.read(n)
File "/usr/lib/python3.5/asyncio/streams.py", line 627, in read
yield from self._wait_for_data('read')
File "/usr/lib/python3.5/asyncio/streams.py", line 457, in _wait_for_data
yield from self._waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/selector_events.py", line 662, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
This exception is non-fatal and the program is able to re-connect despite it - what I want to do is prevent this exception from dumping to stdout and mucking up my text interface.
I tried using ensure_future to handle it but it doesn't seem to work. Am I missing something:
#asyncio.coroutine
def handle_exception():
try:
yield from WebSocketCommonProtocol.run()
except Exception:
print("SocketException-Retrying")
asyncio.ensure_future(handle_exception())
#start discord client
client.run(token)
Task exception was never retrieved - is not actually exception propagated to stdout, but a log message that warns you that you never retrieved exception in one of your tasks. You can find details here.
I guess, most easy way to avoid this message in your case is to retrieve exception from task manually:
coro = WebSocketCommonProtocol.run() # you don't need any wrapper
task = asyncio.ensure_future(coro)
try:
#start discord client
client.run(token)
finally:
# retrieve exception if any:
if task.done() and not task.cancelled():
task.exception() # this doesn't raise anything, just mark exception retrieved
The answer provided by Mikhail is perfectly acceptable, but I realized it wouldn't work for me since the task that is raising the exception is buried deep in some module so trying to retrieve it's exception is kind've difficult. I found that instead if I simply set a custom exception handler for my asyncio loop (loop is created by the discord client):
def exception_handler(loop,context):
print("Caught the following exception")
print(context['message'])
client.loop.set_exception_handler(exception_handler)
client.run(token)
I have a web service(REST) where one request might take up to 30 sec to return an answer (lots of calculation). There is a risk, that during the calculation, the client webbrowser aborts(?) the existing connection and retries. Here is the console-output of the server-side:
Exception happened during processing of request from ('127.0.0.1', 53209)
Traceback (most recent call last):
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 290, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 318, in process_request
self.finish_request(request, client_address)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 654, in __init__
self.finish()
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 713, in finish
self.wfile.close()
File "C:\Users\tmx\Anaconda2\lib\socket.py", line 283, in close
self.flush()
File "C:\Users\tmx\Anaconda2\lib\socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
One option is what I thought of is to somehow notify the client that "I'm alivem but the request is still needs some more time", or to somehow set the timeout on server side. What are the possibilities?
It's difficult to run code in Flask after you've already returned some data. Your options are to either use something like a task queue (see Celery), or to yield your response in multiple parts.
Views in Flask can return strings, but they can also return iterables that contain strings. So you could return "abc", ["abc"], or a generator that will yield "abc". If you do your processing between yields, data will get sent to the client while the request is still running.
Take a look at the following example:
def generator_that_does_the_calculation():
sleep(1)
yield "I'm alive, but I need some time\n"
sleep(1)
yield "Still alive here\n"
sleep(1)
yield "Done\n"
#app.route('/calculate')
def calculate():
return Response(generator_that_does_the_calculation())
I'm running a Flask app and internally using a library written in Node.js, which I access through ZeroRPC (the actual node process is managed by Circus). This works fine on its own; I can unit test with no issues. But when starting the Flask app as a listening process, and calling into a REST api which calls this libary, the program throws an exception when trying to start the process. The code to start the service is as follows:
from circus.watcher import Watcher
from circus.arbiter import ThreadedArbiter
from circus.util import (DEFAULT_ENDPOINT_DEALER, DEFAULT_ENDPOINT_SUB,
DEFAULT_ENDPOINT_MULTICAST)
class Node(object):
{... omitted code that initializes self._arbiter and self._client ...}
def start(self):
if self._arbiter and self._client:
return
port = 'ipc:///tmp/inlinejs_%s' % os.getpid()
args = 'lib/server.js --port %s' % port
watcher = Watcher('node', '/usr/local/bin/node', args,
working_dir=INLINEJS_DIR)
self._arbiter = ThreadedArbiter([watcher], DEFAULT_ENDPOINT_DEALER,
DEFAULT_ENDPOINT_SUB, multicast_endpoint=DEFAULT_ENDPOINT_MULTICAST)
self._arbiter.start()
self._client = zerorpc.Client()
self._client.connect(port)
This function returns, but shortly afterwards in a separate thread, I get this error:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/python/lib/python2.7/site-packages/circus/_patch.py", line 21, in _bootstrap_inner
self.run()
File "/python/lib/python2.7/site-packages/circus/arbiter.py", line 647, in run
return Arbiter.start(self)
File "/python/lib/python2.7/site-packages/circus/util.py", line 319, in _log
return func(self, *args, **kw)
File "/python/lib/python2.7/site-packages/circus/arbiter.py", line 456, in start
self.initialize()
File "/python/lib/python2.7/site-packages/circus/util.py", line 319, in _log
return func(self, *args, **kw)
File "/python/lib/python2.7/site-packages/circus/arbiter.py", line 427, in initialize
self.evpub_socket.bind(self.pubsub_endpoint)
File "socket.pyx", line 432, in zmq.core.socket.Socket.bind (zmq/core/socket.c:4022)
File "checkrc.pxd", line 21, in zmq.core.checkrc._check_rc (zmq/core/socket.c:5838)
ZMQError: Address already in use
I have no idea why this is happening, especially since it doesn't happen in unit tests. Can anyone shed any light?
In general, when you get this type of "Address in use" error, it means that your program is trying to bind on an IP port number but something else got there first.
I am not familiar with this library, but since the error is caused by "evpub_socket.bind", I am going to guess that you have a conflict with the port number specified by the constant DEFAULT_ENDPOINT_SUB. From the circus source code I see these constants:
DEFAULT_ENDPOINT_DEALER = "tcp://127.0.0.1:5555"
DEFAULT_ENDPOINT_SUB = "tcp://127.0.0.1:5556"
DEFAULT_ENDPOINT_STATS = "tcp://127.0.0.1:5557"
Check your system (netstat) and see if any process is listening on ports 5555, 5556, 5557. Or perhaps you are running this program twice and you forgot about the first one.
I've tried to simplify this as much as possible but I'm still getting an error. I have a simple http server (bottle) that upon receiving a post request executes a function which is supposed to quickly fork itself. The parent process simply returns a job ID and closes while the child process continues to process same data (which is a list of URLs). I've removed all the input and output functions and hard coded the data but my program is still crashing. The funny part is when I alter the program to run directly at the command line rather then start an http server and wait for bottle to execute it everything works fine!
#!/usr/bin/python
#This is a comment
import sys, time, bottle, os
from threading import Thread
from Queue import Queue
from bottle import route, run, request, abort
num_fetch_threads = 2
url_queue = Queue()
def fetchURLContent(i, q):
while True:
#print '%s: Looking for URLs in queue' % i
url = q.get()
#print 'URL found: %s' % url[0]
q.task_done()
time.sleep(1)
#route('/', method='POST') # or #route('/login', method='POST')
def main():
urls = ['http://www.yahoo.com', 'http://www.google.com']
newpid = os.fork()
if newpid == 0:
for i in range(num_fetch_threads):
worker = Thread(target=fetchURLContent, args=(i, url_queue))
worker.setDaemon(True)
worker.start()
print 'Queuing: ', url
for url in urls:
url_queue.put(url)
time.sleep(2)
print 'main thread waiting...'
url_queue.join()
print 'Done'
else:
print "Your job id is 5"
return
def webServer():
run(host='33.33.33.10', port=8080)
if __name__ == "__main__":
print 'Listening on 8080...'
webServer()
The error message I get is as follows:
Listening on 8080...
Bottle v0.11.3 server starting up (using WSGIRefServer())...
Listening on http://33.33.33.10:8080/
Hit Ctrl-C to quit.
33.33.33.1 - - [19/Oct/2012 21:21:24] "POST / HTTP/1.1" 200 0
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
self.finish_content()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 246, in finish_content
self.send_headers()
9 url_queue = Queue()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 268, in send_headers
self.send_preamble()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 189, in send_preamble
self._write('HTTP/%s %s\r\n' % (self.http_version,self.status))
File "/usr/lib/python2.7/wsgiref/handlers.py", line 389, in _write
self.stdout.write(data)
File "/usr/lib/python2.7/socket.py", line 324, in write
self.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('33.33.33.1', 57615)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 640, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 693, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Any ideas?
Your main() function terminates immediately without returning anything. Bottle writes an empty HTTP response to the socket and the web server closes the connection.
Your forked off process stays a bit longer in main(), but then terminates too and causes Bottle to write another empty response to the already closed socket. Thats the error you get (broken pipe).
Forking at that point cannot work. HTTP does not allow more than one response per request. You can either block until all work is done and then send a response, or send the response immediately and do the work in a differed thread.
I'm using tornado 2.0 and occassionally when I call self.finish() to end an asynchronous request, I'll get an IOError with the message "Stream is closed". It looks as though this happens when the client ends a request (ie by navigating to another page) prior to the server calling finish(). Is this expected behavior and something my code just needs to handle? I found this bug from a year ago that suggests this is NOT something client code should be handling: https://github.com/facebook/tornado/issues/81. Is this indicative of a bug in my code, and if so, what are the likely causes?
Stacktrace:
Traceback (most recent call last):
File "my_code.py", line 260, in my_method
self.finish()
File "/usr/lib/python2.6/site-packages/tornado/web.py", line 634, in finish
self.request.finish()
File "/usr/lib/python2.6/site-packages/tornado/httpserver.py", line 555, in finish
self.connection.finish()
File "/usr/lib/python2.6/site-packages/tornado/httpserver.py", line 349, in finish
self._finish_request()
File "/usr/lib/python2.6/site-packages/tornado/httpserver.py", line 372, in _finish_request
self.stream.read_until(b("\r\n\r\n"), self._header_callback)
File "/usr/lib/python2.6/site-packages/tornado/iostream.py", line 137, in read_until
self._check_closed()
File "/usr/lib/python2.6/site-packages/tornado/iostream.py", line 403, in _check_closed
raise IOError("Stream is closed")
IOError: Stream is closed
self.finish() is called to end the asynchronous request, and some functions like self.render() will call self.finish().
If you call self.finish() after the connection is closed, it will cause the error.
so you can check if you call some functions that finish the connection before self.finish()
or you can do like this:
if not self._finished:
#if the connection is closed, it won't call this function
self.finish()
else:
pass