Develop Reference

Python http requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I'm writing a chat program. Each of my clients has an open get request to the server in a separate thread (and another thread for posting their own messages). I don't want to have a lot of overhead. That is, clients don't send get requests frequently to see if there have been any unseen messages. Instead, they always have exactly one open get request to get the new messages, and as soon as the server responded to them with new unseen messages, they immediately send another get request to the server to stay updated and so on.
So on the client-side, I have something like this:
def coms():
headers = {'data': myAut.strip()}
resp = requests.get("http://localhost:8081/receive", headers=headers,timeout=1000000)
print(resp.text)
t = threading.Thread(target=coms, args=())
t.start()
On the server-side, I have something like this:
def do_GET(self):
if self.path == '/receive':
auth=self.headers['data']
#Using auth, find who has sent this message
u=None
for i in range(len(users)):
print(users[i].aut,auth)
if users[i].aut==auth:
u=users[i]
break
t=threading.Thread(target=long_Poll,args=(u,self))
t.start()
and
def long_Poll(client,con):
while True:
if len(client.unreadMessages) != 0:
print("IM GONNA RESPOND")
con.end_headers()
con.wfile.write(bytes(client.unreadMessages, "utf8"))
client.unreadMessages=[]
break
con.send_response(200)
con.end_headers()
And the logic behind this is that the servers want to do the long-polling, that is, it keeps GET/receive requests open in another busy-waiting thread. When any client sends a message to the server via POST/message it just adds this new message to other clients unseenMessages and so once their thread is running, they come out of the while True: loop, and the server gives them the new messages. In other words, I want to hold client's GET/receive open and not respond it as long as I want.
This process might take so long time. Maybe the chatroom is idle and there is no messages for a long time.
Right now the problem I have is that as soon as my client sends its first GET/receive message, it gets this error, even though I have set the timeout value in GET/receive request to be so much.
C:\Users\erfan\Desktop\web\client\venv\Scripts\python.exe C:\Users\erfan\Desktop\web\client\Client.py
Hossein
Welcome to chatroom Hossein ! Have a nice time !
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
chunked=chunked,
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1321, in getresponse
response.begin()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 296, in begin
version, status, reason = self._read_status()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 265, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\util\retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
chunked=chunked,
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1321, in getresponse
response.begin()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 296, in begin
version, status, reason = self._read_status()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 265, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\erfan\AppData\Local\Programs\Python\Python37\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\erfan\Desktop\web\client\Client.py", line 13, in coms
resp = requests.get("http://localhost:8081/receive", headers=headers,timeout=1000000)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "C:\Users\erfan\Desktop\web\client\venv\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
===========================================================================
UPDATE:
The strange part is whenever I edit the GET/receive module to this:
def do_GET(self):
while True:
pass
everything works fine.
But when I do :
def do_GET(self):
t=threading.Thread(target=long_Poll,args=(self))
t.start()
def long_Poll(con):
client =None
while True:
pass
It gives the same error to the client!
I mean the problem is because I pass the self object to another function to respond? maybe it interrupts the connection? I remember having the same problem in Java socket programming where I would encounter to some bugs sometimes when I wanted to use a socket to communicate in two functions? However, here I only want to communicate in the long-polling function not anywhere else.
=======================================
update:
I also put my server and client code here. For brevity, I post the paste.ubuntu links here.
Client:
https://paste.ubuntu.com/p/qJmRjYy4Y9/
Server:
https://paste.ubuntu.com/p/rVyHPs4Rjz/
First time a client types, he enters his name and after that he starts sending GET/receive requests. Client can then send his messages to other people by sending POST/message requests. Any time a user send a message to the server, the server finds him (by his auth) and updates all other clients unseenMessages so that whenever their long-polling thread continued, they'll get the new messages and their clients also send another GET/receive message immediately.

I have found the answer. I was trying to have a multithreaded server using single thread syntax!
I followed this thread for having a multithreaded HTTP server
Multithreaded web server in python

Python3.5 Asyncio - Preventing task exception from dumping to stdout?

I have a textbased interface (asciimatics module) for my program that uses asyncio and discord.py module and occasionally when my wifi adapter goes down I get an exception like so:
Task exception was never retrieved
future: <Task finished coro=<WebSocketCommonProtocol.run() done, defined at /home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py:428> exception=ConnectionResetError(104, 'Connection reset by peer')>
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 434, in run
msg = yield from self.read_message()
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 456, in read_message
frame = yield from self.read_data_frame(max_size=self.max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 511, in read_data_frame
frame = yield from self.read_frame(max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 546, in read_frame
self.reader.readexactly, is_masked, max_size=max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/framing.py", line 86, in read_frame
data = yield from reader(2)
File "/usr/lib/python3.5/asyncio/streams.py", line 670, in readexactly
block = yield from self.read(n)
File "/usr/lib/python3.5/asyncio/streams.py", line 627, in read
yield from self._wait_for_data('read')
File "/usr/lib/python3.5/asyncio/streams.py", line 457, in _wait_for_data
yield from self._waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/selector_events.py", line 662, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
This exception is non-fatal and the program is able to re-connect despite it - what I want to do is prevent this exception from dumping to stdout and mucking up my text interface.
I tried using ensure_future to handle it but it doesn't seem to work. Am I missing something:
#asyncio.coroutine
def handle_exception():
try:
yield from WebSocketCommonProtocol.run()
except Exception:
print("SocketException-Retrying")
asyncio.ensure_future(handle_exception())
#start discord client
client.run(token)

Task exception was never retrieved - is not actually exception propagated to stdout, but a log message that warns you that you never retrieved exception in one of your tasks. You can find details here.
I guess, most easy way to avoid this message in your case is to retrieve exception from task manually:
coro = WebSocketCommonProtocol.run() # you don't need any wrapper
task = asyncio.ensure_future(coro)
try:
#start discord client
client.run(token)
finally:
# retrieve exception if any:
if task.done() and not task.cancelled():
task.exception() # this doesn't raise anything, just mark exception retrieved

The answer provided by Mikhail is perfectly acceptable, but I realized it wouldn't work for me since the task that is raising the exception is buried deep in some module so trying to retrieve it's exception is kind've difficult. I found that instead if I simply set a custom exception handler for my asyncio loop (loop is created by the discord client):
def exception_handler(loop,context):
print("Caught the following exception")
print(context['message'])
client.loop.set_exception_handler(exception_handler)
client.run(token)

Flask/http- tell the client that the request needs some more time to complete

I have a web service(REST) where one request might take up to 30 sec to return an answer (lots of calculation). There is a risk, that during the calculation, the client webbrowser aborts(?) the existing connection and retries. Here is the console-output of the server-side:
Exception happened during processing of request from ('127.0.0.1', 53209)
Traceback (most recent call last):
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 290, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 318, in process_request
self.finish_request(request, client_address)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 654, in __init__
self.finish()
File "C:\Users\tmx\Anaconda2\lib\SocketServer.py", line 713, in finish
self.wfile.close()
File "C:\Users\tmx\Anaconda2\lib\socket.py", line 283, in close
self.flush()
File "C:\Users\tmx\Anaconda2\lib\socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
One option is what I thought of is to somehow notify the client that "I'm alivem but the request is still needs some more time", or to somehow set the timeout on server side. What are the possibilities?

It's difficult to run code in Flask after you've already returned some data. Your options are to either use something like a task queue (see Celery), or to yield your response in multiple parts.
Views in Flask can return strings, but they can also return iterables that contain strings. So you could return "abc", ["abc"], or a generator that will yield "abc". If you do your processing between yields, data will get sent to the client while the request is still running.
Take a look at the following example:
def generator_that_does_the_calculation():
sleep(1)
yield "I'm alive, but I need some time\n"
sleep(1)
yield "Still alive here\n"
sleep(1)
yield "Done\n"
#app.route('/calculate')
def calculate():
return Response(generator_that_does_the_calculation())

What can shut down a websocket connection?

I use websocket_server in order to provide a one way (server to client) websocket connection.
I have several threads on the server which query at given intervals (while True: ... time.sleep(60)) an API and then perform a server.send_message() call to update the client. All of this works fine.
From time to time, without any particular reason, I get a crash:
Exception in thread Thread-3:
Traceback (most recent call last):
File "C:\Python35\lib\threading.py", line 914, in _bootstrap_inner
self.run()
File "C:\Python35\lib\threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "D:/Dropbox/dev/domotique/webserver.py", line 266, in calendar
server.send_message(client, json.dumps({"calendar": events}))
File "C:\Python35\lib\site-packages\websocket_server\websocket_server.py", line 71, in send_message
self._unicast_(client, msg)
File "C:\Python35\lib\site-packages\websocket_server\websocket_server.py", line 119, in _unicast_
to_client['handler'].send_message(msg)
File "C:\Python35\lib\site-packages\websocket_server\websocket_server.py", line 194, in send_message
self.send_text(message)
File "C:\Python35\lib\site-packages\websocket_server\websocket_server.py", line 240, in send_text
self.request.send(header + payload)
BrokenPipeError: [WinError 10058] A request to send or receive data was disallowed because the socket had already been shut down in that direction with a previous shutdown call
There is no shutdown call in my code. What else can shut a websocket down?

The WebSocket client can ask the server to close the connection (or directly close it). From the library's code:
if not b1:
logger.info("Client closed connection.")
self.keep_alive = 0
return
if opcode == CLOSE_CONN:
logger.info("Client asked to close connection.")
self.keep_alive = 0
return
You could check self.keep_alive to know if the socket is still open.

python xmlrpc timeout error

I am using xmlrpc to contact a local server. On the client side, Sometimes the following socket timeout error and happens and its not a consistent error.
Why is it happening? What could be the reason for socket timeout?
<class 'socket.timeout'>: timed out
args = ('timed out',)
errno = None
filename = None
message = 'timed out'
strerror = None
Traceback on the server side is as follows
Exception happened during processing of request from ('127.0.0.1', 34855)
Traceback (most recent call last):
File "/usr/lib/python2.4/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.4/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.4/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.4/SocketServer.py", line 521, in __init__
self.handle()
File "/usr/lib/python2.4/BaseHTTPServer.py", line 314, in handle
self.handle_one_request()
File "/usr/lib/python2.4/BaseHTTPServer.py", line 308, in handle_one_request
method()
File "/usr/lib/python2.4/SimpleXMLRPCServer.py", line 441, in do_POST
self.send_response(200)
File "/usr/lib/python2.4/BaseHTTPServer.py", line 367, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.4/BaseHTTPServer.py", line 373, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.4/socket.py", line 256, in write
self.flush()
File "/usr/lib/python2.4/socket.py", line 243, in flush
self._sock.sendall(buffer)
error: (32, 'Broken pipe')
I killed the server and restarted it. Its working fine now.
What could be the reason?
My machine's RAM went full yesterday night by a process and came back to normal today morning.
Will this error be because of some swapping of processes?

Looks like the client socket it timing out waiting for the server to respond. Is it possible that your server might take a lot time to respond some times? Also, if the server is causing the machine to go into swap, that would slow it down making a timeout possible.
If I remember right, socket timeout is not set in xmlrpc in python. Are you doing socket.setdefaulttimeout somewhere in your code?
If it is expected that your server will take time once in a while, then you could set a higher timeout value using above.
HTH