I am new to Python 3 and am playing around with asyncio. Thereby, I am experiencing a strange behavior with the following server-side code:
import asyncio
#asyncio.coroutine
def handle_client(reader, writer):
print('Client connected.')
client_connected = True
while client_connected:
print('Waiting for client event.')
line = yield from reader.readline()
if line:
print('Got: {}'.format(line))
if line.decode() == 'echo\n':
print('Sending back echo.')
writer.write(line)
else:
print('Not sending back anything.')
else:
print('Client disconnected.')
client_connected = False
if __name__ == '__main__':
asyncio.async(asyncio.start_server(handle_client, 'localhost', 8888))
asyncio.get_event_loop().run_forever()
When I run this client code (EDIT: client code is entered manually into an IPython session, the server definitely has time to write before I close the socket)...
import socket
client = socket.create_connection(('localhost', 8888))
client.sendall('echo\n'.encode())
client.close()
... I get an error traceback from the server:
C:\Users\Gnar\Anaconda3\python.exe C:/Users/Gnar/Code/echo.py
Client connected.
Waiting for client event.
Got: b'echo\n'
Sending back echo.
Waiting for client event.
Task exception was never retrieved
future: <Task finished coro=<handle_client() done, defined at C:/Users/Gnar/Code/echo.py:4> exception=ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)>
Traceback (most recent call last):
File "C:\Users\Gnar\Anaconda3\lib\asyncio\tasks.py", line 234, in _step
result = coro.throw(exc)
File "C:/Users/Gnar/Code/echo.py", line 10, in handle_client
line = yield from reader.readline()
File "C:\Users\Gnar\Anaconda3\lib\asyncio\streams.py", line 425, in readline
yield from self._wait_for_data('readline')
File "C:\Users\Gnar\Anaconda3\lib\asyncio\streams.py", line 393, in _wait_for_data
yield from self._waiter
File "C:\Users\Gnar\Anaconda3\lib\asyncio\futures.py", line 386, in __iter__
yield self # This tells Task to wait for completion.
File "C:\Users\Gnar\Anaconda3\lib\asyncio\tasks.py", line 287, in _wakeup
value = future.result()
File "C:\Users\Gnar\Anaconda3\lib\asyncio\futures.py", line 275, in result
raise self._exception
File "C:\Users\Gnar\Anaconda3\lib\asyncio\selector_events.py", line 662, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
The issue must be somehow in relation with writer.write, because when I call the following client code (which makes the server skip the writing), there is no error:
import socket
client = socket.create_connection(('localhost', 8888))
client.sendall('foo\n'.encode())
client.close()
The corresponding server log:
C:\Users\Gnar\Anaconda3\python.exe C:/Users/Gnar/Code/echo.py
Client connected.
Waiting for client event.
Got: b'foo\n'
Not sending back anything.
Waiting for client event.
Client disconnected.
What am I missing? Am I using asyncio incorrectly?
Thanks!
You're getting the exception because you're trying to write some data back to the client on the server-side, but the client is closing the socket immediately after sending in the 'echo', without actually receiving the response from the server. If a socket connection is closed while there is unreceived data on the wire, you'll get an error on the sending side, so that you know the remote side may not have received whatever you last sent.
The problem goes away if you add a call to socket.recv(1024) on the client-side prior to calling socket.close(), so that the client waits for a response from the server before closing the socket. You could also just use a try/except around the write call on the server-side if you just want to gracefully handle the exception, even when the client does the wrong thing.
Related
I have a Flask-Mqtt client running, and everything is fine if broker is accessible. The problem starts when the MQTT broker is not accessible. Usually, I would be able to handle the failed connections with exceptions on the on_connect function, however I don't quite understand how to implement it with Flask-Mqtt
I would want to handle failed broker connections while allowing the web server to run the web pages.
Documentations and example of Flask-MQTT: https://flask-mqtt.readthedocs.io/en/latest/usage.html
The code for handling broker connection failure (based on Steve's Internet Guide)
def on_connect(client, userdata, flags, rc):
if rc==0:
client.connected_flag=True #set flag
print("connected OK")
else:
print("Bad connection Returned code=",rc)
client.bad_connection_flag=True
The error on Flask app when broker not connected:
C:\Users\USER\Documents\College\FYP1\flask_testing\venv\Scripts\python.exe C:/Users/USER/Documents/College/FYP1/flask_testing/main.py
Traceback (most recent call last):
File "C:/Users/USER/Documents/College/FYP1/flask_testing/main.py", line 37, in <module>
mqtt = Mqtt(app)
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\flask_mqtt\__init__.py", line 104, in __init__
self.init_app(app)
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\flask_mqtt\__init__.py", line 183, in init_app
self._connect()
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\flask_mqtt\__init__.py", line 209, in _connect
res = self.client.connect(
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\paho\mqtt\client.py", line 941, in connect
return self.reconnect()
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\paho\mqtt\client.py", line 1075, in reconnect
sock = self._create_socket_connection()
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\paho\mqtt\client.py", line 3546, in _create_socket_connection
return socket.create_connection(addr, source_address=source, timeout=self._keepalive)
File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\socket.py", line 807, in create_connection
raise err
File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\socket.py", line 796, in create_connection
sock.connect(sa)
socket.timeout: timed out
I faced a similar error. Everything was working fine when the mqtt server was running but if I started the app with the server switched off, the app failed with the same message than yours.
I finally solved it by using connect_async parameter when creating the MQTT object instance.
mqtt_client = Mqtt(app, connect_async=True)
With that parameter, the web pages will be displayed and, when the mqtt server is available, it will automatically connect.
I am asking where either my though process or my code is incorrect relative to using asyncio client streams to send data and receive responses from a server. When I call the method that disconnects the client an exception is thrown. I am learning python asyncio and ran across exceptions during testing trying to close the client connection. I am trying to 1). Create a client connection to a server 2). leave the client connection open so that it can be used across multiple send/receive cycles 3). close the client connection gracefully when complete.
This is the class that contains the asyncio methods to create the stream writer.
class hl7_client_sender:
SB = b'\x1B'
EB = b'\x1C'
CR = b'\x0D'
def __init__(self,address,port,timeout=-1,retry=3.0):
self._resend=0
self._timeout= timeout
self._retry = retry
#self._reader, self._writer = await asyncio.open_connection(address,port)
self._address = address
self._port = port
self._writer = None
self._reader = None
async def connect(self):
self._reader, self._writer = await asyncio.open_connection(self._address,self._port)
async def disconnect(self):
await self._writer.wait_closed()
and this is the code in my driver where the exception occurs during the call to disconnect
#test send and respond
import asyncio
import string
import unicodedata
import simple_hl7_client
import time
##open a connectino sleep 5 seconds then close###
myclient = simple_hl7_client.hl7_client_sender('192.168.226.128',54321)
asyncio.run(myclient.connect())
time.sleep(3)
asyncio.run(myclient.disconnect())
The exception occurs during the call to asycnio.run(myclient.disconnect())
This is the exception:
Traceback (most recent call last):
File ".\test_simple_hl7_client.py", line 11, in <module>
asyncio.run(myclient.disconnect())
File "C:\Users\billg\AppData\Local\Programs\Python\Python37\lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Users\billg\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 583, in run_until_complete
return future.result()
File "D:\data\FromOldPC\code\ASYNCIOTESTING\simple_hl7_client.py", line 23, in disconnect
self._writer.close()
File "C:\Users\billg\AppData\Local\Programs\Python\Python37\lib\asyncio\streams.py", line 317, in close
return self._transport.close()
File "C:\Users\billg\AppData\Local\Programs\Python\Python37\lib\asyncio\selector_events.py", line 663, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Users\billg\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 687, in call_soon
self._check_closed()
File "C:\Users\billg\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 479, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Is there any code snippet sample to achieve the following requirement,
python-asyncio TCP Server listen for TCP client and manage open connection simultaneously to another remote TCP server.
the tcp server code that I've tried so far, based on: http://asyncio.readthedocs.io/en/latest/tcp_echo.html1
HOST, PORT = '127.0.0.1', 8888
VHOST, VPORT = "localhost", 9999
vsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
vsock.connect((VHOST, VPORT))
async def send_message(message, vsock):
receive = None
try:
vsock.sendall(message)
received = vsock.recv(1024)
except:
print("Unexpected error:", sys.exc_info()[0])
raise
return received
async def handle_echo(reader, writer):
task = loop.create_task(send_message(data, vsock))
response = await task
writer.write(response)
await writer.drain()
writer.close()
loop = asyncio.get_event_loop()
coro = asyncio.start_server(handle_echo, HOST, PORT, loop=loop)
server = loop.run_until_complete(coro)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
vsock.close()
It works on first request, but getting blank message in 2nd request, and getting error the next request
Unexpected error: <class 'BrokenPipeError'>
Task exception was never retrieved
future: <Task finished coro=<handle_echo() done, defined at socket_server.py:27> exception=BrokenPipeError(32, 'Broken pipe')>
Traceback (most recent call last):
File "/home/hery/.pyenv/versions/3.5.1/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "socket_server.py", line 34, in handle_echo
response = await task
File "/home/hery/.pyenv/versions/3.5.1/lib/python3.5/asyncio/futures.py", line 358, in __iter__
yield self # This tells Task to wait for completion.
File "/home/hery/.pyenv/versions/3.5.1/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
future.result()
File "/home/hery/.pyenv/versions/3.5.1/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/home/hery/.pyenv/versions/3.5.1/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "socket_server.py", line 18, in send_message
vsock.sendall(message)
BrokenPipeError: [Errno 32] Broken pipe
I use websocket_server in order to provide a one way (server to client) websocket connection.
I have several threads on the server which query at given intervals (while True: ... time.sleep(60)) an API and then perform a server.send_message() call to update the client. All of this works fine.
From time to time, without any particular reason, I get a crash:
Exception in thread Thread-3:
Traceback (most recent call last):
File "C:\Python35\lib\threading.py", line 914, in _bootstrap_inner
self.run()
File "C:\Python35\lib\threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "D:/Dropbox/dev/domotique/webserver.py", line 266, in calendar
server.send_message(client, json.dumps({"calendar": events}))
File "C:\Python35\lib\site-packages\websocket_server\websocket_server.py", line 71, in send_message
self._unicast_(client, msg)
File "C:\Python35\lib\site-packages\websocket_server\websocket_server.py", line 119, in _unicast_
to_client['handler'].send_message(msg)
File "C:\Python35\lib\site-packages\websocket_server\websocket_server.py", line 194, in send_message
self.send_text(message)
File "C:\Python35\lib\site-packages\websocket_server\websocket_server.py", line 240, in send_text
self.request.send(header + payload)
BrokenPipeError: [WinError 10058] A request to send or receive data was disallowed because the socket had already been shut down in that direction with a previous shutdown call
There is no shutdown call in my code. What else can shut a websocket down?
The WebSocket client can ask the server to close the connection (or directly close it). From the library's code:
if not b1:
logger.info("Client closed connection.")
self.keep_alive = 0
return
if opcode == CLOSE_CONN:
logger.info("Client asked to close connection.")
self.keep_alive = 0
return
You could check self.keep_alive to know if the socket is still open.
I'm having some difficulty attempting to create an SSL socket in Python to use a proxy that requires authentication. I am very sorry for the length, but I felt it was best to include as much detail as possible.
First, the server code looks like this:
class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
def __init__(self, server_address, RequestHandlerClass, client_manager, recv_queue):
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass, bind_and_activate=True)
<snipped out extra code>
class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):
def setup(self):
while True:
try:
print 'trying to wrap in ssl'
self.request = ssl.wrap_socket(self.request,
certfile=(os.getcwd() + '/ssl_certs/newcert.pem'),
keyfile=(os.getcwd() + '/ssl_certs/webserver.nopass.key'),
server_side=True,
cert_reqs=ssl.CERT_NONE,
ssl_version=ssl.PROTOCOL_TLSv1,
do_handshake_on_connect=False,
suppress_ragged_eofs=True)
break
except Exception, ex:
print 'error trying to wrap in ssl %s' % ex
def handle(self):
# Display message that client has connected
print '\r[*] Received connection from %s:%s\r' % (self.client_address[0], self.client_address[1])
while self.stopped() == False:
recv_msg = self.request.read(1024)
if recv_msg == '':
self.stop.set()
server.recv_queue.put(recv_msg)
break
else:
server.recv_queue.put(recv_msg)
if self.stopped():
print '[!] Received STOP signal from %s:%s; Exiting!' % (self.client_address[0], self.client_address[1])
Second, this is the client code where I set up the information needed to connect via the proxy that requires authentication:
class proxyCommsHandler():
def __init__(self, user, password, remote_host, remote_port, list_of_proxies):
# information needed to connect
self.user = 'username'
self.passwd = 'password'
self.remote_host = 'remote_host_ip'
self.remote_port = 8008
self.list_of_proxies = [['proxyserver.hostname.com', 8080]]
# setup basic authentication to send to the proxy when we try to connect
self.user_pass = base64.encodestring(self.user + ':' + self.passwd)
self.proxy_authorization = 'Proxy-authorization: Basic ' + self.user_pass + '\r\n'
self.proxy_connect = 'CONNECT %s:%s HTTP/1.1\r\n' % (self.remote_host, self.remote_port)
self.user_agent = "User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1\r\n"
self.proxy_pieces = self.proxy_connect + self.proxy_authorization + self.user_agent + '\r\n'
Now, here's where I initially connect to the proxy, where I get no errors (I get a '200' status code):
self.proxy = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.proxy.connect( (proxy_host, proxy_port) )
self.proxy.sendall(self.proxy_pieces)
self.response = proxy.recv(1024)
Here's where the client fails (I think). I try to take self.proxy and wrap it in SSL, like this:
sslsock = ssl.wrap_socket(self.proxy, server_side=False, do_handshake_on_connect=True,
ssl_version=ssl.PROTOCOL_TLSv1)
This is the error that I see on the client:
Traceback (most recent call last):
File "C:\Python27\pyrevshell.py", line 467, in <module>
proxyCommsHandler(None, None, None, None, list_of_proxies).run()
File "C:\Python27\pyrevshell.py", line 300, in run
ssl_version=ssl.PROTOCOL_TLSv1)
File "C:\Python27\lib\ssl.py", line 372, in wrap_socket
ciphers=ciphers)
File "C:\Python27\lib\ssl.py", line 134, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 296, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol
The client does connect, like shown from the output here:
trying to wrap in ssl
[*] Received connection from x.x.x.x:47144
[*] x.x.x.x:47144 added to the client list
But then it's immediately followed by an exception:
----------------------------------------
Exception happened during processing of request from ('x.x.x.x', 47144)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 582, in process_request_thread
self.finish_request(request, client_address)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 639, in __init__
self.handle()
File "shell_server.py", line 324, in handle
recv_msg = self.request.read(1024)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 138, in read
return self._sslobj.read(len)
SSLError: [Errno 1] _ssl.c:1348: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
----------------------------------------
While I realize that this sounds like an obvious problem based on the Exceptions that were thrown, the interesting parts are that:
I successfully connect initially via the proxy as shown above
I can successfully connect with a web browser behind behind the same proxy, and no exceptions are thrown; I can pass data to the browser
I've tried different SSL Protocol versions on both the server and client side, as shown in the table here in the Python documentation; it errors on the client-side each time
I've used Wireshark on both ends of the connection. While using a normal browser and connecting to the server, I see the entire SSL handshake and negotiation process, and everything runs smoothly.
However, when I use the client shown above, as soon as I connect with it, I see the client send a Client Hello message, but then my server sends a RST packet to kill the connection (I haven't determined if this is before or after the Exception is thrown).
Again, I apologize for the length, but I am in dire need of expert advice.
I've figured out the issue to my problem. I am sending the self.user_agent to the remote host when I connect via the proxy for the first time, which interferes with the SSL Handshake.
To solve this, I put an initial self.request.recv() in the def setup(self) function before I call ssl.wrap_socket on the socket.