Maybe I'm missing something here in the asynchronous designs of Twisted, but I can't seem to find a way to call the sendMessage() method "externaly". By this I mean, sending messages without being solely at the callback methods of Twisted/AutobahnWebsockets (like at onOpen or when receiving data from server at onMessage())
Of course I could launch a thread and call my_protocol_instance.sendMessage("hello") but that would defeat every purpose of the asynchronous design right?
In a concrete example, I need to have a top wrapper class which opens the connection and manages it, and whenever I need I call my_class.send_my_toplevel_message(msg). How can I implement this?
Hope I've been clear on my explanation.
Thanks
Why do you need a thread to launch protocolInstance.sendMessage() ?
This can be done in a normal reactor loop.
The core of a twisted is reactor and it gives a much easier look at things when you consider twisted itself reactive - meaning it does something as a reaction (response) to something else.
Now I assume that the thread you are talking about, also gets created and made in calling sendMessage because of certain events or activity or status. I can hardly imagine a case where you would just need to send a message out of the blue without any reason to react.
If however there is an event which should trigger sendMessage, there is no need to invoke that in thread: just use twisted mechanisms for catching that event and then calling sendMessage from that particular event's callback.
Now on to your concrete example: can you specify what "whenever I need" means exactly in the context of this question? An input from another connection? An input from the user? Looping activity?
I managed to implement what I needed by running Twisted in another thread, keeping my program free to run and allowing it to trigger send data in Twisted with reactor.callFromThread().
What do you think?
# ----- twisted ----------
class _WebSocketClientProtocol(WebSocketClientProtocol):
def __init__(self, factory):
self.factory = factory
def onOpen(self):
log.debug("Client connected")
self.factory.protocol_instance = self
self.factory.base_client._connected_event.set()
class _WebSocketClientFactory(WebSocketClientFactory):
def __init__(self, *args, **kwargs):
WebSocketClientFactory.__init__(self, *args, **kwargs)
self.protocol_instance = None
self.base_client = None
def buildProtocol(self, addr):
return _WebSocketClientProtocol(self)
# ------ end twisted -------
class BaseWBClient(object):
def __init__(self, websocket_settings):
self.settings = websocket_settings
# instance to be set by the own factory
self.factory = None
# this event will be triggered on onOpen()
self._connected_event = threading.Event()
# queue to hold not yet dispatched messages
self._send_queue = Queue.Queue()
self._reactor_thread = None
def connect(self):
log.debug("Connecting to %(host)s:%(port)d" % self.settings)
self.factory = _WebSocketClientFactory(
"ws://%(host)s:%(port)d" % self.settings,
debug=True)
self.factory.base_client = self
c = connectWS(self.factory)
self._reactor_thread = threading.Thread(target=reactor.run,
args=(False,))
self._reactor_thread.daemon = True
self._reactor_thread.start()
def send_message(self, body):
if not self._check_connection():
return
log.debug("Queing send")
self._send_queue.put(body)
reactor.callFromThread(self._dispatch)
def _check_connection(self):
if not self._connected_event.wait(timeout=10):
log.error("Unable to connect to server")
self.close()
return False
return True
def _dispatch(self):
log.debug("Dispatching")
while True:
try:
body = self._send_queue.get(block=False)
except Queue.Empty:
break
self.factory.protocol_instance.sendMessage(body)
def close(self):
reactor.callFromThread(reactor.stop)
Related
I have code (python class, I'll call it Arduino) that writes control packets to a serial port (using serial_asyncio), and the serial port replies with confirmation packets. However the remote device also sends event packets to the python side randomly. I want the class to provide the decoded packets to the class that instantiates my class (I'll call it Controller). I am confused on how to handle this.
My first thought is to provide callback to the Arduino class:
class Controller:
def __init__(self):
self.arduino = Arduino("/dex/ttyS0", self._serialPacketCallback)
def _serialPacketCallback(self, obj: dict):
pass # handle spontaneous event packet from remote
But this does not seem very async-y. What is the asyncio way to do this? I think this would look like:
class Controller:
def __init__(self):
self.arduino = Arduino("/dex/ttyS0")
async readEventPacket(self):
pass
#somewhere, not sure where, or how to start it:
async def _handleEvents(self)
while True:
event = await self._readEventPacket()
async def start(self):
await self.arduino.start()
await asyncio.wait([self._handleEvents()])
if __name__ == '__main__':
controller = Controller()
loop = asyncio.get_event_loop()
loop.create_task(controller.start())
loop.run_forever()
I've looked around and I've seen suggestions of using callbacks, multi-processing pipes, additional event loops, and I am sure they work, but I'm not sure what the proper approach is. For me, I don't want to start any additional event loops or threads, leading me to think the callback is best, but that is not very async, and I would like to know how to do this as async, without additional event loops or callbacks.
An additional concern that I want to articulate is that I'd like this as loosely coupled as the Arduino class will be used in other controllers.
Side note: I am not sure when in Python a new event loop is required to be created?
Another question: how does the Arduino class generate an event and have Controller pick it up in await self._readEventPacket()?
The nice thing about asyncio is that you can always convert a callback-based interface to a coroutine-based one, and vice versa.
Let's assume your Arduino class implements a callback-based interface, like this (untested):
class Arduino:
def __init__(self, device, callback):
self._device = device
self._callback = callback
async def start(self):
reader, writer = await serial_asyncio.connect(url=self._device, baudrate=115200)
while True:
data = await reader.read(1024)
self._callback(data)
You can convert that interface into a coroutine-based one by using a queue:
def callback_to_coro():
# Return two functions, one that can be passed as callback to
# code that expects it, and the other a coroutine that can be
# awaited to get the values the callback was invoked with.
queue = asyncio.Queue()
return queue.put_nowait, queue.get
With that code you can implement Controller.read_event_packet like this:
class Controller:
def __init__(self):
callback, wait = callback_to_coro()
self.arduino = Arduino("/dex/ttyS0", callback)
self.read_event_packet = wait
I have written a python script that idles waiting for gmail to push a notification indicating that an email has been received. Then the contents are parsed and a database is searched, then database data is emailed back to the original sender.
Currently once an email is received, the script cannot process another email until the script has emailed back. I am wondering if there is a way for the script to be continually listening for an email. As at this point in time if two emails are received at similar times the second one will not be processed.
I think multi-threading might be a solution but I am not sure. Possibly create a new thread for the processEmail.py section of code below?
Sorry if I have explained badly, I am struggling to explain this adequately, feel free to ask for more information.
EDIT: Instead of down voting me could you help me by commenting telling me what more information I should include?
EDIT 2: Here is a code example, I am trying to have the ability to still listen for an email while an email is being processed in processEmail.py
import imaplib2
import time
import subprocess
from threading import *
from subprocess import call
import processEmail
class Idler(object):
def __init__(self, conn):
self.thread = Thread(target=self.idle)
self.M = conn
self.event = Event()
def start(self):
self.thread.start()
def stop(self):
self.event.set()
def join(self):
self.thread.join()
def idle(self):
while True:
if self.event.isSet():
return
self.needsync = False
def callback(args):
if not self.event.isSet():
self.needsync = True
self.event.set()
self.M.idle(callback=callback)
self.event.wait()
if self.needsync:
self.event.clear()
self.dosync()
def dosync(self):
print "An email has been received, please wait...\n"
self.execute()
def execute(self):
processEmail.main()
M = imaplib2.IMAP4_SSL("imap.gmail.com")
M.login("email_address","email_pass")
M.select("Folder")
idler = Idler(M)
idler.start()
x = False
while not x: time.sleep(0.1)
You can directly inherit from threading.Thread and override its run method:
class SomeTask(threading.Thread):
def run(self):
# Will be executed in separate thread
Start the thread via its start method.
Communication between threads should be handled via queues.
I'm extending socketserver.ThreadingMixIn in Python 3.4 to build my own threaded server while keeping the original callbacks overwintered only for logging porpoises.
The activation and creation is very simple and according to python documentation the problem i'm having is to stop that server with server.shutdown(). It frizzes and doesn't exit. I need way to shutdown that server without using ctrl-c because it will also involve some GUI to that server.
The basic server:
class ServerBasic(socketserver.ThreadingMixIn,socketserver.TCPServer):
logging.basicConfig(level=logging.DEBUG,format='%(name)s: %(message)s',)
def __init__(self, log_name,server_address, handler_class=ThreadedRequestHandler):
self.logger = logging.getLogger(log_name)
self.logger.debug('__init__')
socketserver.TCPServer.__init__(self, server_address, handler_class)
return
def server_activate(self):
self.logger.debug('server_activate')
socketserver.TCPServer.server_activate(self)
return
def serve_forever(self):
self.logger.debug('waiting for request')
self.logger.info('Handling requests, press <Ctrl-C> to quit')
while True:
self.handle_request()
return
The extending class:
class ManagerServer(PIRServerBasic):
def __init__(self, log_name, handler_class=T_ManagerRequestHandler):
self.tup_socket = (ipAddress, WELCOME_PORT) # tuple of the address and port
self.log_name = log_name
return ServerBasic.__init__(self, log_name, self.tup_socket, handler_class=handler_class)
And here how it all created and running:
o_serverManager = ManagerServer('Manager_Server', T_ManagerRequestHandler)
t_managerServer = threading.Thread(target=o_serverManager.serve_forever)
t_managerServer.daemon = True
t_managerServer.start()
sleep(15)
o_serverManager.shutdown()
After the shutdown command the program is stuck.
In your overwrite of the serve_forever method, you have removed the condition that breaks the while loop on a shutdown request. The original method looks like:
def serve_forever(self, poll_interval=0.5):
"""Handle one request at a time until shutdown.
Polls for shutdown every poll_interval seconds. Ignores
self.timeout. If you need to do periodic tasks, do them in
another thread.
"""
self.__is_shut_down.clear()
try:
while not self.__shutdown_request:
r, w, e = _eintr_retry(select.select, [self], [], [],
poll_interval)
if self in r:
self._handle_request_noblock()
finally:
self.__shutdown_request = False
self.__is_shut_down.set()
You need to implement a similar system in your overwrite, to look for a set __shutdown_request flag and take the appropriate action. This also requires that your handler is non-blocking.
I am implementing a WebSockets server in Tornado 3.2. The client connecting to the server won't be a browser.
For cases in which there is back-and-forth communication between server and client, I would like to add a max. time the server will wait for a client response before closing the connection.
This is roughly what I've been trying:
import datetime
import tornado
class WSHandler(WebSocketHandler):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.timeout = None
def _close_on_timeout(self):
if self.ws_connection:
self.close()
def open(self):
initialize()
def on_message(self, message):
# Remove previous timeout, if one exists.
if self.timeout:
tornado.ioloop.IOLoop.instance().remove_timeout(self.timeout)
self.timeout = None
if is_last_message:
self.write_message(message)
self.close()
else:
# Add a new timeout.
self.timeout = tornado.ioloop.IOLoop.instance().add_timeout(
datetime.timedelta(milliseconds=1000), self._close_on_timeout)
self.write_message(message)
Am I being a klutz and is there a much simpler way of doing this? I can't even seem to schedule a simple print statement via add_timeout above.
I also need some help testing this. This is what I have so far:
from tornado.websocket import websocket_connect
from tornado.testing import AsyncHTTPTestCase, gen_test
import time
class WSTests(AsyncHTTPTestCase):
#gen_test
def test_long_response(self):
ws = yield websocket_connect('ws://address', io_loop=self.io_loop)
# First round trip.
ws.write_message('First message.')
result = yield ws.read_message()
self.assertEqual(result, 'First response.')
# Wait longer than the timeout.
# The test is in its own IOLoop, so a blocking sleep should be okay?
time.sleep(1.1)
# Expect either write or read to fail because of a closed socket.
ws.write_message('Second message.')
result = yield ws.read_message()
self.assertNotEqual(result, 'Second response.')
The client has no problem writing to and reading from the socket. This is presumably because the add_timeout isn't firing.
Does the test need to yield somehow to allow the timeout callback on the server to run? I would have thought not since the docs say the tests run in their own IOLoop.
Edit
This is the working version, per Ben's suggestions.
import datetime
import tornado
class WSHandler(WebSocketHandler):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.timeout = None
def _close_on_timeout(self):
if self.ws_connection:
self.close()
def open(self):
initialize()
def on_message(self, message):
# Remove previous timeout, if one exists.
if self.timeout:
tornado.ioloop.IOLoop.current().remove_timeout(self.timeout)
self.timeout = None
if is_last_message:
self.write_message(message)
self.close()
else:
# Add a new timeout.
self.timeout = tornado.ioloop.IOLoop.current().add_timeout(
datetime.timedelta(milliseconds=1000), self._close_on_timeout)
self.write_message(message)
The test:
from tornado.websocket import websocket_connect
from tornado.testing import AsyncHTTPTestCase, gen_test
import time
class WSTests(AsyncHTTPTestCase):
#gen_test
def test_long_response(self):
ws = yield websocket_connect('ws://address', io_loop=self.io_loop)
# First round trip.
ws.write_message('First message.')
result = yield ws.read_message()
self.assertEqual(result, 'First response.')
# Wait a little more than the timeout.
yield gen.Task(self.io_loop.add_timeout, datetime.timedelta(seconds=1.1))
# Expect either write or read to fail because of a closed socket.
ws.write_message('Second message.')
result = yield ws.read_message()
self.assertEqual(result, None)
The timeout-handling code in your first example looks correct to me.
For testing, each test case gets its own IOLoop, but there is only one IOLoop for both the test and anything else it runs, so you must use add_timeout instead of time.sleep() here as well to avoid blocking the server.
Ey Ben, I know this question was long ago resolved, but I wanted to share with any user reading this the solution I made for this.
It's basically based on yours, but it solves the problem from an external Service that can be easily integrated within any websocket using composition instead of inheritance:
class TimeoutWebSocketService():
_default_timeout_delta_ms = 10 * 60 * 1000 # 10 min
def __init__(self, websocket, ioloop=None, timeout=None):
# Timeout
self.ioloop = ioloop or tornado.ioloop.IOLoop.current()
self.websocket = websocket
self._timeout = None
self._timeout_delta_ms = timeout or TimeoutWebSocketService._default_timeout_delta_ms
def _close_on_timeout(self):
self._timeout = None
if self.websocket.ws_connection:
self.websocket.close()
def refresh_timeout(self, timeout=None):
timeout = timeout or self._timeout_delta_ms
if timeout > 0:
# Clean last timeout, if one exists
self.clean_timeout()
# Add a new timeout (must be None from clean).
self._timeout = self.ioloop.add_timeout(
datetime.timedelta(milliseconds=timeout), self._close_on_timeout)
def clean_timeout(self):
if self._timeout is not None:
# Remove previous timeout, if one exists.
self.ioloop.remove_timeout(self._timeout)
self._timeout = None
In order to use the service, itś as easy as create a new TimeoutWebService instance (optionally with the timeout in ms, as well as the ioloop where it should be executed) and call the method ''refresh_timeout'' to either set the timeout for the first time or reset an already existing timeout, or ''clean_timeout'' to stop the timeout service.
class BaseWebSocketHandler(WebSocketHandler):
def prepare(self):
self.timeout_service = TimeoutWebSocketService(timeout=(1000*60))
## Optionally starts the service here
self.timeout_service.refresh_timeout()
## rest of prepare method
def on_message(self):
self.timeout_service.refresh_timeout()
def on_close(self):
self.timeout_service.clean_timeout()
Thanks to this approach, you can control when exactly and under which conditions you want to restart the timeout which might differ from app to app. As an example you might only want to refresh the timeout if user acomplish X conditions, or if the message is the expected one.
I hope ppl enjoy this solution !
I'm trying to write unittests for my application that uses Autobahn.
I want to test my controllers which gets received data from protocol, parses it and reacts to it.
But when my test comes to a point when protocol should be disconnected (self.sendClose) then it raises error
exceptions.AttributeError: 'MyProtocol' object has no attribute 'state'.
I was trying to makeConnection using proto_helpers.StringTransport but then I have errors too
exceptions.AttributeError: StringTransport instance has no attribute 'setTcpNoDelay'`
I'm using trial and I don't want to run dummy server/client for testing purposes only, because it's not recommended.
How should I write my tests so I can test functions that sends data, read data, disconnects etc. using fake connection and trial ?
It is difficult to say exactly what is going on without having a peek at MyProtocol class. The problem sounds a lot like it is caused by the fact that you are directly messing round with low level functions and therefore also the state attribute of WebSocket class, which is, well, a representation of the internal state of the WebSocket connection.
According to the autobahn reference doc, the APIs from the WebSicketProtocol that you could directly use and override are:
onOpen
onMessage
onClose
sendMessage
sendClose
Your approach of using the StringTransport to test your protocol is not ideal. The problem lays in the fact that MyProtocol is a tiny layer on top of the WebSocketProtocol framework provided by autobahn which, for better or worse, hides the details about managing the connection, the transport and the internal protocol state.
If you think about it, you want to test your stuff, not WebSocketProtocol and therefore if you do not want to embed a dummy server or client, your best bet is to test directly the methods that MyProtocol overrides.
An example of what I am saying is the following
class MyPublisher(object):
cbk=None
def publish(self, msg):
if self.cbk:
self.cbk(msg)
class MyProtocol(WebSocketServerProtocol):
def __init__(self, publisher):
WebSocketServerProtocol.__init__(self)
#Defining callback for publisher
publisher.cbk = self.sendMessage
def onMessage(self, msg, binary)
#Stupid echo
self.sendMessage(msg)
class NotificationTest(unittest.TestCase):
class MyProtocolFactory(WebSocketServerFactory):
def __init__(self, publisher):
WebSocketServerFactory.__init__(self, "ws://127.0.0.1:8081")
self.publisher = publisher
self.openHandshakeTimeout = None
def buildProtocol(self, addr):
protocol = MyProtocol(self.listener)
protocol.factory = self
protocol.websocket_version = 13 #Hybi version 13 is supported by pretty much everyone (apart from IE <8 and android browsers)
return protocol
def setUp(self):
publisher = task.LoopingCall(self.send_stuff, "Hi there")
factory = NotificationTest.MyProtocolFactory(listener)
protocol = factory.buildProtocol(None)
transport = proto_helpers.StringTransport()
def play_dumb(*args): pass
setattr(transport, "setTcpNoDelay", play_dumb)
protocol.makeConnection(transport)
self.protocol, self.transport, self.publisher, self.fingerprint_handler = protocol, transport, publisher, fingerprint_handler
def test_onMessage(self):
#Following 2 lines are the problematic part. Here you are manipulating explicitly a hidden state which your implementation should not be concerned with!
self.protocol.state = WebSocketProtocol.STATE_OPEN
self.protocol.websocket_version = 13
self.protocol.onMessage("Whatever")
self.assertEqual(self.transport.value()[2:], 'Whatever')
def test_push(self):
#Following 2 lines are the problematic part. Here you are manipulating explicitly a hidden state which your implementation should not be concerned with!
self.protocol.state = WebSocketProtocol.STATE_OPEN
self.protocol.websocket_version = 13
self.publisher.publish("Hi there")
self.assertEqual(self.transport.value()[2:], 'Hi There')
As you might have noticed, using the StringTransport here is very cumbersome. You must have knowledge of the underline framework and bypass its state management, something you don't really want to do. Unfortunately autobahn does not provide a ready-to-use test object that would permit easy state manipulation and therefore my suggestion of using dummy servers and clients is still valid
Testing your server WITH network
The test provided shows how you can test server push, asserting that what your are getting is what you expect, and using also a hook on how to determine when to finish.
The server protocol
from twisted.trial.unittest import TestCase as TrialTest
from autobahn.websocket import WebSocketServerProtocol, WebSocketServerFactory, WebSocketClientProtocol, WebSocketClientFactory, connectWS, listenWS
from twisted.internet.defer import Deferred
from twisted.internet import task
START="START"
class TestServerProtocol(WebSocketServerProtocol):
def __init__(self):
#The publisher task simulates an event that triggers a message push
self.publisher = task.LoopingCall(self.send_stuff, "Hi there")
def send_stuff(self, msg):
#this method sends a message to the client
self.sendMessage(msg)
def _on_start(self):
#here we trigger the task to execute every second
self.publisher.start(1.0)
def onMessage(self, message, binary):
#According to this stupid protocol, the server starts sending stuff when the client sends a "START" message
#You can plug other commands in here
{
START : self._on_start
#Put other keys here
}[message]()
def onClose(self, wasClean, code, reason):
#After closing the connection, we tell the task to stop sending messages
self.publisher.stop()
The client protocol and factory
Next class is the client protocol. It basically tells the server to start pushing messages. It calls the close_condition on them to see if it is time to close the connection and as a last thing, it calls the assertion function on the messages it received to see if the test was successful or not
class TestClientProtocol(WebSocketClientProtocol):
def __init__(self, assertion, close_condition, timeout, *args, **kwargs):
self.assertion = assertion
self.close_condition = close_condition
self._received_msgs = []
from twisted.internet import reactor
#This is a way to set a timeout for your test
#in case you never meet the conditions dictated by close_condition
self.damocle_sword = reactor.callLater(timeout, self.sendClose)
def onOpen(self):
#After the connection has been established,
#you can tell the server to send its stuff
self.sendMessage(START)
def onMessage(self, msg, binary):
#Here you get the messages pushed from the server
self._received_msgs.append(msg)
#If it is time to close the connection
if self.close_condition(msg):
self.damocle_sword.cancel()
self.sendClose()
def onClose(self, wasClean, code, reason):
#Now it is the right time to check our test assertions
self.assertion.callback(self._received_msgs)
class TestClientProtocolFactory(WebSocketClientFactory):
def __init__(self, assertion, close_condition, timeout, **kwargs):
WebSocketClientFactory.__init__(self, **kwargs)
self.assertion = assertion
self.close_condition = close_condition
self.timeout = timeout
#This parameter needs to be forced to None to not leave the reactor dirty
self.openHandshakeTimeout = None
def buildProtocol(self, addr):
protocol = TestClientProtocol(self.assertion, self.close_condition, self.timeout)
protocol.factory = self
return protocol
The trial based test
class WebSocketTest(TrialTest):
def setUp(self):
port = 8088
factory = WebSocketServerFactory("ws://localhost:{}".format(port))
factory.protocol = TestServerProtocol
self.listening_port = listenWS(factory)
self.factory, self.port = factory, port
def tearDown(self):
#cleaning up stuff otherwise the reactor complains
self.listening_port.stopListening()
def test_message_reception(self):
#This is the test assertion, we are testing that the messages received were 3
def assertion(msgs):
self.assertEquals(len(msgs), 3)
#This class says when the connection with the server should be finalized.
#In this case the condition to close the connectionis for the client to get 3 messages
class CommunicationHandler(object):
msg_count = 0
def close_condition(self, msg):
self.msg_count += 1
return self.msg_count == 3
d = Deferred()
d.addCallback(assertion)
#Here we create the client...
client_factory = TestClientProtocolFactory(d, CommunicationHandler().close_condition, 5, url="ws://localhost:{}".format(self.port))
#...and we connect it to the server
connectWS(client_factory)
#returning the assertion as a deferred purely for demonstration
return d
This is obviously just an example, but as you can see I did not have to mess around with makeConnection or any transport explicitly