I am dipping my toes into networking and multithreading in python. I have gone through the docs on concurrent.futures and SocketServer and am attempting to use these in the example I am working on. The examples in the docs seem strait forward enough but am struggling to apply them to the example I am working on.
The example is as follows. 2 GUI applications, one that sends information to the other via user interaction and the other displays this information.
So far I have the 2 applications up and running. Application A has the server running in a separate thread. Application B is able to connect to the server and send the desired information.
At this point I cannot seem to find a nice way to get the information displayed in the GUI of application A. I can think of several hacky ways of doing it but am interested in the nicest / most pythonic way. This seems a common problem so there must be a common pattern to use here. So my questions are.
While the server is running, how to get the information from the custom request handler to the server.
how to get information from the thread to the main application while it is running?
Example code is as follows
Server Window
import SocketServer
import concurrent.futures
import sys
from PyQt4 import QtGui
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
class MyRequestHandler(SocketServer.StreamRequestHandler):
def handle(self):
print('...connected from:', self.client_address)
data = self.rfile.readline().strip()
print('Data from client %s' % data)
class ServerWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 100, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
t = QtGui.QTextEdit()
l.addWidget(t)
self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
self.startServerThread()
self.show()
def startServerThread(self):
self.executor.submit(self.startServer)
# How to get information from the thread while it is still running?
def startServer(self):
print('starting server')
tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
print('waiting for connection...')
tcpServ.serve_forever()
# How to get information from the client (custom request handler)
# back to the GUI in a thread safe manner?
def launch():
app = QtGui.QApplication(sys.argv)
ex = ServerWindow()
sys.exit(app.exec_())
if __name__ == '__main__':
launch()
Client Window
import socket
import sys
import functools
from PyQt4 import QtGui
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
class ClientWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 650, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
for i in range(5):
name = 'test %d' % i
b = QtGui.QPushButton(name)
l.addWidget(b)
b.pressed.connect(functools.partial(self.onButtonClick, name))
self.show()
def onButtonClick(self, buttonName):
print('connecting to server')
tcpCliSock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcpCliSock.connect(ADDR)
print('Sending name %s' % buttonName)
tcpCliSock.send(buttonName)
tcpCliSock.close()
def launch():
app = QtGui.QApplication(sys.argv)
ex = ClientWindow()
sys.exit(app.exec_())
if __name__ == '__main__':
launch()
So one of the few ways I know of, that transfers data from a thread to the main GUI thread of an application, is to place the data in a python Queue. This Queue is read by a QThread (a Qt threading implementation that supports Qt signals and slots). The QThread makes a blocking call to queue.get(). When data is placed in the Queue by your handle() method, the QThread unblocks, reads the data out of the queue, and emits a thread-safe signal to the GUI thread. As a demonstration, I added the data into the QTextEdit.
So basically you need an intermediary between a python thread and the Qt GUI which generally interacts via signals/slots. The QThread performs this task, linking the thread-safe Queue object, with the thread-safe qt signal emission.
This effectively follows a similar answer I gave here, but here you have a socket instead of a custom thread putting the data in the queue.
You might also be interested in this SO post, which explains why some of the lines of code I have used to make the QThread and connect the signals, etc, are written in the order they are!
P.S. I added a line to shutdown the socket server when the app window is closed (the socket server used to keep running in the background)
Server Window code
import SocketServer
import concurrent.futures
import sys
from PyQt4 import QtGui
from PyQt4 import QtCore
from Queue import Queue
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
# create a global queue object that both the handle() method and the QThread (see later in the code) can access
queue = Queue()
class MyRequestHandler(SocketServer.StreamRequestHandler):
def handle(self):
print('...connected from:', self.client_address)
data = self.rfile.readline().strip()
print('Data from client %s' % data)
queue.put(data)
class ServerWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 100, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
self.t = QtGui.QTextEdit()
l.addWidget(self.t)
self.executor = futures.ThreadPoolExecutor(max_workers=1)
self.startServerThread()
self.show()
#QtCore.pyqtSlot(str)
def receive_data(self, data):
self.t.moveCursor(QtGui.QTextCursor.End)
self.t.insertPlainText( data )
def startServerThread(self):
self.executor.submit(self.startServer)
# How to get information from the thread while it is still running?
def startServer(self):
print('starting server')
self.tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
print('waiting for connection...')
self.tcpServ.serve_forever()
# How to get information from the client (custom request handler)
# back to the GUI in a thread safe manner?
# This class runs in a QThread and listens on the end of a queue and emits a signal to the GUI
class MyReceiver(QtCore.QObject):
mysignal = QtCore.pyqtSignal(str)
def __init__(self,queue,*args,**kwargs):
QtCore.QObject.__init__(self,*args,**kwargs)
self.queue = queue
#QtCore.pyqtSlot()
def run(self):
while True:
text = self.queue.get()
self.mysignal.emit(text)
def launch():
app = QtGui.QApplication(sys.argv)
ex = ServerWindow()
# Create thread that will listen on the other end of the queue, and send the text to the textedit in our application
thread = QtCore.QThread()
my_receiver = MyReceiver(queue)
my_receiver.mysignal.connect(ex.receive_data)
my_receiver.moveToThread(thread)
thread.started.connect(my_receiver.run)
thread.start()
ret_code = app.exec_()
ex.tcpServ.shutdown()
sys.exit(ret_code)
if __name__ == '__main__':
launch()
While the server is running, how to get the information from the custom request handler to the server.
how to get information from the thread to the main application while it is running?
These are my thoughts of how it should work.
class MyRequestHandler(SocketServer.StreamRequestHandler):
#property
def application_window(self):
# you would override setup() for this, usually.
return self.server.application_window
def handle(self):
print(self.application_window)
# ...
def startServer(self):
print('starting server')
tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
tcpServ.application_window = self # !!!!!!!!!!!!!!!!!!!!!!!!! added
print('waiting for connection...', self)
tcpServ.serve_forever()
Maybe you need to adjust something. By the way there are techniques for sharing information between server and client.
Related
I'm trying to build a little python application, where a simple webserver runs in the background, and you can the use GUI to send different messages.
I'm using PyQt5 and Python3.6, and I've managed to pass data from the working thread to the GUI, but I don't know how to do that the other way around.
Here's a skeleton of my code:
MainWindow:
class Ui_MainWindow(object):
def __init__(self):
super().__init__()
self.input = True
# 1 - create Worker and Thread inside the Form
self.obj = WebServer.WebServer(self.input) # no parent!
self.thread = QtCore.QThread() # no parent!
# 2 - Connect Worker`s Signals to Form method slots to post data.
self.obj.dataReady.connect(self.onDataReady)
# 3 - Move the Worker object to the Thread object
self.obj.moveToThread(self.thread)
# 4 - Connect Worker Signals to the Thread slots
self.obj.finished.connect(self.thread.quit)
# 5 - Connect Thread started signal to Worker operational slot method
self.thread.started.connect(self.obj.startServer)
# 6 - Start the thread
self.thread.start()
# 7 - Start the form
self.setupUi()
def setupUi(self):
# Set up the GUI
#...
self.MainWindow.show()
def onDataReady(self, data):
# Data received from working thread
# Do stuff...
def on_click_set2(self):
self.input = not self.input
print(self.input)
if __name__ == "__main__":
import sys
app = QtWidgets.QApplication(sys.argv)
ui = Ui_MainWindow()
sys.exit(app.exec_())
WebServer:
class WebServer(QObject):
finished = pyqtSignal()
dataReady = pyqtSignal(dict)
def __init__(self, input):
super().__init__()
self.input = input
#pyqtSlot()
def startServer(self): # A slot takes no params
# self.loop = asyncio.get_event_loop()
self.loop = asyncio.new_event_loop()
asyncio.set_event_loop(self.loop)
coro = asyncio.start_server(self.handle_update, '192.168.2.1', 8888, loop=self.loop)
self.server = self.loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(self.server.sockets[0].getsockname()))
try:
self.loop.run_forever()
except KeyboardInterrupt:
pass
self.finished.emit()
async def handle_update(self, reader, writer):
data = await reader.read(100)
message = data.decode()
addr = writer.get_extra_info('peername')
print(f'Received: {message} from {addr}')
if self.input:
print("Input is True")
else:
print("Input is False")
reply = self.input
print(f'Send: {reply}')
writer.write(str(reply).encode())
await writer.drain()
print("Close the client socket")
writer.close()
self.dataReady.emit(reply)
So for example I want to pass input to the thread, and if I do like above (obviously) input always stays the initial value, and won't change in the thread when I hit the button on the GUI.
How can I do it, that the value of the input is updated whenever I hit the button (so passing a value to the thread from GUI during runtime)? I assume is similar to passing from the thread to the GUI, so emitting a signal from GUI and connecting to it on the tread, but I don't know how to find a reference to the GUI from the working thread.
So any advice on how to do that? And of course any other input regarding the code or approach to the application/background server solution is welcomed! Thanks for the help in advance!
UPDATE:
Maybe it wasn't clear what my question was, so here it is:
How can I send a value from the GUI thread to the worker thread while both of them are running parallel? (If the above code makes any sense to you use that as an example, otherwise a general example would be appreciated)
It is not necessary that the WebServer live in another thread, it is only necessary that the eventloop be executed in another thread. In this case, the WebServer exchanges the data with the server thread through a queue. Although the QObjects are not thread-safe, the signals are so, there is no problem in emitting the signal from a thread other than the one that the QObject lives
import asyncio
import threading
import queue
from functools import partial
from PyQt5 import QtCore, QtGui, QtWidgets
class WebServer(QtCore.QObject):
dataReady = QtCore.pyqtSignal(object)
def startServer(self):
self.m_loop = asyncio.new_event_loop()
self.m_queue = queue.Queue()
asyncio.set_event_loop(self.m_loop)
coro = asyncio.start_server(
self.handle_update, "127.0.0.1", 10000, loop=self.m_loop
)
self.server = self.m_loop.run_until_complete(coro)
print("Serving on {}".format(self.server.sockets[0].getsockname()))
threading.Thread(target=self.m_loop.run_forever, daemon=True).start()
#QtCore.pyqtSlot(object)
def setData(self, data):
if hasattr(self, "m_queue"):
self.m_queue.put(data)
def stop(self):
if hasattr(self, "m_loop"):
self.m_loop.stop()
async def handle_update(self, reader, writer):
reply = ""
data = await reader.read(100)
message = data.decode()
addr = writer.get_extra_info("peername")
print(f"Received: {message} from {addr}")
if not self.m_queue.empty():
data = self.m_queue.get(block=False)
reply = data
print(f"Send: {reply}")
writer.write(str(reply).encode())
await writer.drain()
print("Close the client socket")
writer.close()
self.dataReady.emit(reply)
class Widget(QtWidgets.QWidget):
dataChanged = QtCore.pyqtSignal(object)
def __init__(self, parent=None):
super().__init__(parent)
self.m_lineedit = QtWidgets.QLineEdit()
button = QtWidgets.QPushButton("Send", clicked=self.onClicked)
lay = QtWidgets.QVBoxLayout(self)
lay.addWidget(self.m_lineedit)
lay.addWidget(button)
self.m_web = WebServer()
self.m_web.startServer()
self.dataChanged.connect(self.m_web.setData)
#QtCore.pyqtSlot()
def onClicked(self):
text = self.m_lineedit.text()
self.dataChanged.emit(text)
#QtCore.pyqtSlot(object)
def onDataReady(self, data):
print(data)
def closeEvent(self, event):
self.m_web.stop()
super().closeEvent(event)
if __name__ == "__main__":
import sys
app = QtWidgets.QApplication(sys.argv)
w = Widget()
w.show()
sys.exit(app.exec_())
I'm doing a simple socket connection and the read.All() and write() are not giving me anything back.
import PySide.QtNetwork as Network
import PySide.QtCore as Core
proxyAddress = '127.0.0.1'
proxyPort = 1025
tcpSocket = Network.QTcpSocket()
tcpSocket.connectToHost(proxyAddress, proxyPort)
tcpSocket.state() gives
"PySide.QtNetwork.QAbstractSocket.SocketState.ConnectingState"
When I then try to send something, I get this back:
tcpSocket.write("Hello")
5L
And for readAll():
tcpSocket.readAll()
PySide.QtCore.QByteArray('')
Oh and to disconnect I try:
tcpSocket.disconnectFromHost()
and it comes back with the next line to write as if it was successful in disconnecting but when I check the state:
tcpSocket.state()
"PySide.QtNetwork.QAbstractSocket.SocketState.ConnectingState"
Any help is appreciated.
First of all the state that signals is ConnectingState that indicates that the connection has not been made so it is not correct to perform any task at that moment, you must wait for the status to be ConnectedState. On the other hand, the port you point out is probably being used by another application since it is a small number that are usually reserved for another task. In the next part I show an example of a server and a client (first launch the server and then you can launch the number of clients you want.). Finally in Qt the tasks should not be performed synchronously but through signals since the Qt event-loop needs to update internal and external variables states.
server.py
import uuid
from PySide import QtCore, QtNetwork
class ServerManager(QtCore.QObject):
def __init__(self, parent=None):
super(ServerManager, self).__init__(parent)
self._server = QtNetwork.QTcpServer(self)
self._server.newConnection.connect(self.on_newConnection)
self._clients = {}
def launch(self, address=QtNetwork.QHostAddress.Any, port=9999):
return self._server.listen(QtNetwork.QHostAddress(address), port)
#QtCore.Slot()
def on_newConnection(self):
socket = self._server.nextPendingConnection()
socket.readyRead.connect(self.on_readyRead)
if socket not in self._clients:
self._clients[socket] = uuid.uuid4()
#QtCore.Slot()
def on_readyRead(self):
socket = self.sender()
resp = socket.readAll()
code = self._clients[socket]
print("From[{}]- message: {}".format(code, resp))
socket.write("Server: " + str(resp)[::-1])
if __name__ == '__main__':
import sys
import signal
signal.signal(signal.SIGINT, signal.SIG_DFL)
app = QtCore.QCoreApplication(sys.argv)
address = '127.0.0.1'
port = 9000
server = ServerManager()
if not server.launch(address, port):
sys.exit(-1)
sys.exit(app.exec_())
client.py
from PySide import QtCore, QtNetwork
class ClientManager(QtCore.QObject):
def __init__(self, parent=None):
super(ClientManager, self).__init__(parent)
self._socket = QtNetwork.QTcpSocket(self)
self._socket.stateChanged.connect(self.on_stateChanged)
self._socket.readyRead.connect(self.on_readyRead)
self._timer = QtCore.QTimer(self, interval=1000)
self._timer.timeout.connect(self.sendMessage)
def launch(self, address=QtNetwork.QHostAddress.Any, port=9999):
return self._socket.connectToHost(QtNetwork.QHostAddress(address), port)
#QtCore.Slot(QtNetwork.QAbstractSocket.SocketState)
def on_stateChanged(self, state):
if state == QtNetwork.QAbstractSocket.ConnectedState:
self._timer.start()
print("connected")
elif state == QtNetwork.QAbstractSocket.UnconnectedState:
print("disconnected")
QtCore.QCoreApplication.quit()
#QtCore.Slot()
def sendMessage(self):
if self._socket.state() == QtNetwork.QAbstractSocket.ConnectedState:
msg = QtCore.QDateTime.currentDateTime().toString()
self._socket.write(msg)
#QtCore.Slot()
def on_readyRead(self):
print("Response: ", self._socket.readAll())
if __name__ == '__main__':
import sys
import signal
signal.signal(signal.SIGINT, signal.SIG_DFL)
app = QtCore.QCoreApplication(sys.argv)
address = '127.0.0.1'
port = 9000
server = ClientManager()
server.launch(address, port)
sys.exit(app.exec_())
I have a server running a loop that reads data from a device and I want to send them to all clients who connect on a websocket on tornado.
I tried putting the loop inside the open function but then it can't handle on_close function or new connections.
What is best practice to do that?
#!/usr/bin/env python
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
import socket
class MyWebSocketServer(tornado.websocket.WebSocketHandler):
def open(self):
print('new connection'+self.request.remote_ip)
try:
while True:
'''
read and send data
'''
except Exception,error:
print "Error on Main: "+str(error)
def on_close(self):
print('connection closed'+self.request.remote_ip)
application=tornado.web.Application([(r'/ws',MyWebSocketServer),])
if __name__=="__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8000)
print('start')
tornado.ioloop.IOLoop.instance().start()
Thanks
Here's a full example about running your blocking code in a separate thread and broadcasting messages to all connected clients.
...
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(max_workers=1) # spawn only 1 thread
class MyWebSocketServer(tornado.websocket.WebSocketHandler):
connections = set() # create a set to hold connections
def open(self):
# put the new connection in connections set
self.connections.add(self)
def on_close(self):
print('connection closed'+self.request.remote_ip)
print('new connection'+self.request.remote_ip)
# remove client from connections
self.connections.remove(self)
#classmethod
def send_message(cls, msg):
for client in cls.connections:
client.write_message(msg)
def read_from_serial(loop, msg_callback):
"""This function will read from serial
and will run in aseparate thread
`loop` is the IOLoop instance
`msg_allback` is the function that will be
called when new data is available from usb
"""
while True:
# your code ...
# ...
# when you get new data
# tell the IOLoop to schedule `msg_callback`
# to send the data to all clients
data = "new data"
loop.add_callback(msg_callback, data)
...
if __name__ == '__main__':
loop = tornado.ioloop.IOLoop.current()
msg_callback = MyWebSocketServer.send_message
# run `read_from_serial` in another thread
executor.submit(read_from_serial, loop, msg_callback)
...
loop.start()
this is probably more a question about threading than about my websocket.
I'm using "SimpleWebSocket" from github ( https://github.com/dpallot/simple-websocket-server )
The example works fine:
from SimpleWebSocketServer import SimpleWebSocketServer, WebSocket
class SimpleEcho(WebSocket):
def handleMessage(self):
# echo message back to client
self.sendMessage(self.data)
def handleConnected(self):
print self.address, 'connected'
def handleClose(self):
print self.address, 'closed'
server = SimpleWebSocketServer('', 8000, SimpleEcho)
server.serveforever()
The Server is running, I can connect and send Messages.
Now i try to run it as a Thread with those classes:
This one is supposed to create many threads including the WebSocketServer
from websockethread import WebSocketThread
class startManyThreads:
def __init__(self):
self.thread1 = WebSocketThread()
self.thread1.start()
if __name__ == "__main__":
startManyThreads = startManyThreads()
This class should run as my thread:
import threading
from SimpleWebSocketServer import SimpleWebSocketServer
from webSocketServer import WebSocketServer
class WebSocketThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
server = SimpleWebSocketServer('', 8000, WebSocketServer)
server.serveforever()
And this is the "customized" echo example:
from SimpleWebSocketServer import SimpleWebSocketServer, WebSocket
class SimpleEcho(WebSocket):
def handleMessage(self):
# echo message back to client
self.sendMessage(self.data)
def handleConnected(self):
print self.address, 'connected'
def handleClose(self):
print self.address, 'closed'
I have also tried to derive this: class SimpleEcho(WebSocket, threading.Thread):
Any Ideas what i'm doing wrong?
&
Thanks alot in advance!
Edit:
The result when i run "simpleEcho" is that i get a prompt can connect via the websocket.html (provided on github), send and receive Messages
The result when i put it in a thread (anyone of the 3 ways i tried) is the same behaviour except when i try to "connect" from the websocket.html i get a "error: undefined". With nmap i checked and the Server seems so be running & listening to port 8000
Edit 2: Derived new Class from SimpleWebSocketServer
import threading
from SimpleWebSocketServer import SimpleWebSocketServer
class ThreadSimpleWebSocketThread(threading.Thread, SimpleWebSocketServer):
def __init__(self, serversocket):
threading.Thread.__init__(self)
self.serversocket = serversocket
def serveforever(self):
SimpleWebSocketServer.serversocket = self.serversocket
SimpleWebSocketServer.selectInterval = 0.1
SimpleWebSocketServer.listeners = [self.serversocket]
super(ThreadSimpleWebSocketThread, self).serveforever()
def run(self):
self.serveforever()
The main problem seems to be where you're starting the server. The Thread.__init__() method runs inside the main thread (of the caller), not the actual WebSocketThread(). This needs to be done in the Thread.run() method:
class WebSocketThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
server = SimpleWebSocketServer('', 8000, WebSocketServer)
server.serveforever()
The code inside run() actually runs inside the thread.
Note that because of the Global Interpreter Lock, threads won't improve performance much, and you'll probably need multiprocessing. However, if you just want to offload the I/O waiting, this should work fine.
Edit: From looking at this GitHub project, and rethinking what you're trying to do, this isn't trivial. You'll have to override WebSocket.serveforever() in your SimpleEcho() class and change it to accept the socket and pass the accepted socket to a Thread (see here).
I'm completely lost trying to create a UDP server/client for my game in python. I'm new to the language and only have limited experience with networking. Right now, the server runs, but doesn't seem to be getting any messages from the client.
Server:
class GameServer:
class GameServerUDPHandler(socketserver.BaseRequestHandler):
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
print("{} wrote:".format(self.client_address[0]))
print(data)
socket.sendto(data.upper(), self.client_address)
def __init__(self, port):
self.server = socketserver.UDPServer(("localhost", port), self.GameServerUDPHandler)
def start_server(self):
self.server.serve_forever(
Client:
import socket
import sys
class GameClient:
def __init__(self, port, host):
self.port = port
self.host = host
self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def register(self):
self.socket.sendto(bytes("register\n", "utf-8"), (self.host, self.port))
self.numberID = int(self.socket.recv(1024))
print("Received: {}".format(self.numberID))
-Main/Start of program
import gameserver
import gameclient
if __name__ == "__main__":
server = gameserver.GameServer(1300)
server.start_server()
client = gameclient.GameClient(1300, "localhost")
client.register()
NOTE: I'm most likely to multiple things wrong and may be violating several best practices in the language. I really have no clue.
The problem is that some of these calls are blocking. In particular, the serve_forever() method will run forever, so you need to put that on a separate thread if you want the rest of your program to continue:
import threading
if __name__ == "__main__":
server = GameServer(1300)
server_thread = threading.Thread(target=lambda: server.start_server())
server_thread.start()
time.sleep(1) # Give it time to start up; not production quality code of course
client = GameClient(1300, "localhost")
client.register()
socket.recv() is also a blocking call but that might be okay in this case.
Seems like this library isn't asynchronous so your first call to serve_forever will not return and your client never gets started. You can create a new thread to launch the server on or split your client and server into seperate processes.