Develop Reference

Python QtNetwork.QTcpSocket.readAll() and QtNetwork.QTcpSocket.write() not working

I'm doing a simple socket connection and the read.All() and write() are not giving me anything back.
import PySide.QtNetwork as Network
import PySide.QtCore as Core
proxyAddress = '127.0.0.1'
proxyPort = 1025
tcpSocket = Network.QTcpSocket()
tcpSocket.connectToHost(proxyAddress, proxyPort)
tcpSocket.state() gives
"PySide.QtNetwork.QAbstractSocket.SocketState.ConnectingState"
When I then try to send something, I get this back:
tcpSocket.write("Hello")
5L
And for readAll():
tcpSocket.readAll()
PySide.QtCore.QByteArray('')
Oh and to disconnect I try:
tcpSocket.disconnectFromHost()
and it comes back with the next line to write as if it was successful in disconnecting but when I check the state:
tcpSocket.state()
"PySide.QtNetwork.QAbstractSocket.SocketState.ConnectingState"
Any help is appreciated.

First of all the state that signals is ConnectingState that indicates that the connection has not been made so it is not correct to perform any task at that moment, you must wait for the status to be ConnectedState. On the other hand, the port you point out is probably being used by another application since it is a small number that are usually reserved for another task. In the next part I show an example of a server and a client (first launch the server and then you can launch the number of clients you want.). Finally in Qt the tasks should not be performed synchronously but through signals since the Qt event-loop needs to update internal and external variables states.
server.py
import uuid
from PySide import QtCore, QtNetwork
class ServerManager(QtCore.QObject):
def __init__(self, parent=None):
super(ServerManager, self).__init__(parent)
self._server = QtNetwork.QTcpServer(self)
self._server.newConnection.connect(self.on_newConnection)
self._clients = {}
def launch(self, address=QtNetwork.QHostAddress.Any, port=9999):
return self._server.listen(QtNetwork.QHostAddress(address), port)
#QtCore.Slot()
def on_newConnection(self):
socket = self._server.nextPendingConnection()
socket.readyRead.connect(self.on_readyRead)
if socket not in self._clients:
self._clients[socket] = uuid.uuid4()
#QtCore.Slot()
def on_readyRead(self):
socket = self.sender()
resp = socket.readAll()
code = self._clients[socket]
print("From[{}]- message: {}".format(code, resp))
socket.write("Server: " + str(resp)[::-1])
if __name__ == '__main__':
import sys
import signal
signal.signal(signal.SIGINT, signal.SIG_DFL)
app = QtCore.QCoreApplication(sys.argv)
address = '127.0.0.1'
port = 9000
server = ServerManager()
if not server.launch(address, port):
sys.exit(-1)
sys.exit(app.exec_())
client.py
from PySide import QtCore, QtNetwork
class ClientManager(QtCore.QObject):
def __init__(self, parent=None):
super(ClientManager, self).__init__(parent)
self._socket = QtNetwork.QTcpSocket(self)
self._socket.stateChanged.connect(self.on_stateChanged)
self._socket.readyRead.connect(self.on_readyRead)
self._timer = QtCore.QTimer(self, interval=1000)
self._timer.timeout.connect(self.sendMessage)
def launch(self, address=QtNetwork.QHostAddress.Any, port=9999):
return self._socket.connectToHost(QtNetwork.QHostAddress(address), port)
#QtCore.Slot(QtNetwork.QAbstractSocket.SocketState)
def on_stateChanged(self, state):
if state == QtNetwork.QAbstractSocket.ConnectedState:
self._timer.start()
print("connected")
elif state == QtNetwork.QAbstractSocket.UnconnectedState:
print("disconnected")
QtCore.QCoreApplication.quit()
#QtCore.Slot()
def sendMessage(self):
if self._socket.state() == QtNetwork.QAbstractSocket.ConnectedState:
msg = QtCore.QDateTime.currentDateTime().toString()
self._socket.write(msg)
#QtCore.Slot()
def on_readyRead(self):
print("Response: ", self._socket.readAll())
if __name__ == '__main__':
import sys
import signal
signal.signal(signal.SIGINT, signal.SIG_DFL)
app = QtCore.QCoreApplication(sys.argv)
address = '127.0.0.1'
port = 9000
server = ClientManager()
server.launch(address, port)
sys.exit(app.exec_())

Non-blocking client FTP retrieval and write

I am building an FTP test server and client with Twisted. The server runs great. It is basically the same from the Twisted ftpserver.py example. The client is where I am having some blocking issues during my file retrieval and writing. I have tried to resolve it through some quick Twisted threading utilities, but to no avail.
Here is my server:
#!/usr/bin/env python2
from __future__ import print_function, division, absolute_import
# Import twisted things
from twisted.protocols.ftp import FTPFactory
from twisted.protocols.ftp import FTPRealm
from twisted.internet import reactor
from twisted.cred.portal import Portal
from twisted.cred.checkers import AllowAnonymousAccess
p = Portal(FTPRealm("test/"), [AllowAnonymousAccess()])
f = FTPFactory(p)
f.timeOut = None
reactor.listenTCP(5504, f)
reactor.run()
The client side, paired with this, is a simple wxPython GUI that presents a text box to write the name of the file you want to retrieve in. Within this GUI there is a wx.Timer that executes a method every 50 milliseconds. This is what is blocking my FTP file retrieval. I find that because the main thread is being used up, the protocol that receives the data is having hiccups. If you are wondering why I have this setup I am simulating the use case for a much larger project.
My attempt to solve this has been to use deferToThread on the specific point when I need to retrieve a file. However, by printing the current thread I find that the protocol that is receiving the data is running in the main thread. This is the problem I am trying to solve. Any help is much appreciated.
My client code:
#!/usr/bin/env python2
from __future__ import print_function, division, absolute_import
import wx
import sys
import threading
from twisted.internet import wxreactor
wxreactor.install()
from twisted.internet import reactor
from twisted.protocols.ftp import FTPClient
from twisted.internet import protocol
from twisted.internet import threads
from twisted.python import log
# This is the GUI
class TextSend(wx.Frame):
def __init__(self):
wx.Frame.__init__(self, None, -1, "Request Files", size=(200, 75))
self.protocol = None # ftp client protocol
self.factory = None
panel = wx.Panel(self)
vertSizer = wx.BoxSizer(wx.VERTICAL)
horzSizer = wx.BoxSizer(wx.HORIZONTAL)
self.fileName = None
self.textbox = wx.TextCtrl(parent=panel, id=100, size=(100,-1))
self.btn = wx.Button(panel, label="Retr.")
# timer and checkbox for timer
self.timer = wx.Timer(self, id=wx.ID_ANY)
self.check = wx.CheckBox(parent=panel, label="Start blocking")
#Bind
self.textbox.Bind(wx.EVT_TEXT, self.getText)
self.btn.Bind(wx.EVT_BUTTON, self.press)
self.check.Bind(wx.EVT_CHECKBOX, self.onCheck)
self.Bind(wx.EVT_TIMER, self.onTimer, self.timer)
horzSizer.Add(self.textbox, flag=wx.ALIGN_CENTER)
horzSizer.Add(self.btn, flag=wx.ALIGN_CENTER)
vertSizer.Add(horzSizer, flag=wx.ALIGN_CENTER)
vertSizer.Add(self.check, flag=wx.ALIGN_CENTER)
panel.SetSizer(vertSizer)
panel.Layout()
def getText(self, evt):
self.fileName = str(self.textbox.GetValue())
def onCheck(self, evt):
yes = self.check.GetValue()
if yes:
print("Starting timer")
self.timer.Start(50)
else: # no
self.timer.Stop()
def onTimer(self, evt):
#print("Triggered timer")
pass
def press(self, evt):
print("Send:", self.fileName)
d = threads.deferToThread(self.retrieve)
d.addCallback(self.done)
def retrieve(self):
print(threading.current_thread())
# This is what does the retrieving. Pass in FileWriter and
# FileWriter's dataReceived method is called by main thread
self.protocol.retrieveFile(self.fileName, FileWriter(self.fileName), offset=0).addCallbacks(self.done, self.fail)
return "Done with deferToThread"
def done(self, msg):
print(threading.current_thread())
print("DONE Retrieving:", msg)
def fail(self, error):
print('Failed. Error was:')
print(error)
# This writes to the file of a same name as the one retrieved.
class FileWriter(protocol.Protocol):
def __init__(self, fileName):
self.f = open(fileName, 'wb')
print("FROM FileWriter __init__:", threading.current_thread())
def dataReceived(self, data):
print("Byte size", len(data))
print("FROM FileWriter dataReceived:", threading.current_thread())
self.f.write(data)
def connectionLost(self, reason):
print("Writing closed and done")
print("FROM FileWriter connectionLost:", threading.current_thread())
self.f.close()
# Client FTP Protocol
class TestClient(FTPClient, object):
def __init__(self, factory, username, password, passive):
super(TestClient, self).__init__(username=username, password=password, passive=passive)
self.factory = factory
def connectionMade(self):
print("hello")
gui = self.factory.gui
gui.protocol = self
# Twisted Client Factory
class FileClientFactory(protocol.ClientFactory):
def __init__(self, gui):
self.gui = gui
self.protocol = None
def buildProtocol(self, addr):
user = 'anonymous'
passwd = 'twisted#matrix.com'
self.protocol = TestClient(self, username=user, password=passwd, passive=1)
return self.protocol
def clientConnectionLost(self, transport, reason):
print("Connectiong lost normally:", reason)
def clientConnectionFailed(self, transport, reason):
print("Connection failed:", reason)
if __name__ == "__main__":
# Initialize and show GUI
logger = log.startLogging(sys.stdout)
app = wx.App(False)
app.frame = TextSend()
app.frame.Show()
reactor.registerWxApp(app)
# Build Factory
f = FileClientFactory(app.frame)
# Connect to FTP server
reactor.connectTCP("localhost", 5504, f)
reactor.run()
wxPython main loop.
app.MainLoop()

You cannot deferToThread(function_that_uses_twisted_apis). Twisted APIs are almost all non-thread-safe. You must use them in the reactor thread only (the exceptions are a couple thread-scheduling-related APIs).
Instead, get rid of your blocking code. Put it into another thread, another process, or rewrite it to be non-blocking.

Twisted doesn't exit when reactor.stop is called

I am trying to reimplement netcat in Python:
#!/usr/bin/env python2
from sys import stdin, stdout
from twisted.internet import reactor
from twisted.internet.protocol import Protocol
from twisted.internet.endpoints import TCP4ClientEndpoint, connectProtocol
class NcClient(Protocol):
def dataReceived(self, data):
stdout.write(data)
def sendData(self, data):
self.transport.write(data)
self.transport.write("\n")
client = NcClient()
def cmdloop():
while True:
line = stdin.readline()
if line == "":
break
else:
client.sendData(line)
if reactor.running:
reactor.stop()
point = TCP4ClientEndpoint(reactor, "localhost", 6004)
connectProtocol(point, client)
reactor.callInThread(cmdloop)
reactor.run()
When cmdloop detects end of input, it calls reactor.stop.
As I understand, reactor.stop sends shutdown events to all things managed by Twisted, such as threads, connections etc. In response to these events connections are closed, threads wait for the completion of their procedures etc. So when reactor.stop() is called, the connection to localhost:6004 should close, and the program should exit.
However, it doesn't happen immediately, but only when NcClient receives a message from server. As if it is reading those messages blockingly, in a loop, and only when it receives one does it proceed to handle shutdown requests.
How to make it shut down before receiving a message? I know of reactor.crash(), but is there a more polite option?

Your main problem is that cmdloop is running in a non-reactor thread, and yet it is calling reactor methods other than callFromThread (specifically: transport.write via client.sendData). The documentation is quite clear on this:
Methods within Twisted may only be invoked from the reactor thread unless otherwise noted. Very few things within Twisted are thread-safe.
Luckily, implementing the equivalent of netcat doesn't require threading at all. You can simply use Twisted's built-in support for standard I/O as a source of data. Here's an example version:
import sys
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, Factory
from twisted.internet.endpoints import clientFromString
from twisted.internet.stdio import StandardIO
class NcClient(Protocol):
def __init__(self, forwardTo):
self.forwardTo = forwardTo
def connectionMade(self):
self.transport.registerProducer(self.forwardTo.transport, True)
self.forwardTo.transport.resumeProducing()
self.forwardTo.transport.registerProducer(self.transport, True)
def dataReceived(self, data):
self.forwardTo.transport.write(data)
def connectionLost(self, reason):
reactor.stop()
class StdIo(Protocol):
def connectionMade(self):
self.transport.pauseProducing()
f4p = Factory.forProtocol(lambda: NcClient(self))
d = endpoint.connect(f4p)
#d.addCallback
def connected(proto):
self.client = proto
def dataReceived(self, data):
self.client.transport.write(data)
def connectionLost(self, reason):
reactor.stop()
endpoint = clientFromString(reactor, sys.argv[1])
output = StandardIO(proto=StdIo(), reactor=reactor)
reactor.run()
If you still want to use threads for some reason, you can modify NcClient to use callFromThread to invoke sendData instead of calling it directly.

How to have two programs communicate with one another

I am dipping my toes into networking and multithreading in python. I have gone through the docs on concurrent.futures and SocketServer and am attempting to use these in the example I am working on. The examples in the docs seem strait forward enough but am struggling to apply them to the example I am working on.
The example is as follows. 2 GUI applications, one that sends information to the other via user interaction and the other displays this information.
So far I have the 2 applications up and running. Application A has the server running in a separate thread. Application B is able to connect to the server and send the desired information.
At this point I cannot seem to find a nice way to get the information displayed in the GUI of application A. I can think of several hacky ways of doing it but am interested in the nicest / most pythonic way. This seems a common problem so there must be a common pattern to use here. So my questions are.
While the server is running, how to get the information from the custom request handler to the server.
how to get information from the thread to the main application while it is running?
Example code is as follows
Server Window
import SocketServer
import concurrent.futures
import sys
from PyQt4 import QtGui
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
class MyRequestHandler(SocketServer.StreamRequestHandler):
def handle(self):
print('...connected from:', self.client_address)
data = self.rfile.readline().strip()
print('Data from client %s' % data)
class ServerWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 100, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
t = QtGui.QTextEdit()
l.addWidget(t)
self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
self.startServerThread()
self.show()
def startServerThread(self):
self.executor.submit(self.startServer)
# How to get information from the thread while it is still running?
def startServer(self):
print('starting server')
tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
print('waiting for connection...')
tcpServ.serve_forever()
# How to get information from the client (custom request handler)
# back to the GUI in a thread safe manner?
def launch():
app = QtGui.QApplication(sys.argv)
ex = ServerWindow()
sys.exit(app.exec_())
if __name__ == '__main__':
launch()
Client Window
import socket
import sys
import functools
from PyQt4 import QtGui
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
class ClientWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 650, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
for i in range(5):
name = 'test %d' % i
b = QtGui.QPushButton(name)
l.addWidget(b)
b.pressed.connect(functools.partial(self.onButtonClick, name))
self.show()
def onButtonClick(self, buttonName):
print('connecting to server')
tcpCliSock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcpCliSock.connect(ADDR)
print('Sending name %s' % buttonName)
tcpCliSock.send(buttonName)
tcpCliSock.close()
def launch():
app = QtGui.QApplication(sys.argv)
ex = ClientWindow()
sys.exit(app.exec_())
if __name__ == '__main__':
launch()

So one of the few ways I know of, that transfers data from a thread to the main GUI thread of an application, is to place the data in a python Queue. This Queue is read by a QThread (a Qt threading implementation that supports Qt signals and slots). The QThread makes a blocking call to queue.get(). When data is placed in the Queue by your handle() method, the QThread unblocks, reads the data out of the queue, and emits a thread-safe signal to the GUI thread. As a demonstration, I added the data into the QTextEdit.
So basically you need an intermediary between a python thread and the Qt GUI which generally interacts via signals/slots. The QThread performs this task, linking the thread-safe Queue object, with the thread-safe qt signal emission.
This effectively follows a similar answer I gave here, but here you have a socket instead of a custom thread putting the data in the queue.
You might also be interested in this SO post, which explains why some of the lines of code I have used to make the QThread and connect the signals, etc, are written in the order they are!
P.S. I added a line to shutdown the socket server when the app window is closed (the socket server used to keep running in the background)
Server Window code
import SocketServer
import concurrent.futures
import sys
from PyQt4 import QtGui
from PyQt4 import QtCore
from Queue import Queue
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
# create a global queue object that both the handle() method and the QThread (see later in the code) can access
queue = Queue()
class MyRequestHandler(SocketServer.StreamRequestHandler):
def handle(self):
print('...connected from:', self.client_address)
data = self.rfile.readline().strip()
print('Data from client %s' % data)
queue.put(data)
class ServerWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 100, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
self.t = QtGui.QTextEdit()
l.addWidget(self.t)
self.executor = futures.ThreadPoolExecutor(max_workers=1)
self.startServerThread()
self.show()
#QtCore.pyqtSlot(str)
def receive_data(self, data):
self.t.moveCursor(QtGui.QTextCursor.End)
self.t.insertPlainText( data )
def startServerThread(self):
self.executor.submit(self.startServer)
# How to get information from the thread while it is still running?
def startServer(self):
print('starting server')
self.tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
print('waiting for connection...')
self.tcpServ.serve_forever()
# How to get information from the client (custom request handler)
# back to the GUI in a thread safe manner?
# This class runs in a QThread and listens on the end of a queue and emits a signal to the GUI
class MyReceiver(QtCore.QObject):
mysignal = QtCore.pyqtSignal(str)
def __init__(self,queue,*args,**kwargs):
QtCore.QObject.__init__(self,*args,**kwargs)
self.queue = queue
#QtCore.pyqtSlot()
def run(self):
while True:
text = self.queue.get()
self.mysignal.emit(text)
def launch():
app = QtGui.QApplication(sys.argv)
ex = ServerWindow()
# Create thread that will listen on the other end of the queue, and send the text to the textedit in our application
thread = QtCore.QThread()
my_receiver = MyReceiver(queue)
my_receiver.mysignal.connect(ex.receive_data)
my_receiver.moveToThread(thread)
thread.started.connect(my_receiver.run)
thread.start()
ret_code = app.exec_()
ex.tcpServ.shutdown()
sys.exit(ret_code)
if __name__ == '__main__':
launch()

While the server is running, how to get the information from the custom request handler to the server.
how to get information from the thread to the main application while it is running?
These are my thoughts of how it should work.
class MyRequestHandler(SocketServer.StreamRequestHandler):
#property
def application_window(self):
# you would override setup() for this, usually.
return self.server.application_window
def handle(self):
print(self.application_window)
# ...
def startServer(self):
print('starting server')
tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
tcpServ.application_window = self # !!!!!!!!!!!!!!!!!!!!!!!!! added
print('waiting for connection...', self)
tcpServ.serve_forever()
Maybe you need to adjust something. By the way there are techniques for sharing information between server and client.

python twisted simple server client

I am trying to build a simple "quote of the day" server and client modified from the twisted tutorial documentation. I want the "quote of the day" to be printed from the client to prove the communication. However, from what I can tell the client is not connecting. Here is my code.
Server
from twisted.python import log
from twisted.internet.protocol import Protocol
from twisted.internet.protocol import Factory
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet import reactor
class QOTD(Protocol):
def connectionMade(self):
self.transport.write("An apple a day keeps the doctor away\r\n")
self.transport.loseConnection()
class QOTDFactory(Factory):
def buildProtocol(self, addr):
return QOTD()
# 8007 is the port you want to run under. Choose something >1024
endpoint = TCP4ServerEndpoint(reactor, 8007)
endpoint.listen(QOTDFactory())
reactor.run()
Client
import sys
from twisted.python import log
from twisted.internet import reactor
from twisted.internet.protocol import Factory, Protocol
from twisted.internet.endpoints import TCP4ClientEndpoint
class SimpleClient(Protocol):
def connectionMade(self):
log.msg("connection made")
#self.transport.loseConnection()
def lineReceived(self, line):
print "receive:", line
class SimpleClientFactory(Factory):
def buildProtocol(self, addr):
return SimpleClient()
def startlog():
log.startLogging(sys.stdout)
point = TCP4ClientEndpoint(reactor, "localhost", 8007)
d = point.connect(SimpleClientFactory)
reactor.callLater(0.1, startlog)
reactor.run()

pass instance of SimpleClientFactory, not the class itself to point.connect()
subclass SimpleClient from twisted.protocol.basic.LineReceiver instead of Protocol if you use lineReceived
call addErrback on the results of endpoint.listen and point.connect to handle errors