I've written a Twisted based server and I'd like to test it using twisted as well.
But I'd like to write a load test starting a bunch of request at the same time.
But I believe that I didn't get the concepts of Twisted, mainly client side, because I'm stucked with this problem:
from twisted.internet import reactor, protocol
from threading import Thread
from twisted.protocols.basic import LineReceiver
__author__="smota"
__date__ ="$30/10/2009 17:17:50$"
class SquitterClient(LineReceiver):
def connectionMade(self):
self.sendLine("message from " % threading.current_thread().name);
pass
def connectionLost(self, reason):
print "connection lost"
def sendMessage(self, msg):
for m in [ "a", "b", "c", "d", "e"]:
self.sendLine(msg % " - " % m);
class SquitterClientFactory(protocol.ClientFactory):
protocol = SquitterClient
def clientConnectionFailed(self, connector, reason):
print "Connection failed - goodbye!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
print "Connection lost - goodbye!"
reactor.stop()
def createAndRun():
f = SquitterClientFactory()
reactor.connectTCP("localhost", 4010, f)
reactor.run(installSignalHandlers=0)
# this connects the protocol to a server runing on port 8000
def main():
for n in range(0,10):
th=Thread(target=createAndRun)
th.start()
# this only runs if the module was *not* imported
if __name__ == '__main__':
main()
socket_client.py:35:
DeprecationWarning: Reactor already
running! This behavior is deprecated
since Twisted 8.0
reactor.run(installSignalHandlers=0)
What am I missing?
How to test it?
Thank you,
Samuel
The direct cause for your failure is that you attemp to call run() on the reactor multiple times. You are supposed to ever only call run() once. I think you are expecting to have multiple reactors, each in its own thread, but actually you only have one. The bad thing is that having multiple reactors is difficult or impossible - the good thing is that it's also unnecessary. In fact you don't even need multiple threads. You can multiplex multiple client connections in one reactor almost as easily as you can listen for multiple connections.
Modifying your sample code, something like the following should work. The key idea is that you don't need multiple reactors to do things concurrently. The only thing that could ever be concurrent with the regular Python implementation is I/O anyway.
from twisted.internet import reactor, protocol
from twisted.protocols.basic import LineReceiver
__author__="smota"
__date__ ="$30/10/2009 17:17:50$"
class SquitterClient(LineReceiver):
def connectionMade(self):
self.messageCount = 0
# The factory provides a reference to itself, we'll use it to enumerate the clients
self.factory.n += 1
self.name = "Client %d" %self.factory.n
# Send initial message, and more messages a bit later
self.sendLine("Client %s starting!" % self.name);
reactor.callLater(0.5, self.sendMessage, "Message %d" %self.messageCount)
def connectionLost(self, reason):
print "connection lost"
def sendMessage(self, msg):
for m in [ "a", "b", "c", "d", "e"]:
self.sendLine("Copy %s of message %s from client %s!" % (m, msg, self.name))
if self.factory.stop:
self.sendLine("Client %s disconnecting!" % self.name)
self.transport.loseConnection()
else:
self.messageCount += 1
reactor.callLater(0.5, self.sendMessage, "Message %d" %self.messageCount)
class SquitterClientFactory(protocol.ClientFactory):
protocol = SquitterClient
def __init__(self):
self.n = 0
self.stop = False
def stopTest():
self.stop = True
def clientConnectionFailed(self, connector, reason):
print "Connection failed - goodbye!"
def clientConnectionLost(self, connector, reason):
print "Connection lost - goodbye!"
# this connects the protocol to a server running on port 8000
def main():
# Create 10 clients
f = SquitterClientFactory()
for i in range(10):
reactor.connectTCP("localhost", 8000, f)
# Schedule end of test in 10 seconds
reactor.callLater(10, f.stopTest)
# And let loose the dogs of war
reactor.run()
# this only runs if the module was *not* imported
if __name__ == '__main__':
main()
Related
I know a similar question has been asked here, but I am still battling with the following issue:
I am using putty as a telnet client and am using Win10. The code is given below. When I start the reactor and then connect a client, I get a response after each character is typed which is printed using the dataReceived function. However I can never seem to get the function lineReceived() to fire. I also tried a simple chat server example which worked fine using the lineReceived() function (and that example had no dataReceived() function. I tried commenting out dataReceived(), thinking perhaps it was masking out lineReceived().
In the code below, I cannot get lineReceived() to fire , only dataReceived() fires after each character is typed.
#! C:/Python37/python.exe
from twisted.internet import reactor
from twisted.internet.protocol import Factory, Protocol
from datetime import datetime, tzinfo, timedelta
from twisted.protocols.basic import LineReceiver
class Echo(Protocol):
def dataReceived(self, data):
self.transport.write(data)
class LineReceiver(Protocol):
print("Starting.......")
delimiter = "\n"
TIMEOUT = 300 # Client timeout period in seconds
def timeOut(self):
print("Client: %s. %s" % (self.addr, "Connection Timed out"))
self.transport.loseConnection()
def lineLengthExceeded(self, line):
return self.transport.loseConnection()
def connectionMade(self):
print("Connected......")
self.transport.write(b"hell...")
self.timeout = reactor.callLater(
self.TIMEOUT, self.timeOut
) # start client timeout timer
self.addr = self.transport.getPeer().host
addr = self.addr
self.addr_test = self.transport.getPeer().host
self.factory.NUM_CLIENTS += 1
def connectionLost(self, reason):
print("Lost.......")
# self.sendMsg("- %s left." % self.name)
self.transport.write(b"a client left")
self.factory.NUM_CLIENTS -= 1
print("Client disconnected: - " + str(self.addr))
print("Number of connections = %s" % self.factory.NUM_CLIENTS)
# def dataReceived(self, data): # this runs a few times after an initial connection
# self.transport.write(b"you typed: " + data +b"\n" + b"\r")
# print(data) # prints to log file with byte order marks
def lineReceived(self, line):
self.sendLine(b"Welcome, %s!" % (name,))
self.transport.write(b"line rx function...")
class DataFactory(Factory):
protocol = LineReceiver
NUM_CLIENTS = 0
def main():
print("Started...Listening for incoming connections.")
if __name__ == "__main__":
main()
reactor.listenTCP(10003, DataFactory())
reactor.run()
You overwrote LineReceiver with your own Protocol subclass, which does not have a lineReceived() method. You just gave me a good reason for using the module path as a namespace instead of importing a specific object :D
from twisted.protocols import basic
from twisted.internet import endpoints, protocol, reactor
class LnRcv(basic.LineReceiver):
def lineReceived(self, line):
print(line)
class DataFactory(protocol.Factory):
protocol = LnRcv
server = endpoints.TCP4ServerEndpoint(reactor, 10003)
server.listen(DataFactory())
reactor.run()
Update
I had some spare time to fix your code. You have string/bytes mismatching all over and unreferenced objects that were not in your original code. Here's an example that should work and give you a base to off of.
from uuid import uuid4
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet.protocol import Factory, Protocol
from twisted.protocols.basic import LineReceiver
class LnRcv(LineReceiver):
# Client timeout period in seconds
TIMEOUT = 10
def timeOut(self):
print ("Client: %s. %s" % (self.addr, 'Connection Timed out' ))
self.transport.loseConnection()
def connectionMade(self):
self.factory.NUM_CLIENTS += 1
print(f"Connected......number of connections = {self.factory.NUM_CLIENTS}")
peer = self.transport.getPeer()
self.addr = f"{peer.host}:{peer.port}"
self.name = uuid4().hex[:9].upper()
self.transport.write(f"Hello! I temporarily set your ID to {self.name}. What is your name? ".encode("utf8"))
# Start client timeout timer
self.timeout = reactor.callLater(self.TIMEOUT, self.timeOut)
def connectionLost(self, reason):
self.factory.NUM_CLIENTS -= 1
print(f"- {self.name} left because:\n{reason.getErrorMessage()}")
print(f"- Client disconnected: - {self.addr}")
print(f"- Number of connections = {self.factory.NUM_CLIENTS}")
def lineReceived(self, line):
self.sendLine(f"Welcome, {line.decode('utf8')}!".encode("utf8"))
class DataFactory(Factory):
protocol = LnRcv
NUM_CLIENTS = 0
def main():
print("Started...Listening for incoming connections.")
server = TCP4ServerEndpoint(reactor, 10003)
server.listen(DataFactory())
reactor.run()
if __name__ == "__main__":
main()
I am creating a robot which is going to be driven by the commands received over TCP connection. Therefore, I will have a robot class with methods (e.g. sense(), drive()...) and the class for TCP connection.
To establish TCP connection, I looked at examples from twisted. On the client side, I have written a client.py script for connection handling:
from twisted.internet import reactor, protocol
import random
from eventhook import EventHook
import common
#from Common.socketdataobjects import response
# a client protocol
class EchoClient(protocol.Protocol):
"""Once connected, send a message, then print the result."""
def connectionMade(self):
self.transport.write("hello, world!")
#the server should be notified that the connection to the robot has been established
#along with robot state (position)
#eventConnectionEstablishedHook.fire()
def dataReceived(self, data):
print "Server said:", data
self.transport.write("Hello %s" % str(random.randint(1,10)))
'''
serverMessage = common.deserializeJson(data)
command = serverMessage.command
arguments = serverMessage.arguments
#here we get for example command = "DRIVE"
#arguments = {motor1Speed: 50, motor2Speed: 40}
instead of above response, used for testing purposes,
the commands should be extracted from the data and according to the command,
the method in Robot instance should be called.
When the command execution finishes, the self.transport.write() method should be called
to notify the server that the command execution finished
'''
def connectionLost(self, reason):
print "connection lost"
class EchoFactory(protocol.ClientFactory):
protocol = EchoClient
def clientConnectionFailed(self, connector, reason):
print "Connection failed - goodbye!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
print "Connection lost - goodbye!"
reactor.stop()
# this connects the protocol to a server runing on port 8000
def initializeEventHandlers(connectionEstablishedHook):
global connection
connection.established = 0
global eventConnectionEstablishedHook
eventConnectionEstablishedHook = connectionEstablishedHook
def main():
f = EchoFactory()
reactor.connectTCP("localhost", 8000, f)
reactor.run()
# this only runs if the module was *not* imported
if __name__ == '__main__':
main()
Beside this script, I have a robot class:
Class Robot(object():
def __init(self)__:
self.position = (0,0)
def drive(self, speedMotor1, speedMotor2, driveTime)
updateMotor1State(speedMotor1)
updateMotor2State(speedMotor2)
time.sleep(driveTime)
#when the execution finished, the finish status should be sent to client in order to inform the server
return "Finished"
def sense(self)
#logic to get the data from the environment
What I would like to do, is to receive the data(commands) from TCP connection and then call the according method in Robot instance. Some procedures might take longer (e.g. driving), so I tried to use events, but haven't figured out the appropriate way to communicate between TCP client and robot using events:
if __name__ == '__main__':
robotController = Robot()
eventController = Controller()
connectionEstablishedHook = EventHook()
client.initializeEventHandlers(connectionEstablishedHook)
eventController.connection = connectionEstablishedHook
client.main()
I tried to create ClientMainProgram script, where I wanted to create an instance of a robot, an instance of TCP client and implement the communication between them using events.
Previously I have managed to implement event handling using Michael Foord's events pattern on a simpler example. I would be very thankful if anyone could provide the solution to this question or any similar example which might be helpful to solve this problem.
Events are easily represented using regular Python function calls.
For example, if your protocol looks like this:
from twisted.internet.protocol import Protocol
class RobotController(Protocol):
def __init__(self, robot):
self.robot = robot
def dataReceived(self, data):
for byte in data:
self.commandReceived(byte)
def commandReceived(self, command):
if command == "\x00":
# drive:
self.robot.drive()
elif command == "\x01":
# sense:
self.robot.sense()
...
(The specifics of the protocol used in this example are somewhat incidental. I picked this protocol because it's very simple and has almost no parsing logic. For your real application I suggest you use twisted.protocols.amp.)
Then all you need to do is make sure the robot attribute is properly initialized. You can do this easily using the somewhat newer endpoint APIs that can often replace use of factories:
from sys import argv
from twisted.internet.endpoints import clientFromString, connectProtocol
from twisted.internet.task import react
def main(reactor, description):
robot = ...
endpoint = clientFromString(reactor, description)
connecting = connectProtocol(endpoint, RobotController(robot))
def connected(controller):
...
connecting.addCallback(connected)
return connecting
react(main, argv[1:])
I'm new to twisted framework.
And I know there are many callback function will trigger automatically
When the connection made or lost.
But I have no idea how to send the data without those callbacks.
For example , I want to put an method custom_write() for sending the data out.
def custom_write(self,data):
self.transport.write(
data)
And trigger the function in my main(): method.
def main():
try:
p_red("I'm Client")
f = EchoFactory()
reactor.connectTCP("localhost",
8000,
f)
by the reactor.custom_write("HAHAHA")
And what if I create multiple reactor binding in different port.
Eg: localhost:1234, localhsot:5678
and send the different two messages to the 2 connections.
Eg: "Thanks" to port 1234 , Bye~ to port 5678
Any information can give me.
Thanks
class EchoClient(protocol.Protocol):
def connectionMade(self):
self.transport.write(
"I'm cli")
def custom_write(self,data):
self.transport.write(
data)
def dataReceived(self, data):
print "Server said:", data
self.transport.loseConnection()
pass
def connectionLost(self, reason):
print("[{0}] Lose connection...".format(
self.__class__.__name__)
)
pass
class EchoFactory(protocol.ClientFactory):
protocol = EchoClient
"""docstring for EchoFactory"""
def clientConnectionFailed(self,
connector,
reason):
print "[{0}] Connection failed - goodbye".format(
self.__class__.__name__)
reactor.stop()
def clientConnectionLost(self,
connector,
reason):
print "[{0}] Connection lost - goodbye".format(
self.__class__.__name__)
reactor.stop()
def main():
try:
p_red("I'm Client")
f = EchoFactory()
reactor.connectTCP("localhost",
8000,
f)
try:
reactor.run()
except BaseException as e:
traceback.print_exc(file=sys.stdout)
raise e
pass
except BaseException as e:
traceback.print_exc(file=sys.stdout)
raise e
pass
You can call connectTCP() more than once and using different hosts, ports. connectTCP() returns immediately without waiting for the full exchange to complete. To send different strings, you could pass them into the factories that can make them available to protocols.
I'm trying to create a client/server using Twisted.
I'd like to create a daemon, which will be connected to another server as a client and act as a server for other clients.
I've writen something like that which I think describes my problem:
server = sys.argv[1]
control_port = 8001
class ControlClient(protocol.Protocol):
def makeConnection(self, transport):
[some code here -snip-]
self.firstOrder(order, transport)
def firstOrder(self, action, transport):
self.t = transport
self.t.write(action + "\0")
def sendOrder(self, action):
self.t.write(action + "\0")
def dataReceived(self, data):
[some code here -snip-]
[HERE I WANT TO SEND DATA TO CLIENTS CONNECTED TO MY TWISTED SERVER, USING CONTROL SERVER]
class ControlServer(ControlClient):
def dataReceived(self, data):
print "client said " + data
def makeConnection(self, transport):
self.t = transport
self.t.write("make connection")
print "make connection"
def sendData(self, data):
self.t.write("data")
class ClientFactory(protocol.ClientFactory):
protocol = ControlClient
def clientConnectionFailed(self, connector, reason):
print "Connection failed - goodbye!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
print "Connection lost - goodbye!"
reactor.stop()
class ServerFactory(protocol.ServerFactory):
protocol = ControlServer
def main():
c = ClientFactory()
reactor.connectTCP(server, control_port, c)
s = ServerFactory()
reactor.listenTCP(9000, s)
reactor.run()
if __name__ == '__main__':
main()
As you can see, I'd like to send (as a server) some data received (as a client). My problem is of course my ServerControl is not instantiated in my ClientControl so I don't have access to transport which is required to send data to clients.
The only thing you seem to be missing is that you can keep a list of your client connections and make that list available to the code that's trying to send out data to all the clients.
There's an example of this in the Twisted FAQ: http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputononeconnectionresultinoutputonanother
That example only has one factory, but the idea is the same. To handle your case with two factories, just give one factory a reference to the other.
I have the following code (almost an exact copy of the Chat server example listed here:
import twisted.scripts.twistd
from twisted.protocols import basic
from twisted.internet import protocol, reactor
from twisted.application import service, internet
class MyChat(basic.LineReceiver):
def connectionMade(self):
print "Got new client!"
self.factory.clients.append(self)
def connectionLost(self, reason):
print "Lost a client!"
self.factory.clients.remove(self)
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.message(line)
def message(self, message):
self.transport.write(message + '\n')
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
if __name__ == "__main__":
print "Building reactor...."
reactor.listenTCP(50000, factory)
print "Running ractor...."
reactor.run()
else:
application = service.Application("chatserver")
internet.TCPServer(50000, factory).setServiceParent(application)
The server runs without error, and if I connect to it via Telnet, I can send data and the server prints to the console and relays it to all clients (as is expected). However, if I connect to it via a different tool (a MUD client), it never gets the data.
I have ensured that the client is sending the data (Traced the packets with Wireshark, and they're going across the wire), but the server either never receives it, or is choosing to ignore it for some reason.
I have tried this with two MUD clients, gmud, and JMC. If it is important, I am running Windows 7 x64.
Does anyone have any idea why this could be happening?
Thanks,
Mike
EDIT:
Thanks to the hints provided by Maiku Mori, I tried adding another method that was specified in the Twisted API Docs, dataReceived. Once this was added, the MUD clients worked perfectly, but Telnet is now sending every character as it's own set of data, instead of waiting for the user to press Enter.
Here's a snipped of the new code:
def dataReceived(self, data):
print "Dreceived", repr(data)
for c in self.factory.clients:
c.message(data)
# def lineReceived(self, line):
# print "received", repr(line)
# for c in self.factory.clients:
# c.message(line)
Has anyone experiences this before, and if so, how do you get around it? Ideally, I would like Telnet and MUD clients to work with this application.
Thanks again.
In case anyone stumbles across this question with similar problems, I'm leaving my findings as the accepted answer so that people don't have to hunt the way I did.
I fixed the issue by changing the delimiter value from in my Twisted protocol from "\r\n" (default), to just "\n" (which is what my MUD clients send. This means that in Telnet, when you enter the string:
Hello, World
Your application will receive it as:
Hello, World\r
You may need to do data sanitation on the server side to keep things in order. My final code was as follows:
import twisted.scripts.twistd
from twisted.protocols import basic
from twisted.internet import protocol, reactor
from twisted.application import service, internet
class MyChat(basic.LineReceiver):
def __init__(self):
self.delimiter = "\n"
def connectionMade(self):
print "Got new client!"
self.factory.clients.append(self)
def connectionLost(self, reason):
print "Lost a client!"
self.factory.clients.remove(self)
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.message(line)
def message(self, message):
self.transport.write(message + '\n')
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
if __name__ == "__main__":
print "Building reactor...."
reactor.listenTCP(50000, factory)
print "Running ractor...."
reactor.run()
else:
application = service.Application("chatserver")
internet.TCPServer(50000, factory).setServiceParent(application)
Thanks for all the help.
Are you sure that the MUD clients send line ending chars after each line? The lineReceived will only be called after line ending char has been sent.
EDIT:
Here I found API docs for LineReceiver. You could play around with dataReceived method to see if you are actually getting any kind of data. If I recall you can use it just like lineReceived.