Develop Reference

How to be sure that server received full data fron the client (and if not - resend it) using python - twisted?

I'm new in twisted and in web programming itself.
What I want is to implement server and client (and pass some string data between them). The problem is, that the data is important, so I want client to resend it to server in case of loosing connection or in case it wasn't full on the server side. But I don't want to resend it in case it was fully received, so I don't think that just adding the logic to def connectionLost() will do. How could this be done?
This is my server (just the same as in doc examples) and (after ------------) is the client:
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet import reactor
class ConsoleReceiver(Protocol):
def connectionMade(self):
self.transport.write(
"Welcome!\n".encode('utf-8'))
def dataReceived(self, data):
self.transport.write('Data received, thanks'.encode('utf-8'))
data = data.decode('ascii')
print(data)
self.transport.loseConnection()
class ServerFactory(Factory):
def buildProtocol(self, addr):
return ConsoleReceiver()
if __name__ == '__main__':
endpoint = TCP4ServerEndpoint(reactor, 21285)
endpoint.listen(ServerFactory())
reactor.run()```
-----------------------------------------------------------------------------
#some_flask_route.route('/test/')
urgent_information = <getting some urgent information from db with flask_sqlalchemy>
reactor.connectTCP('localhost', 21285, ShiftInfoSenderFactory(urgent_information))
reactor.run()
class ShiftInfoSender(Protocol):
def __init__(self, urgent_information):
self.urgent_information = urgent_information
def connectionMade(self):
self.transport.write('\nInformation to be red:{}\n'.format(self.urgent_information[1]).encode('utf-8'))
for i in self.urgent_information[2]:
self.transport.write('Some unpacked information: {}'.format(i).encode('utf-8')
def dataReceived(self, data):
print(data.decode('ascii'))
class ShiftInfoSenderFactory(ClientFactory):
def __init__(self, urgent_information):
self.urgent_information = urgent_information
def startedConnecting(self, connector):
print('Started to connect')
def buildProtocol(self, addr):
print('Connected')
return ShiftInfoSender(self.urgent_information)
def clientConnectionLost(self, connector, reason):
print('Lost connection. Reason:', reason)
def clientConnectionFailed(self, connector, reason):
print('Connection failed. Reason:', reason) ```

Twisted Python - Push data to websocket

I've a web-socket server which connects with the clients. Following is the code:-
from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor
class Chat(LineReceiver):
def __init__(self, users):
self.users = users
self.name = None
self.state = "GETNAME"
def connectionMade(self):
self.sendLine("What's your name?")
def connectionLost(self, reason):
if self.users.has_key(self.name):
del self.users[self.name]
def lineReceived(self, line):
if self.state == "GETNAME":
self.handle_GETNAME(line)
else:
self.handle_CHAT(line)
def handle_GETNAME(self, name):
if self.users.has_key(name):
self.sendLine("Name taken, please choose another.")
return
self.sendLine("Welcome, %s!" % (name,))
self.name = name
self.users[name] = self
self.state = "CHAT"
def handle_CHAT(self, message):
# Need to send the message to the connected clients.
class ChatFactory(Factory):
def __init__(self):
self.users = {} # maps user names to Chat instances
def buildProtocol(self, addr):
return Chat(self.users)
reactor.listenTCP(8123, ChatFactory())
reactor.run()
Clients get connected to the above code(server), and sends the data to the server.
Now, I've another python script, basically a scraper which scrapes the web, processes it and finally need to send the data to the connected clients.
script.py
while True:
# call `send_message` function and send data to the connected clients.
How can I achieve it?? Any example would be of great help!!
UPDATE
After using Autobahn
I've a server that fetches data from 3rd party API. I want to send this data to all the connected web-socket clients. Here is my code:-
class MyServerProtocol(WebSocketServerProtocol):
def __init__(self):
self.connected_users = []
self.send_data()
def onConnect(self, request):
print("Client connecting: {0}".format(request.peer))
def onOpen(self):
print("WebSocket connection open.")
self.connected_users.append(self) # adding users to the connected_list
def send_data(self):
# fetch data from the API and forward it to the connected_users.
for u in self.users:
print 1111
u.sendMessage('Hello, Some Data from API!', False)
def onClose(self, wasClean, code, reason):
connected_users.remove(self) # remove user from the connected list of users
print("WebSocket connection closed: {0}".format(reason))
if __name__ == '__main__':
import sys
from twisted.python import log
from twisted.internet import reactor
factory = WebSocketServerFactory(u"ws://127.0.0.1:9000")
factory.protocol = MyServerProtocol
reactor.listenTCP(9000, factory)
reactor.run()
My Server will never receive a message or probably will receive, but as of right now there's no such use-case, hence no need for OnMessage event for this example).
How do I write my send_data function in order to send data to all my connected clients??

You need to avoid this pattern when writing software with Twisted:
while True:
# call `send_message` function and send data to the connected clients.
Twisted is a cooperative multitasking system. "Cooperative" means that you have to give up control of execution periodically so that other tasks get a chance to run.
twisted.internet.task.LoopingCall can be used to replace many while ... loops (particularly while True loops):
from twisted.internet.task import LoopingCall
LoopingCall(one_iteration).start(iteration_interval)
This will call one_iteration every iteration_interval seconds. In between, it will give up control of execution so other tasks can run.
Making one_iteration send a message to a client is just a matter of giving one_iteration a reference to that client (or those clients, if there are many).
This is a variation on the FAQ How do I make Input on One Connection Result in Output on Another.
If you have a ChatFactory with a dict containing all your clients, just pass that factory to one_iteration:
LoopingCall(one_iteration, that_factory)
or
LoopingCall(lambda: one_iteration(that_factory))

How to handle TCP connection events in order to call methods within other class?

I am creating a robot which is going to be driven by the commands received over TCP connection. Therefore, I will have a robot class with methods (e.g. sense(), drive()...) and the class for TCP connection.
To establish TCP connection, I looked at examples from twisted. On the client side, I have written a client.py script for connection handling:
from twisted.internet import reactor, protocol
import random
from eventhook import EventHook
import common
#from Common.socketdataobjects import response
# a client protocol
class EchoClient(protocol.Protocol):
"""Once connected, send a message, then print the result."""
def connectionMade(self):
self.transport.write("hello, world!")
#the server should be notified that the connection to the robot has been established
#along with robot state (position)
#eventConnectionEstablishedHook.fire()
def dataReceived(self, data):
print "Server said:", data
self.transport.write("Hello %s" % str(random.randint(1,10)))
'''
serverMessage = common.deserializeJson(data)
command = serverMessage.command
arguments = serverMessage.arguments
#here we get for example command = "DRIVE"
#arguments = {motor1Speed: 50, motor2Speed: 40}
instead of above response, used for testing purposes,
the commands should be extracted from the data and according to the command,
the method in Robot instance should be called.
When the command execution finishes, the self.transport.write() method should be called
to notify the server that the command execution finished
'''
def connectionLost(self, reason):
print "connection lost"
class EchoFactory(protocol.ClientFactory):
protocol = EchoClient
def clientConnectionFailed(self, connector, reason):
print "Connection failed - goodbye!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
print "Connection lost - goodbye!"
reactor.stop()
# this connects the protocol to a server runing on port 8000
def initializeEventHandlers(connectionEstablishedHook):
global connection
connection.established = 0
global eventConnectionEstablishedHook
eventConnectionEstablishedHook = connectionEstablishedHook
def main():
f = EchoFactory()
reactor.connectTCP("localhost", 8000, f)
reactor.run()
# this only runs if the module was *not* imported
if __name__ == '__main__':
main()
Beside this script, I have a robot class:
Class Robot(object():
def __init(self)__:
self.position = (0,0)
def drive(self, speedMotor1, speedMotor2, driveTime)
updateMotor1State(speedMotor1)
updateMotor2State(speedMotor2)
time.sleep(driveTime)
#when the execution finished, the finish status should be sent to client in order to inform the server
return "Finished"
def sense(self)
#logic to get the data from the environment
What I would like to do, is to receive the data(commands) from TCP connection and then call the according method in Robot instance. Some procedures might take longer (e.g. driving), so I tried to use events, but haven't figured out the appropriate way to communicate between TCP client and robot using events:
if __name__ == '__main__':
robotController = Robot()
eventController = Controller()
connectionEstablishedHook = EventHook()
client.initializeEventHandlers(connectionEstablishedHook)
eventController.connection = connectionEstablishedHook
client.main()
I tried to create ClientMainProgram script, where I wanted to create an instance of a robot, an instance of TCP client and implement the communication between them using events.
Previously I have managed to implement event handling using Michael Foord's events pattern on a simpler example. I would be very thankful if anyone could provide the solution to this question or any similar example which might be helpful to solve this problem.

Events are easily represented using regular Python function calls.
For example, if your protocol looks like this:
from twisted.internet.protocol import Protocol
class RobotController(Protocol):
def __init__(self, robot):
self.robot = robot
def dataReceived(self, data):
for byte in data:
self.commandReceived(byte)
def commandReceived(self, command):
if command == "\x00":
# drive:
self.robot.drive()
elif command == "\x01":
# sense:
self.robot.sense()
...
(The specifics of the protocol used in this example are somewhat incidental. I picked this protocol because it's very simple and has almost no parsing logic. For your real application I suggest you use twisted.protocols.amp.)
Then all you need to do is make sure the robot attribute is properly initialized. You can do this easily using the somewhat newer endpoint APIs that can often replace use of factories:
from sys import argv
from twisted.internet.endpoints import clientFromString, connectProtocol
from twisted.internet.task import react
def main(reactor, description):
robot = ...
endpoint = clientFromString(reactor, description)
connecting = connectProtocol(endpoint, RobotController(robot))
def connected(controller):
...
connecting.addCallback(connected)
return connecting
react(main, argv[1:])

Sending message from one server to another in Twisted

I'm a complete Twisted AND Python noob, so my apologies if any of my terminology is wrong or anything I've done is silly. Nonetheless....
I've implemented my servers in the following way:
def makeServer(application, port):
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
tempServer = internet.TCPServer(port, factory)
tempServer.setServiceParent(application)
return tempServer
application = service.Application("chatserver")
server1 = makeServer(application, port=1025)
server2 = makeServer(application, port=1026)
server3 = makeServer(application, port=1027)
Note that MyChat is an event handling class that has a "receiveMessage" action:
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.transport.write(message + '\n')
I want server1 to be able to pass messages to server2. Rather, I want server1 to be treated as a client of server2. If server1 receives the message "hi" then I want it to send that same exact message to server2. The only thing server1 needs to be able to do is to send the message it received from its client to server2.
How can I accomplish this?
NOTE: You can totally change the way I'm implementing my server if it helps.

Different parts of your application can interact with each other using method calls.
Send a message to server2 really just means Call a method on one of the objects related to server2.
For example, in MyChat, you might have:
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.transport.write(message + '\n')
for server in self.factory.otherServers:
server.otherServerMessage(self, line)
This supposes a couple things:
You add a new otherServers attribute to your factory. Its contents are objects related to the other listening servers you have set up. These might be factory objects or protocol objects. It depends on what's most convenient based on what you intend to do with the message.
You give those related objects a new method, otherServerMessage, to handle messages delivered this way. If you were to deliver the messages directly to MyChat.lineReceived (which you easily could, if you wanted) then I would expect you to end up with infinite recursion; having a different method lets you differentiate between messages received from a client and messages received from another server.

You will probably need to implement a separate client. It is possible that an object can be both a client and a server, but I doubt it will be worth it and you are likely to run into trouble.
I suggest that the server instantiates a client object, which you connect to the 'next' server. The client can for example be an instance variable on the server.
Example:
class MyChat(LineReceiver):
def connectionMade(self):
print "Proxy: connected"
factory = protocol.ClientFactory()
class Proxy(protocol.Protocol):
def relayMessage(self, msg):
self.transport.write(msg)
factory.protocol = Proxy
point = TCP4ClientEndpoint(reactor, "localhost", 1025)
conn = point.connect(factory)
conn.addCallback(self.hasConnection)
def hasConnection(self, client):
print "Proxy: Connected to relay", client
self.client = client
def lineReceived(self, line):
print "Proxy: received", repr(line)
self.client.transport.write(line+"\n")
class MyEcho(LineReceiver):
def lineReceived(self, line):
print "Echo: received", repr(line)
factory = protocol.ServerFactory()
factory.protocol = MyChat
reactor.listenTCP(1024, factory)
factory = protocol.ServerFactory()
factory.protocol = MyEcho
reactor.listenTCP(1025, factory)

You need just declare clients inside your server, like this:
factory = SomeClientFactory('ws://127.0.0.1')
connectWS(factory)
and in your Client Class:
class SomeClient(WebSocketClientProtocol):
def __init__(self):
pass
def sendCommand(self):
self.sendMessage('A message to another server')
def onOpen(self):
self.sendCommand()
def onClose(self, wasClean, code, reason):
print(reason)
def onMessage(self, payload, isBinary):
print('A answer from another server')
class SomeClientFactory(WebSocketClientFactory):
def __init__(self, url):
WebSocketClientFactory.__init__(self,url)
self.proto = DeltaClient()
self.proto.factory = self
def buildProtocol(self, addr):
return self.proto
Tip: use a "Controller" class to manage that instances of clients inside your servers.

Python twisted: Functions are not called properly?

I've got a problem with setting up a client which connects to a "distributor" server to send certain data.
The server's purpose is to get data from the client and then send that data to it's all connected clients. The server works without any issues.
The main client is also supposed to work as an IRC bot.
Here's a text example of how it should work like:
(IRC) John: Hello there!
1. The IRC client got the message, we need to send it to the distributor now.
2. Distributor should get this "John: Hello there!" string and send it back to it's all connected clients.
3. If other clients send data to the distributor, which this will broadcast to all clients, the IRC client should output at it's turn the received data to a specified channel
The following code is the IRC bot client (ircbot.py):
import sys
import socket
import time
import traceback
from twisted.words.protocols import irc
from twisted.internet import reactor
from twisted.internet import protocol
VERBOSE = True
f = None
class IRCBot(irc.IRCClient):
def _get_nickname(self):
return self.factory.nickname
nickname = property(_get_nickname)
def signedOn(self):
self.msg("NickServ", "id <password_removed>") # Identify the bot
time.sleep(0.1) # Wait a little...
self.join(self.factory.channel) # Join channel #chantest
print "Signed on as %s." % (self.nickname,)
def joined(self, channel):
print "Joined %s." % (channel,)
def privmsg(self, user, channel, msg):
name = user.split('!', 1)[0]
prefix = "%s: %s" % (name, msg)
print prefix
if not user:
return
if self.nickname in msg:
msg = re.compile(self.nickname + "[:,]* ?", re.I).sub('', msg)
print msg
else:
prefix = ''
if msg.startswith("!"):
if name.lower() == "longdouble":
self.msg(channel, "Owner command") # etc just testing stuff
else:
self.msg(channel, "Command")
if channel == "#testchan" and name != "BotName":
EchoClient().sendData('IRC:'+' '.join(map(str, [name, msg])))
# This should make the bot send chat data to the distributor server (NOT IRC server)
def irc_NICK(self, prefix, params):
"""Called when an IRC user changes their nickname."""
old_nick = prefix.split('!')[0]
new_nick = params[0]
self.msg(, "%s is now known as %s" % (old_nick, new_nick))
def alterCollidedNick(self, nickname):
return nickname + '1'
class BotFactory(protocol.ClientFactory):
protocol = IRCBot
def __init__(self, channel, nickname='BotName'):
self.channel = channel
self.nickname = nickname
def clientConnectionLost(self, connector, reason):
print "Lost connection (%s), reconnecting." % (reason,)
connector.connect()
def clientConnectionFailed(self, connector, reason):
print "Could not connect: %s" % (reason,)
class EchoClient(protocol.Protocol):
def connectionMade(self):
pass
def sendData(self, data):
self.transport.write(data)
def dataReceived(self, data):
if VERBOSE:
print "RECV:", data
IRC.msg("#chantest", data)
#This one should send the received data from the distributor to the IRC channel
def connectionLost(self, reason):
print "Connection was lost."
class EchoFactory(protocol.ClientFactory):
def startedConnecting(self, connector):
print 'Started to connect.'
def buildProtocol(self, addr):
print 'Connected to the Distributor'
return EchoClient()
def clientConnectionFailed(self, connector, reason):
print "Cannot connect to distributor! Check all settings!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
print "Distributor Lost connection!!"
reactor.stop()
if __name__ == "__main__":
IRC = BotFactory('#chantest')
reactor.connectTCP('irc.rizon.net', 6667, IRC) # Our IRC connection
f = EchoFactory()
reactor.connectTCP("localhost", 8000, f) # Connection to the Distributor server
reactor.run()
The following code is the distributor server (distributor.py):
(This one works fine, but maybe it could be useful for further reference)
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
class MultiEcho(Protocol):
def __init__(self, factory):
self.factory = factory
def connectionMade(self):
print "Client connected:",self
self.factory.echoers.append(self)
self.factory.clients = self.factory.clients+1
#self.transport.write("Welcome to the server! There are currently "+`self.factory.clients`+" clients connected.")
def dataReceived(self, data):
print "RECV:",data
for echoer in self.factory.echoers:
echoer.transport.write(data)
def connectionLost(self, reason):
print "Client disconnected:",self
self.factory.echoers.remove(self)
self.factory.clients = self.factory.clients-1
class MultiEchoFactory(Factory):
def __init__(self):
self.clients = 0
self.names = []
self.echoers = []
def buildProtocol(self, addr):
return MultiEcho(self)
if __name__ == '__main__':
print "Running..."
reactor.listenTCP(8000, MultiEchoFactory())
reactor.run()
I want the client to output all incoming chat data from the IRC server to the "distributor" server and also output incoming data from the "distributor".
However, I get errors like this:
For the following line in ircbot.py,
EchoClient().sendData('IRC'+' '.join(map(str, [name, msg])))
I get the following error:
Joined #chantest.
Longdouble: test
Traceback (most recent call last):
File "C:\Python\lib\site-packages\twisted\internet\tcp.py", line 460, in doRea
d
return self.protocol.dataReceived(data)
File "C:\Python\lib\site-packages\twisted\words\protocols\irc.py", line 2277,
in dataReceived
basic.LineReceiver.dataReceived(self, data.replace('\r', ''))
File "C:\Python\lib\site-packages\twisted\protocols\basic.py", line 564, in da
taReceived
why = self.lineReceived(line)
File "C:\Python\lib\site-packages\twisted\words\protocols\irc.py", line 2285,
in lineReceived
self.handleCommand(command, prefix, params)
--- <exception caught here> ---
File "C:\Python\lib\site-packages\twisted\words\protocols\irc.py", line 2329,
in handleCommand
method(prefix, params)
File "C:\Python\lib\site-packages\twisted\words\protocols\irc.py", line 1813,
in irc_PRIVMSG
self.privmsg(user, channel, message)
File "C:\Python\Traance\kwlbot\ircbot.py", line 51, in privmsg
EchoClient().sendData('IRC'+' '.join(map(str, [name, msg])))
File "C:\Python\Traance\kwlbot\ircbot.py", line 90, in sendData
self.transport.write(data)
exceptions.AttributeError: 'NoneType' object has no attribute 'write'
And same goes to this line in the same ircbot.py
IRC.msg("#chantest", data)
->
RECV: Hello from Distributor Server
Traceback (most recent call last):
File "C:\Python\Traance\kwlbot\ircbot.py", line 96, in dataReceived
IRC.msg("#chantest", data)
AttributeError: BotFactory instance has no attribute 'msg'
What am I doing wrong? How can I call the right function from the IRCbot class to make it send the data to the distributor server and data received from the distributor server to output in the specified channel via IRC?
Any suggestions and possible solutions are welcome.
If I missed any other details, please let me know.
Thank you for your time!

You should avoid writing blocking code like this:
def signedOn(self):
self.msg("NickServ", "id <password_removed>") # Identify the bot
time.sleep(0.1) # Wait a little...
self.join(self.factory.channel) # Join channel #chantest
print "Signed on as %s." % (self.nickname,)
For details, see Tail -f log on server, process data, then serve to client via twisted.
Apart from that, the main problem here is that you are trying to send data without having a connection. When you write something like:
EchoClient().sendData('IRC'+' '.join(map(str, [name, msg])))
you're creating a protocol instance which is responsible for handling a connection and then trying to use it, but you're not creating a connection. The attempt to send data fails because the protocol hasn't been attached to any transport.
Your snippet already demonstrates the correct way to create a connection, twice in fact:
IRC = BotFactory('#chantest')
reactor.connectTCP('irc.rizon.net', 6667, IRC) # Our IRC connection
f = EchoFactory()
reactor.connectTCP("localhost", 8000, f) # Connection to the Distributor server
The mistake is creating a new EchoClient instance, one with no connection. The reactor.connectTCP call creates a new connection and a new EchoClient instance and associates them with each other.
Instead of EchoClient().sendData(...), you want to use the EchoClient instance created by your factory:
def buildProtocol(self, addr):
print 'Connected to the Distributor'
return EchoClient()
Your buildProtocol implementation creates the instance, all that's missing is for it to save the instance so it can be used by your IRC bot.
Consider something like this:
def buildProtocol(self, addr):
print 'Connected to the Distributor'
self.connection = EchoClient()
return self.connection
Your IRC client can then use the saved EchoClient instance:
if channel == "#testchan" and name != "BotName":
f.connection.sendData('IRC:'+' '.join(map(str, [name, msg])))
# This should make the bot send chat data to the distributor server (NOT IRC server)
Note that the specific code I give here is a very crude approach. It uses the global variable f to find the EchoFactory instance. As with most global variable usage this makes the code a little hard to follow. Further, I haven't added any code to handle connectionLost events to clear the connection attribute out. This means you might think you're sending data to the distributed server when the connection has already been lost. And similarly, there's no guarantee that the connection to the distributed server will have been created by the time the IRC client first tries to use it, so you may have an AttributeError when it tries to use f.connection.sendData.
However, fixing these doesn't require much of a leap. Fix the global variable usage as you would any other - by passing arguments to functions, saving objects as references on other objects, etc. Fix the possible AttributeError by handling it, or by not creating the IRC connection until after you've created the distributed connection, etc. And handle lost connections by resetting the attribute value to None or some other sentinel, and paying attention to such a case in the IRC code before trying to use the distributed client connection to send any data.

TFM is never defined in your code, so I don't know what the deal is there.
The other error is that you're instantiating a client, but never connecting it to anything, as with reactor.connectTCP(...) or endpoint.connect(...). The transport attribute will be None until it's set by something.
(It would be helpful for you to come up with a simpler version of this code which is complete and doesn't include unnecessary details like all the printed log messages. It makes it harder to see what the real issues are.)