Develop Reference

Twisted Python - Push data to websocket

I've a web-socket server which connects with the clients. Following is the code:-
from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor
class Chat(LineReceiver):
def __init__(self, users):
self.users = users
self.name = None
self.state = "GETNAME"
def connectionMade(self):
self.sendLine("What's your name?")
def connectionLost(self, reason):
if self.users.has_key(self.name):
del self.users[self.name]
def lineReceived(self, line):
if self.state == "GETNAME":
self.handle_GETNAME(line)
else:
self.handle_CHAT(line)
def handle_GETNAME(self, name):
if self.users.has_key(name):
self.sendLine("Name taken, please choose another.")
return
self.sendLine("Welcome, %s!" % (name,))
self.name = name
self.users[name] = self
self.state = "CHAT"
def handle_CHAT(self, message):
# Need to send the message to the connected clients.
class ChatFactory(Factory):
def __init__(self):
self.users = {} # maps user names to Chat instances
def buildProtocol(self, addr):
return Chat(self.users)
reactor.listenTCP(8123, ChatFactory())
reactor.run()
Clients get connected to the above code(server), and sends the data to the server.
Now, I've another python script, basically a scraper which scrapes the web, processes it and finally need to send the data to the connected clients.
script.py
while True:
# call `send_message` function and send data to the connected clients.
How can I achieve it?? Any example would be of great help!!
UPDATE
After using Autobahn
I've a server that fetches data from 3rd party API. I want to send this data to all the connected web-socket clients. Here is my code:-
class MyServerProtocol(WebSocketServerProtocol):
def __init__(self):
self.connected_users = []
self.send_data()
def onConnect(self, request):
print("Client connecting: {0}".format(request.peer))
def onOpen(self):
print("WebSocket connection open.")
self.connected_users.append(self) # adding users to the connected_list
def send_data(self):
# fetch data from the API and forward it to the connected_users.
for u in self.users:
print 1111
u.sendMessage('Hello, Some Data from API!', False)
def onClose(self, wasClean, code, reason):
connected_users.remove(self) # remove user from the connected list of users
print("WebSocket connection closed: {0}".format(reason))
if __name__ == '__main__':
import sys
from twisted.python import log
from twisted.internet import reactor
factory = WebSocketServerFactory(u"ws://127.0.0.1:9000")
factory.protocol = MyServerProtocol
reactor.listenTCP(9000, factory)
reactor.run()
My Server will never receive a message or probably will receive, but as of right now there's no such use-case, hence no need for OnMessage event for this example).
How do I write my send_data function in order to send data to all my connected clients??

You need to avoid this pattern when writing software with Twisted:
while True:
# call `send_message` function and send data to the connected clients.
Twisted is a cooperative multitasking system. "Cooperative" means that you have to give up control of execution periodically so that other tasks get a chance to run.
twisted.internet.task.LoopingCall can be used to replace many while ... loops (particularly while True loops):
from twisted.internet.task import LoopingCall
LoopingCall(one_iteration).start(iteration_interval)
This will call one_iteration every iteration_interval seconds. In between, it will give up control of execution so other tasks can run.
Making one_iteration send a message to a client is just a matter of giving one_iteration a reference to that client (or those clients, if there are many).
This is a variation on the FAQ How do I make Input on One Connection Result in Output on Another.
If you have a ChatFactory with a dict containing all your clients, just pass that factory to one_iteration:
LoopingCall(one_iteration, that_factory)
or
LoopingCall(lambda: one_iteration(that_factory))

Twisted Python - IRC Client

first question here.
So for a club at school we are working on making a IRC client in Python and Twisted.
So I take the example bot that twisted gives you. I've managed to get it connected to a irc channel, and it logs.
I know I've probably gotta use 2 threads to have both reading from the server and input simultaneous, which I can achieve, but only if it command line input. Mind you it is still logging the data from the channel at the same time.
So to do this I used: d = threads.deferToThread(aSillyBlockingMethod)
Which calls my raw_input() loop.
My problem lies in not being able to figure out how to change this from just typing into and printing form the commandline; to being able to actually send messages to the irc server for other people to read.
Any help would be greatly appreciated. I am a novice python programmer and don't know too much web stuff; like protocols, ports, and stuff like that but I am slowly picking them up. If anyone knows of an easier way to do this let me know please I am not committed to using Twisted.
Here is my code, or rather the modified bot I'm tinkering with:
# twisted imports
from twisted.words.protocols import irc
from twisted.internet import reactor, protocol
from twisted.python import log
from twisted.internet import threads
# system imports
import time, sys
class MessageLogger:
"""
An independent logger class (because separation of application
and protocol logic is a good thing).
"""
def __init__(self, file):
self.file = file
def log(self, message):
"""Write a message to the file."""
timestamp = time.strftime("[%H:%M:%S]", time.localtime(time.time()))
self.file.write('%s %s\n' % (timestamp, message))
self.file.flush()
def close(self):
self.file.close()
class LogBot(irc.IRCClient):
"""A logging IRC bot."""
nickname = "twistedbot"
def connectionMade(self):
irc.IRCClient.connectionMade(self)
self.logger = MessageLogger(open(self.factory.filename, "a"))
self.logger.log("[connected at %s]" %
time.asctime(time.localtime(time.time())))
def connectionLost(self, reason):
irc.IRCClient.connectionLost(self, reason)
self.logger.log("[disconnected at %s]" %
time.asctime(time.localtime(time.time())))
self.logger.close()
# callbacks for events
def signedOn(self):
"""Called when bot has succesfully signed on to server."""
self.join(self.factory.channel)
def joined(self, channel):
"""This will get called when the bot joins the channel."""
self.logger.log("[I have joined %s]" % channel)
def privmsg(self, user, channel, msg):
"""This will get called when the bot receives a message."""
user = user.split('!', 1)[0]
self.logger.log("<%s> %s" % (user, msg))
# Check to see if they're sending me a private message
if channel == self.nickname:
msg = "It isn't nice to whisper! Play nice with the group."
self.msg(user, msg)
return
# Otherwise check to see if it is a message directed at me
if msg.startswith(self.nickname + ":"):
msg = "%s: I am a log bot" % user
self.msg(channel, msg)
self.logger.log("<%s> %s" % (self.nickname, msg))
def action(self, user, channel, msg):
"""This will get called when the bot sees someone do an action."""
user = user.split('!', 1)[0]
self.logger.log("* %s %s" % (user, msg))
# irc callbacks
def irc_NICK(self, prefix, params):
"""Called when an IRC user changes their nickname."""
old_nick = prefix.split('!')[0]
new_nick = params[0]
self.logger.log("%s is now known as %s" % (old_nick, new_nick))
# For fun, override the method that determines how a nickname is changed on
# collisions. The default method appends an underscore.
def alterCollidedNick(self, nickname):
"""
Generate an altered version of a nickname that caused a collision in an
effort to create an unused related name for subsequent registration.
"""
return nickname + '^'
def aSillyBlockingMethod(self):
import time
while True:
msg = raw_input()
print msg
class LogBotFactory(protocol.ClientFactory):
"""A factory for LogBots.
A new protocol instance will be created each time we connect to the server.
"""
def __init__(self, channel, filename):
self.channel = channel
self.filename = filename
def buildProtocol(self, addr):
p = LogBot()
p.factory = self
return p
def clientConnectionLost(self, connector, reason):
"""If we get disconnected, reconnect to server."""
connector.connect()
def clientConnectionFailed(self, connector, reason):
print "connection failed:", reason
reactor.stop()
if __name__ == '__main__':
# initialize logging
log.startLogging(sys.stdout)
# create factory protocol and application
f = LogBotFactory("#goon.squad.dev", "test.txt")
# connect factory to this host and port
reactor.connectTCP("irc.freenode.net", 6667, f)
#Use this to keep user input open
d = threads.deferToThread(aSillyBlockingMethod)
# run bot
reactor.run()
--TyrZaraki

I know I've probably gotta use 2 threads to have both reading from the server and input simultaneous, which I can achieve, but only if it command line input. Mind you it is still logging the data from the channel at the same time.
Actually, it's not necessary to use two threads for this. A major strength of Twisted is doing I/O without using threads. The question and answer Michael linked to talk about Twisted's non-threaded support for interacting with standard input and standard output via twisted.internet.stdio.StandardIO.
My problem lies in not being able to figure out how to change this from just typing into and printing form the commandline; to being able to actually send messages to the irc server for other people to read.
IRCClient has a method for sending messages to the IRC server - the sample code you included in your question even uses this method already, IRCClient.msg. All you need to do is call it.
Your threaded code could change like this to do so (again, threads are unnecessary, but in the interest of just showing you how to send a message from your input handling code, I'll base this part of the answer on threads, to avoid other changes to the code which might make the answer harder to understand. You do not need threads to do this.):
def aSillyBlockingMethod(bot):
import time
while True:
msg = raw_input()
bot.threadSafeMsg("#bottest", msg)
class LogBot(irc.IRCClient):
"""A logging IRC bot."""
nickname = "twistedbot"
def connectionMade(self):
irc.IRCClient.connectionMade(self)
self.logger = MessageLogger(open(self.factory.filename, "a"))
self.logger.log("[connected at %s]" %
time.asctime(time.localtime(time.time())))
# The bot is now connected. Start reading input here.
# Pass a reference to this protocol instance, so that
# messages can be sent to this protocol instance.
deferToThread(aSillyBlockingMethod, self)
# Define a helper function for aSillyBlockingMethod to use.
# Since aSillyBlockingMethod runs in a thread, it cannot just call
# IRCClient.msg, since that method - like almost all methods in Twisted -
# is not thread-safe. Instead it must call this thread-safe wrapper.
def threadSafeMsg(self, channel, message):
reactor.callFromThread(self.msg, channel, message)
Notice that all that's happening here is that aSillyBlockingMethod is calling a method on LogBot. The thread-safe wrapper would not be necessary when using StandardIO, since that eliminates the need for threads.

Python socket connection not working properly

Included below is the code that I am currently using:
soc1 = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
ADDR = (HOST,PORT)
soc1.connect(ADDR)
soc1.send('WILL SEND')
The error message that I receive when running the above code is:
Traceback (most recent call last):
File "C:\workspace\wx_python_test\chat_server.py", line 25, in <module>
soc1.connect(ADDR)
File "C:\Python27\lib\socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 10061]
Could anyone please explain what the issue is that I am experiencing and how I can correct it?
Full source code: http://pastie.org/4245314

socket.error: [Errno 10061] indicates that the port you are trying to connect to is not open. You need to make sure that the port is open and something is listening for your connection to be made.
It appears that you are trying to test a chat server. In order for the chat server to work properly you will want to make sure that it is currently listening on the specified port.
Twisted provides a good framework if you haven't checked it out previously.
from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor
class Chat(LineReceiver):
def __init__(self, users):
self.users = users
self.name = None
self.state = "GETNAME"
def connectionMade(self):
self.sendLine("What's your name?")
def connectionLost(self, reason):
if self.users.has_key(self.name):
del self.users[self.name]
def lineReceived(self, line):
if self.state == "GETNAME":
self.handle_GETNAME(line)
else:
self.handle_CHAT(line)
def handle_GETNAME(self, name):
if self.users.has_key(name):
self.sendLine("Name taken, please choose another.")
return
self.sendLine("Welcome, %s!" % (name,))
self.name = name
self.users[name] = self
self.state = "CHAT"
def handle_CHAT(self, message):
message = "<%s> %s" % (self.name, message)
for name, protocol in self.users.iteritems():
if ':' in message:
self.exc(message.split(':')[0])
if protocol != self:
protocol.sendLine(message)
def exc(self, cmd):
print cmd
if cmd == 'who':
for i in self.users:
print i
class ChatFactory(Factory):
def __init__(self):
self.users = {} # maps user names to Chat instances
def buildProtocol(self, addr):
return Chat(self.users)
reactor.listenTCP(8123, ChatFactory())
reactor.run()

Your python is fine - check your network config and that you can actually connect to the specified host/port. You are getting Windows socket error 10061 - this is defined below:.
WSAECONNREFUSED 10061 Connection refused. No connection could be made
because the target computer actively refused it. This usually results
from trying to connect to a service that is inactive on the foreign
host—that is, one with no server application running.
Try doing telnet host port from your terminal window - and see if you can connect. If not, resolve the network issue first.

Sending message from one server to another in Twisted

I'm a complete Twisted AND Python noob, so my apologies if any of my terminology is wrong or anything I've done is silly. Nonetheless....
I've implemented my servers in the following way:
def makeServer(application, port):
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
tempServer = internet.TCPServer(port, factory)
tempServer.setServiceParent(application)
return tempServer
application = service.Application("chatserver")
server1 = makeServer(application, port=1025)
server2 = makeServer(application, port=1026)
server3 = makeServer(application, port=1027)
Note that MyChat is an event handling class that has a "receiveMessage" action:
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.transport.write(message + '\n')
I want server1 to be able to pass messages to server2. Rather, I want server1 to be treated as a client of server2. If server1 receives the message "hi" then I want it to send that same exact message to server2. The only thing server1 needs to be able to do is to send the message it received from its client to server2.
How can I accomplish this?
NOTE: You can totally change the way I'm implementing my server if it helps.

Different parts of your application can interact with each other using method calls.
Send a message to server2 really just means Call a method on one of the objects related to server2.
For example, in MyChat, you might have:
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.transport.write(message + '\n')
for server in self.factory.otherServers:
server.otherServerMessage(self, line)
This supposes a couple things:
You add a new otherServers attribute to your factory. Its contents are objects related to the other listening servers you have set up. These might be factory objects or protocol objects. It depends on what's most convenient based on what you intend to do with the message.
You give those related objects a new method, otherServerMessage, to handle messages delivered this way. If you were to deliver the messages directly to MyChat.lineReceived (which you easily could, if you wanted) then I would expect you to end up with infinite recursion; having a different method lets you differentiate between messages received from a client and messages received from another server.

You will probably need to implement a separate client. It is possible that an object can be both a client and a server, but I doubt it will be worth it and you are likely to run into trouble.
I suggest that the server instantiates a client object, which you connect to the 'next' server. The client can for example be an instance variable on the server.
Example:
class MyChat(LineReceiver):
def connectionMade(self):
print "Proxy: connected"
factory = protocol.ClientFactory()
class Proxy(protocol.Protocol):
def relayMessage(self, msg):
self.transport.write(msg)
factory.protocol = Proxy
point = TCP4ClientEndpoint(reactor, "localhost", 1025)
conn = point.connect(factory)
conn.addCallback(self.hasConnection)
def hasConnection(self, client):
print "Proxy: Connected to relay", client
self.client = client
def lineReceived(self, line):
print "Proxy: received", repr(line)
self.client.transport.write(line+"\n")
class MyEcho(LineReceiver):
def lineReceived(self, line):
print "Echo: received", repr(line)
factory = protocol.ServerFactory()
factory.protocol = MyChat
reactor.listenTCP(1024, factory)
factory = protocol.ServerFactory()
factory.protocol = MyEcho
reactor.listenTCP(1025, factory)

You need just declare clients inside your server, like this:
factory = SomeClientFactory('ws://127.0.0.1')
connectWS(factory)
and in your Client Class:
class SomeClient(WebSocketClientProtocol):
def __init__(self):
pass
def sendCommand(self):
self.sendMessage('A message to another server')
def onOpen(self):
self.sendCommand()
def onClose(self, wasClean, code, reason):
print(reason)
def onMessage(self, payload, isBinary):
print('A answer from another server')
class SomeClientFactory(WebSocketClientFactory):
def __init__(self, url):
WebSocketClientFactory.__init__(self,url)
self.proto = DeltaClient()
self.proto.factory = self
def buildProtocol(self, addr):
return self.proto
Tip: use a "Controller" class to manage that instances of clients inside your servers.

Create client/server with Twisted

I'm trying to create a client/server using Twisted.
I'd like to create a daemon, which will be connected to another server as a client and act as a server for other clients.
I've writen something like that which I think describes my problem:
server = sys.argv[1]
control_port = 8001
class ControlClient(protocol.Protocol):
def makeConnection(self, transport):
[some code here -snip-]
self.firstOrder(order, transport)
def firstOrder(self, action, transport):
self.t = transport
self.t.write(action + "\0")
def sendOrder(self, action):
self.t.write(action + "\0")
def dataReceived(self, data):
[some code here -snip-]
[HERE I WANT TO SEND DATA TO CLIENTS CONNECTED TO MY TWISTED SERVER, USING CONTROL SERVER]
class ControlServer(ControlClient):
def dataReceived(self, data):
print "client said " + data
def makeConnection(self, transport):
self.t = transport
self.t.write("make connection")
print "make connection"
def sendData(self, data):
self.t.write("data")
class ClientFactory(protocol.ClientFactory):
protocol = ControlClient
def clientConnectionFailed(self, connector, reason):
print "Connection failed - goodbye!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
print "Connection lost - goodbye!"
reactor.stop()
class ServerFactory(protocol.ServerFactory):
protocol = ControlServer
def main():
c = ClientFactory()
reactor.connectTCP(server, control_port, c)
s = ServerFactory()
reactor.listenTCP(9000, s)
reactor.run()
if __name__ == '__main__':
main()
As you can see, I'd like to send (as a server) some data received (as a client). My problem is of course my ServerControl is not instantiated in my ClientControl so I don't have access to transport which is required to send data to clients.

The only thing you seem to be missing is that you can keep a list of your client connections and make that list available to the code that's trying to send out data to all the clients.
There's an example of this in the Twisted FAQ: http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputononeconnectionresultinoutputonanother
That example only has one factory, but the idea is the same. To handle your case with two factories, just give one factory a reference to the other.