Develop Reference

Connect to Twisted TCP server

I have set up a TCP server using the twisted example (with some modifications).
from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor
from os import path
import yaml
class User(LineReceiver):
def __init__(self,users):
self.users = users
self.name = None
def connectionMade(self):
print 'new connection'
self.sendLine('username:')
def connectionLost(self,reason):
print 'connection lost'
if not self.name == None:
msg = '%s has disconnected' % (self.name)
print msg
self.toAll(msg,None)
del self.users[self.name]
def lineRecieved(self,line):
print line
if self.name == None:
self.setName(line)
else:
self.toChat(line)
def toAll(self,msg,to_self):
for name, protocol in self.users.iteritems():
if protocol == self and not to_self == None:
self.sendLine(to_self)
else:
protocol.sendLine(msg)
def setName(self,name):
if self.users.has_key(name):
self.sendLine('username in use')
return
elif ' ' in name:
self.sendLine('no spaces!')
return
print 'new user %s' % (name)
self.sendLine('logged in as %s' % (name))
self.name = name
self.users[name] = self
def toChat(self,message):
msg = '<%s> %s' % (self.name,message)
print msg
to_self = '<%s (you)> %s' % (self.name,message)
self.toAll(msg,to_self)
class Main(Factory):
def __init__(self,motd=None):
self.users = {}
self.motd = motd
print 'loaded, waiting for connections...'
def buildProtocol(self,addr):
return User(self.users)
if not path.isfile('config.yml'):
open('config.yml','w').write('port: 4444\nmotd: don\'t spam')
with open('config.yml','r') as f:
dump = yaml.load(f.read())
motd = dump['motd']
port = dump['port']
reactor.listenTCP(port,Main(motd=motd))
reactor.run()
I was wondering how I would be able to connect to it? I've tried adapting their example Echo client and Echo server, but my server only gives a giant error when data is sent back to it.
(The echo server is here and the echo client is here)
The client I am using is
from twisted.internet import reactor
from twisted.internet.protocol import Protocol,ClientFactory
class Main(Protocol):
def dataReceived(self,data):
print data
self.transport.write(data)
class MainFactory(ClientFactory):
def buildProtocol(self,addr):
print 'connected'
return Main()
def clientConnectionLost(self,connector,reason):
print 'connection lost'
def clientConnectionFailed(self,connector,reason):
print 'connection failed'
reactor.connectTCP('localhost',4444,MainFactory())
reactor.run()
Here is a picture of the error
What do I need to do to send data back to the server? What class do I need to inherit from?

The problem is a simple typo.
LineReceiver calls its lineReceived method on each line. You're supposed to override that. But you don't, you define lineRecieved instead. So, you get the default implementation, which raises NotImplemented.
If you fix that, your code is still more than a little odd. Trace through the communication.
The client connects, which calls the server's User.connectionMade, which does this:
self.sendLine('username:')
So the client gets that in Main.dataReceived and does this:
self.transport.write(data)
So, it's sending the prompt back as a response.
The server will receive that in lineReceived (once you fix the name) and do this:
if self.name == None:
self.setName(line)
So, you're going to set the username to 'username:'.

Getting iOS app to talk to my home server

Currently, I have a home server in my closet basically doing nothing. It has Ubuntu Server 8.0.4 installed, apache for web development ill be using later, ssh, and python/twisted installed.
Here's the issue:
I created an app to talk to "localhost" port-40, using socket implementation, here is a link of what I did but what I want to develop off of : http://www.raywenderlich.com/3932/how-to-create-a-socket-based-iphone-app-and-server
Now connecting to the localhost is no problem but I want to expand this to work with my server.
I implemented the python protocol onto my server and changed the ip address im accessing in the iOS app. Here's the implementation I have, it's exactly the same as the tutorial except for the port i'm accessing.
from twisted.internet.protocol import Factory, Protocol
from twisted.internet import reactor
class IphoneChat(Protocol):
def connectionMade(self):
self.factory.clients.append(self)
print "clients are ", self.factory.clients
def connectionLost(self, reason):
self.factory.clients.remove(self)
def dataReceived(self, data):
a = data.split(':')
print a
#b = password.split(':')
#print b
if len(a) > 1:
command = a[0]
content = a[1]
#username = a[2]
#password = a[2]
msg = ""
#msg2 = ""
if command == "username":
self.name = data
msg = self.name + " has joined"
#self.name = password
#msg2 = self.name + " this is his password"
#print msg
elif command == "password":
self.name = data
msg = self.name + " this is their password"
elif command == "msg":
msg = self.name + ": " + data
print msg
#msg2 = self.name + ": " + password
#print msg2
for c in self.factory.clients:
c.message(msg)#1, msg2)
def message(self, message):
self.transport.write(message + '\n')
factory = Factory()
factory.protocol = IphoneChat #here1#
factory.clients = []
reactor.listenTCP(40, factory)
print "Iphone Chat server started"
reactor.run()
So the REAL issue is, I cannot connect clients to my server or something..... I'm sorry but I am very new to networking.
Any opinion will help.

If you've not established networking communication to your server from the internet before you'll want to establish that you can do that with simple test cases first, there's a lot that can stop traffic from the internet reaching your program on your server.
Is your server connected to the internet through a router? If so, can you communicate with the server from inside the local network (i.e. use the 192.168.xxx.xxx ip address), using something like netcat or telnet?
If that works you should try from outside the network (i.e. using the other ip address, from whatismyip.net or similar). If you've really no prior experience with networking you may have neglected to set up port forwarding, this is a setting on your router.
There are a lot of tutorials around teaching you how to set up a Ubuntu home server, I suggest learning how to host a (very) simple webpage as a means of learning how to network, this will help a lot with debugging a networked program like the one you're making.

Twisted Python - IRC Client

first question here.
So for a club at school we are working on making a IRC client in Python and Twisted.
So I take the example bot that twisted gives you. I've managed to get it connected to a irc channel, and it logs.
I know I've probably gotta use 2 threads to have both reading from the server and input simultaneous, which I can achieve, but only if it command line input. Mind you it is still logging the data from the channel at the same time.
So to do this I used: d = threads.deferToThread(aSillyBlockingMethod)
Which calls my raw_input() loop.
My problem lies in not being able to figure out how to change this from just typing into and printing form the commandline; to being able to actually send messages to the irc server for other people to read.
Any help would be greatly appreciated. I am a novice python programmer and don't know too much web stuff; like protocols, ports, and stuff like that but I am slowly picking them up. If anyone knows of an easier way to do this let me know please I am not committed to using Twisted.
Here is my code, or rather the modified bot I'm tinkering with:
# twisted imports
from twisted.words.protocols import irc
from twisted.internet import reactor, protocol
from twisted.python import log
from twisted.internet import threads
# system imports
import time, sys
class MessageLogger:
"""
An independent logger class (because separation of application
and protocol logic is a good thing).
"""
def __init__(self, file):
self.file = file
def log(self, message):
"""Write a message to the file."""
timestamp = time.strftime("[%H:%M:%S]", time.localtime(time.time()))
self.file.write('%s %s\n' % (timestamp, message))
self.file.flush()
def close(self):
self.file.close()
class LogBot(irc.IRCClient):
"""A logging IRC bot."""
nickname = "twistedbot"
def connectionMade(self):
irc.IRCClient.connectionMade(self)
self.logger = MessageLogger(open(self.factory.filename, "a"))
self.logger.log("[connected at %s]" %
time.asctime(time.localtime(time.time())))
def connectionLost(self, reason):
irc.IRCClient.connectionLost(self, reason)
self.logger.log("[disconnected at %s]" %
time.asctime(time.localtime(time.time())))
self.logger.close()
# callbacks for events
def signedOn(self):
"""Called when bot has succesfully signed on to server."""
self.join(self.factory.channel)
def joined(self, channel):
"""This will get called when the bot joins the channel."""
self.logger.log("[I have joined %s]" % channel)
def privmsg(self, user, channel, msg):
"""This will get called when the bot receives a message."""
user = user.split('!', 1)[0]
self.logger.log("<%s> %s" % (user, msg))
# Check to see if they're sending me a private message
if channel == self.nickname:
msg = "It isn't nice to whisper! Play nice with the group."
self.msg(user, msg)
return
# Otherwise check to see if it is a message directed at me
if msg.startswith(self.nickname + ":"):
msg = "%s: I am a log bot" % user
self.msg(channel, msg)
self.logger.log("<%s> %s" % (self.nickname, msg))
def action(self, user, channel, msg):
"""This will get called when the bot sees someone do an action."""
user = user.split('!', 1)[0]
self.logger.log("* %s %s" % (user, msg))
# irc callbacks
def irc_NICK(self, prefix, params):
"""Called when an IRC user changes their nickname."""
old_nick = prefix.split('!')[0]
new_nick = params[0]
self.logger.log("%s is now known as %s" % (old_nick, new_nick))
# For fun, override the method that determines how a nickname is changed on
# collisions. The default method appends an underscore.
def alterCollidedNick(self, nickname):
"""
Generate an altered version of a nickname that caused a collision in an
effort to create an unused related name for subsequent registration.
"""
return nickname + '^'
def aSillyBlockingMethod(self):
import time
while True:
msg = raw_input()
print msg
class LogBotFactory(protocol.ClientFactory):
"""A factory for LogBots.
A new protocol instance will be created each time we connect to the server.
"""
def __init__(self, channel, filename):
self.channel = channel
self.filename = filename
def buildProtocol(self, addr):
p = LogBot()
p.factory = self
return p
def clientConnectionLost(self, connector, reason):
"""If we get disconnected, reconnect to server."""
connector.connect()
def clientConnectionFailed(self, connector, reason):
print "connection failed:", reason
reactor.stop()
if __name__ == '__main__':
# initialize logging
log.startLogging(sys.stdout)
# create factory protocol and application
f = LogBotFactory("#goon.squad.dev", "test.txt")
# connect factory to this host and port
reactor.connectTCP("irc.freenode.net", 6667, f)
#Use this to keep user input open
d = threads.deferToThread(aSillyBlockingMethod)
# run bot
reactor.run()
--TyrZaraki

I know I've probably gotta use 2 threads to have both reading from the server and input simultaneous, which I can achieve, but only if it command line input. Mind you it is still logging the data from the channel at the same time.
Actually, it's not necessary to use two threads for this. A major strength of Twisted is doing I/O without using threads. The question and answer Michael linked to talk about Twisted's non-threaded support for interacting with standard input and standard output via twisted.internet.stdio.StandardIO.
My problem lies in not being able to figure out how to change this from just typing into and printing form the commandline; to being able to actually send messages to the irc server for other people to read.
IRCClient has a method for sending messages to the IRC server - the sample code you included in your question even uses this method already, IRCClient.msg. All you need to do is call it.
Your threaded code could change like this to do so (again, threads are unnecessary, but in the interest of just showing you how to send a message from your input handling code, I'll base this part of the answer on threads, to avoid other changes to the code which might make the answer harder to understand. You do not need threads to do this.):
def aSillyBlockingMethod(bot):
import time
while True:
msg = raw_input()
bot.threadSafeMsg("#bottest", msg)
class LogBot(irc.IRCClient):
"""A logging IRC bot."""
nickname = "twistedbot"
def connectionMade(self):
irc.IRCClient.connectionMade(self)
self.logger = MessageLogger(open(self.factory.filename, "a"))
self.logger.log("[connected at %s]" %
time.asctime(time.localtime(time.time())))
# The bot is now connected. Start reading input here.
# Pass a reference to this protocol instance, so that
# messages can be sent to this protocol instance.
deferToThread(aSillyBlockingMethod, self)
# Define a helper function for aSillyBlockingMethod to use.
# Since aSillyBlockingMethod runs in a thread, it cannot just call
# IRCClient.msg, since that method - like almost all methods in Twisted -
# is not thread-safe. Instead it must call this thread-safe wrapper.
def threadSafeMsg(self, channel, message):
reactor.callFromThread(self.msg, channel, message)
Notice that all that's happening here is that aSillyBlockingMethod is calling a method on LogBot. The thread-safe wrapper would not be necessary when using StandardIO, since that eliminates the need for threads.

What's wrong with my twisted server that's supposed to take a .exe and send its stdio to anyone who asks. Instead, it doesn't send anything

print 'Preall test works!'
from twisted.internet import reactor, protocol
from twisted.python import log
import sys
print 'Imports done'
class PrgShell(protocol.Protocol):
data = ''
class PrgProto(protocol.ProcessProtocol):
def __init__(self, out):
print 'Prgproto instance made'
self.transportout = out.transport
self.out = out
def outReceived(self, data):
"""Called when process sends data. We send it on to transport, however if it's 'I want input', we need to activate input."""
print 'Sub said: '+data
if data == "input":
print 'Sub wants input'
self.transportout.write("input")
sleep(0.01)
self.transport(self.out.getWrit())
else:
self.transportout.write(data)
def getWrit(self):
print 'Proto gave input to prg'
data = self.data
self.data = ''
return data
def connectionMade(self):
global reactor
print 'Connected'
proto = self.PrgProto(self)
addr = "C:\\Documents and Settings\\papa\\My Documents\\Python\\Files\\Maze\\exe\\maze.exe"
reactor.spawnProcess(proto, addr)
print 'Procces spawned!'
def dataReceived(self, data):
print 'Data recived: '+data
self.data+=data
print 'About to do stuff'
factory = protocol.ServerFactory()
factory.protocol = PrgShell
#f = open("errors.txt", 'w')
#log.startLogging(f)
#print 'Logging started'
reactor.listenTCP(8000,factory)
print 'Runing'
reactor.run()
The program in question prints stuff first thing. When I connect to it, via raw sockets, it doesn't send anything. Here's the output:
Preall test works!
Imports done
About to do stuff
Runing (connect)
Connected
Prgproto instance made
Procces spawned!
Am I missing anything?
Thanks in advance.

Replace reactor.spawnProcess(proto, addr) with reactor.spawnProcess(proto, addr, ['maze'], {}).
Past experience has show that if you don't pass exe name as the first argument then nothing useful happens. However I have yet to find a reasonable explanation for why this happens.
Also you don't need global reactor. When you import the reactor you add it to the top level script namespace. This means that all functions and class in the same file can use it without declaring a global or importing again.
Also, you should not be using sleep(0.01) because:
Its not a builtin function. You need to import it from the time module.
Twisted is a asynchronous framework where function should avoid blocking at all costs, and time.sleep() (link) by its definition is a blocking call.
You should instead use reactor.callLater() link where you provide it will a callback and a time period. This will let twisted handle other things (like a new connection) while you wait.
Finally, you code at the moment would require the user to enter input before the program asks for any. This is because getWrit just sends stuff already in the buffer rather than asking the user. This means that if the user hasn't sent any data before getWrit is called then it will just return an empty string.
It would be a better idea if you used a deferred. Then what you would do is call getWrit which would immanently return a deferred and clear the data buffer. Then in dataReceived you would append data to the buffer until you got a newline character (\n). At which point you would call the deferred set up in getWrit.
Something like this:
print 'Preall test works!'
from twisted.internet import reactor, protocol, defer
from twisted.python import log
import sys
print 'Imports done'
class PrgShell(protocol.Protocol):
data = ''
class PrgProto(protocol.ProcessProtocol):
def __init__(self, out):
print 'Prgproto instance made'
self.transportout = out.transport
self.out = out
def outReceived(self, data):
"""Called when process sends data. We send it on to transport, however if it's 'I want input', we need to activate input."""
print 'Sub said: '+data
if data == "input":
print 'Sub wants input'
self.transportout.write("input")
d = self.out.getWrit() # getWrit returns a deferred. We store it in d to make the code more readable
d.addCallback(self.sendInput) # Here we add self.sendInput to the callback chain.
# This way self.sendInput gets called with the user input.
else:
self.transportout.write(data)
def sendInput(self, data):
self.transport.write(data)
def getWrit(self):
print 'Proto gave input to prg'
self.deferred = defer.deferred()
self.data = ''
return self.deferred
def connectionMade(self):
print 'Connected'
proto = self.PrgProto(self)
addr = "C:\\Documents and Settings\\papa\\My Documents\\Python\\Files\\Maze\\exe\\maze.exe"
reactor.spawnProcess(proto, addr, ['maze'], {})
print 'Procces spawned!'
def dataReceived(self, data):
print 'Data recived: '+data
self.data+=data
if self.data.endswith('\n'):
if self.deferred:
# We got a newline character, and there is a deferred to call, so lets call it
d, self.deferred = self.deferred, None # This will set self.deferred to none to stop mistakes later
d.callback(self.data) # Call the deferred with data. This will send the data to sendInput above.
self.data = '' # Clear the buffer
print 'About to do stuff'
factory = protocol.ServerFactory()
factory.protocol = PrgShell
#f = open("errors.txt", 'w')
#log.startLogging(f)
#print 'Logging started'
reactor.listenTCP(8000,factory)
print 'Runing'
reactor.run()

Twisted ignoring data sent from MUD Clients?

I have the following code (almost an exact copy of the Chat server example listed here:
import twisted.scripts.twistd
from twisted.protocols import basic
from twisted.internet import protocol, reactor
from twisted.application import service, internet
class MyChat(basic.LineReceiver):
def connectionMade(self):
print "Got new client!"
self.factory.clients.append(self)
def connectionLost(self, reason):
print "Lost a client!"
self.factory.clients.remove(self)
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.message(line)
def message(self, message):
self.transport.write(message + '\n')
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
if __name__ == "__main__":
print "Building reactor...."
reactor.listenTCP(50000, factory)
print "Running ractor...."
reactor.run()
else:
application = service.Application("chatserver")
internet.TCPServer(50000, factory).setServiceParent(application)
The server runs without error, and if I connect to it via Telnet, I can send data and the server prints to the console and relays it to all clients (as is expected). However, if I connect to it via a different tool (a MUD client), it never gets the data.
I have ensured that the client is sending the data (Traced the packets with Wireshark, and they're going across the wire), but the server either never receives it, or is choosing to ignore it for some reason.
I have tried this with two MUD clients, gmud, and JMC. If it is important, I am running Windows 7 x64.
Does anyone have any idea why this could be happening?
Thanks,
Mike
EDIT:
Thanks to the hints provided by Maiku Mori, I tried adding another method that was specified in the Twisted API Docs, dataReceived. Once this was added, the MUD clients worked perfectly, but Telnet is now sending every character as it's own set of data, instead of waiting for the user to press Enter.
Here's a snipped of the new code:
def dataReceived(self, data):
print "Dreceived", repr(data)
for c in self.factory.clients:
c.message(data)
# def lineReceived(self, line):
# print "received", repr(line)
# for c in self.factory.clients:
# c.message(line)
Has anyone experiences this before, and if so, how do you get around it? Ideally, I would like Telnet and MUD clients to work with this application.
Thanks again.

In case anyone stumbles across this question with similar problems, I'm leaving my findings as the accepted answer so that people don't have to hunt the way I did.
I fixed the issue by changing the delimiter value from in my Twisted protocol from "\r\n" (default), to just "\n" (which is what my MUD clients send. This means that in Telnet, when you enter the string:
Hello, World
Your application will receive it as:
Hello, World\r
You may need to do data sanitation on the server side to keep things in order. My final code was as follows:
import twisted.scripts.twistd
from twisted.protocols import basic
from twisted.internet import protocol, reactor
from twisted.application import service, internet
class MyChat(basic.LineReceiver):
def __init__(self):
self.delimiter = "\n"
def connectionMade(self):
print "Got new client!"
self.factory.clients.append(self)
def connectionLost(self, reason):
print "Lost a client!"
self.factory.clients.remove(self)
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.message(line)
def message(self, message):
self.transport.write(message + '\n')
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
if __name__ == "__main__":
print "Building reactor...."
reactor.listenTCP(50000, factory)
print "Running ractor...."
reactor.run()
else:
application = service.Application("chatserver")
internet.TCPServer(50000, factory).setServiceParent(application)
Thanks for all the help.

Are you sure that the MUD clients send line ending chars after each line? The lineReceived will only be called after line ending char has been sent.
EDIT:
Here I found API docs for LineReceiver. You could play around with dataReceived method to see if you are actually getting any kind of data. If I recall you can use it just like lineReceived.