Develop Reference

python asyncore client example

I am trying to implement as asynchronous TCP Client - Server model in python. It's my first example of using asyncore module and i need some explanation if anyone can provide me.
I have the following requirements :
[Client]
Initiate a client instance - connect to server if server is running else wait for the server to come up.
I need to receive / transmit data from server.
Notify me whenever data is received on the socket.
I tried running a sample example from net but had some doubts :
import asyncore
import logging
import socket
from cStringIO import StringIO
import urlparse
class Client(asyncore.dispatcher):
def __init__(self,host):
self.logger = logging.getLogger()
self.write_buffer = ""
self.read_buffer = StringIO()
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
address = (host, 80)
self.logger.debug('connecting to %s', address)
self.connect(address)
def handle_connect(self):
self.logger.debug('handle_connect()')
def handle_close(self):
self.logger.debug('handle_close()')
self.close()
def writable(self):
is_writable = (len(self.write_buffer) > 0)
if is_writable:
self.logger.debug('writable() -> %s', is_writable)
return is_writable
def readable(self):
self.logger.debug('readable() -> True')
return True
def handle_write(self):
sent = self.send(self.write_buffer)
self.logger.debug('handle_write() -> "%s"', self.write_buffer[:sent])
self.write_buffer = self.write_buffer[sent:]
def handle_read(self):
data = self.recv(8192)
self.logger.debug('handle_read() -> %d bytes', len(data))
self.read_buffer.write(data)
if __name__ == '__main__':
logging.basicConfig(level=logging.DEBUG,
format='%(name)s: %(message)s',
)
clients = Client("127.0.0.1")
logging.debug('LOOP STARTING')
asyncore.loop()
logging.debug('LOOP DONE')
How does dispatcher class notifies when data is available to read
from socket. Is handle_read called in that scenario ?
Is it a busy polling mechanism ? will it eat my whole cpu even if socket is
sitting idle ?
Above example does not wait for server to come up. How can do this ?
how can write data to socket from client ?

I know this is old, but maybe may be useful to someone:
http://effbot.org/librarybook/asyncore.htm

Python sockets; sending from client receiving on server

I am trying to send messages on TCP/IP all on host machine. This is working, although for some reason the socket needs to be re-instantiated for every new message on the client side only. For example here is a basic client that sends three separate messages:
import socket
host = '127.0.0.1'
class Client:
def __init__(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self):
self.sock.connect((host,12347))
def send(self,message):
self.sock.sendall(message)
def close(self):
self.sock.close()
if __name__ == "__main__":
message1 = "I am message 1"
message2 = "I am message 2"
message3 = "I am message 3"
#exp = Client()
#exp.connect()
for i in range(0,3):
try:
exp = Client()
exp.connect()
if i == 0:
txt = message1
elif i == 1:
txt = message2
elif i == 2:
txt = message3
exp.send(txt)
exp.close()
print i
exp.send(txt)
except:
pass
and the server that receives:
#!/usr/bin/env python
import socket
class communication:
def __init__(self):
try:
host = '127.0.0.1'
self.Server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.Server.bind((host,12347))
self.Server.listen(1)
finally:
print "setup finished"
def recieve(self):
(connection, client_address) = self.Server.accept()
data = connection.recv(128)
return data
def close(self):
self.server.close()
if __name__ == "__main__":
exp = communication()
while True:
try:
(connection,client_address) = exp.Server.accept()
message = connection.recv(128)
finally:
print message
if message == "I am message 3":
exp.close()
You see how I re-call the Client class in each iteration of the for loop. This seems to be necessary for sending messages 2 and 3. If the socket is instantiated only once at the start of the main code along with the connect() function, then the server hangs on the recv() after the first message has been sent.
I can't understand why this is happening and the socket only needs to be setup once on the server side. I am doing something wrong, or is this normal?
Thanks!

It's even worse than you think. Take a look at your server code. exp.Server.accept() accepts a connection from the client, but connection.receive() ignores that connection completely and does a second self.Server.accept(). You ignore half of your connections!
Next, your server only does a single receive.... Even if you tried to send more messages on the connection, the server would ignore them.
But you can't just add a recv loop. Your client and server need some way to mark message boundaries so the server knows how to pull them out. Some text based systems use a new line. Others send a message size or fixed size header that the server can read. HTTP for example uses a combination of new lines and data count.
If you want to learn sockets from the ground up just know that they are complicated and you'll need to study. There are lots of ways to build a server and you'll need to understand the trade-offs. Otherwise, there are many frameworks from XMLRPC to zeromq that do some of the heavy lifting for you.

Writing simple SMTP server in python

I'm trying to write a simple smtp server program. I've written a simple smtp client (in C#) which sends an email. I've tested the program with smtp4dev. So far it all works fine.
I'd also like to write my own simple program which receives the email (instead of smtp4dev). I've tried a number of different code snippets (eg: Here) that I've found around the web but I can't seem to get them working.
I've also tried using twisted.
To start with I can see using TCPView that the port numbers in the code are not the ones being used.
I get the feeling that I'm missing something conceptual though and heading in the wrong direction.
EDIT
Here's the C# code in case you are interested
MailMessage mail = new MailMessage();
mail.Subject = "Your Subject";
mail.From = new MailAddress("test#test.com.au");
mail.To.Add("soslab#soslab.lab");
mail.Body = "Hello! your mail content goes here...";
mail.IsBodyHtml = true;
SmtpClient smtp = new SmtpClient("LOCALHOST", 26);
smtp.EnableSsl = false;
try
{
smtp.Send(mail);
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
here's the python code
import smtpd
import asyncore
class EmailServer(smtpd.SMTPServer):
def process_message(self, peer, mailfrom, rcpttos, data):
print 'a'
def run():
foo = EmailServer(('localhost', 26), None)
try:
asyncore.loop()
except KeyboardInterrupt:
pass
if __name__ == '__main__':
run()

For some reason this program runs fine when I run it from the command line
import smtpd
import asyncore
import winsound
class PYEmailServer(smtpd.SMTPServer):
def __init__(*args, **kwargs):
smtpd.SMTPServer.__init__(*args, **kwargs)
def process_message(self, peer, mailfrom, rcpttos, data):
winsound.Beep(2500, 1000)
def run():
foo = PYEmailServer(('localhost', 26), None)
try:
asyncore.loop()
except KeyboardInterrupt:
foo.close()
if __name__ == '__main__':
run()
It does not work when I run it from IDLE. (C# program just throws an exception like the service isnt there). I dont know why this would be, but I have my original problem working.

To test your smtp server you need to set the smtpclient object on another terminal
import smtplib
smtpclient=smtplib.SMTP('127.0.0.1',8001)
smtpClient.sendmail('sender#gmail.com','recivers#gmail.com','sadfsdf')

Howto connect two programs with Twisted

I have an echoserver in Twisted that needs to get input from an echoclient. The echocient is a GUI (Panda3D). The client just sends a short message when a button is clicked.
So I have messages to send at irregular times (only when button is clicked).
How can I have a permanent connection (the reactor.run() is already started at the beginning of the client program) and send messages.
I don't want to write a polling mechanism in EchoClient/connectionMade. I saw an example of gtk+ but cannot translate it to Panda. How to go about. The code hereunder is not working at all but gives you an idea of what I want (basically permanent connection and once in a while the user sends something when pressing a button).
from direct.showbase.ShowBase import ShowBase
from direct.gui.DirectButton import DirectButton
from panda3d.core import Vec3
from direct.task import Task
from twisted.internet import protocol, reactor, defer
from twisted.internet.task import LoopingCall
from twisted.spread import pb
FRAMERATE = 32
class LoginDialog:
def __init__(self, deferred):
self.deferredResult = deferred
class EchoClient(ShowBase, protocol.Protocol):
def __init__(self):
ShowBase.__init__(self)
self.echoer = echoer
self.button = DirectButton(pos = Vec3(.1,0,.1), text = "Send request",
scale = .1, pad = (.5, .5),
rolloverSound = None, clickSound = None,
command = self.Request)
def Request():
self.echoer.transport.write("Message from client")
def dataReceived(self, data):
print "Server said: ", data
cf = pb.PBClientFactory()
cf.getRootObject().addCallback(EchoClient)
reactor.connectTCP("localhost", 17000, cf)
LoopingCall(taskMgr.step).start(1 / FRAMERATE)
reactor.run()

If you make the client and the server part of the same process then you'll always (well, practically always) be able to pass information from one to the other.
If you make the client and the server different processes then there is probably no solution that satisfies your requirements.
You always have to do something to establish a connection between two processes and there's always a chance it might go away. Sorry.

have a look at multiprocessing: http://docs.python.org/2/library/multiprocessing.html
you can start and join different processes as well as message between them.

I found the answer. Now I have a program that sends a message to a server once a button is pushed. I used a ClientCreator to create an EchoClient. Like this:
self.clientcreate = protocol.ClientCreator(reactor, EchoClient)
self.clientcreate.connectTCP(host, port).addCallbacks(self.connectionMade,
self.connectionFailed)
Now when I connectTCP, and make a connection it calls self.ConnectionMade upon success.
Thereby it also gives the EchoClient to the function. So I can store it (self.client = echoclient), and use it when I want (could not achieve this via Factories).
For instance the button calls the function Request and there I can directly use the transport.write command, because I have the client now. So self.client.transport.write('whatever').
If people did not understand what I wanted before, have better ideas to get this done, I would like to see their comments (i want to learn).
The complete code (Client only):
from twisted.internet import protocol, reactor, defer
from twisted.internet.task import LoopingCall
from direct.showbase.ShowBase import ShowBase
from direct.gui.DirectButton import DirectButton
from panda3d.core import Vec3
from direct.task import Task
FRAMERATE = 32
class ButtonDialog(ShowBase):
def __init__(self):
ShowBase.__init__(self)
self.button = DirectButton(pos = Vec3(.1,0,.1), text = "Send request",
scale = .1, pad = (.5, .5),
rolloverSound = None, clickSound = None,
command = self.Request)
host = "localhost"
port = 17001
self.clientcreate = protocol.ClientCreator(reactor, EchoClient)
self.clientcreate.connectTCP(host, port).addCallbacks(self.connectionMade,
self.connectionFailed)
def connectionFailed(self, f):
print "Connection Failed:", f
reactor.stop()
def connectionMade(self, echoclient):
self.client = echoclient
def Request(self):
self.client.transport.write("Message from button")
class EchoClient(protocol.Protocol):
def dataReceived(self, data):
print "Server said: ", data
def connectionLost(self, err):
print "Connection is lost: " + str(err)
reactor.stop()
ButtonDialog()
LoopingCall(taskMgr.step).start(1 / FRAMERATE)
reactor.run()

twisted client / server communication woes

I'm trying to make a simple distributed job client/server system in Twisted. Basically the steps are:
Start up the JobServer with a few jobs and associated files
Start up JobClient instances, they connect to JobServer and ask for Jobs
Server gives JobClient job and sends serialized JSON over TCP
After perhaps a lot of computation, the JobClient sends back a result and waits for new job
Rinse and repeat
But I'm having trouble debugging my protocol on a local machine.
JobServer.py
from twisted.application import internet, service
from twisted.internet import reactor, protocol, defer
from twisted.protocols import basic
from twisted.protocols.basic import Int32StringReceiver
from twisted.web import client
import random
import json
import base64
from logger import JobLogger
class JobServerProtocol(Int32StringReceiver):
log = JobLogger("server.log")
def connectionMade(self):
self.log.write("Connected to client")
self.sendJob(None)
def stringReceived(self, msg):
self.log.write("Recieved job from client: %s" % msg)
self.sendJob(msg)
def sendJob(self, msg):
d = self.factory.getJob(msg)
def onError(err):
self.transport.write("Internal server error")
d.addErrback(onError)
def sendString(newjob_dict):
encoded_str = json.dumps(newjob_dict)
self.transport.write(encoded_str)
self.log.write("Sending job to client: %s" % encoded_str)
d.addCallback(sendString)
def lengthLimitExceeded(self, msg):
self.transport.loseConnection()
class JobServerFactory(protocol.ServerFactory):
protocol = JobServerProtocol
def __init__(self, jobs, files):
assert len(jobs) == len(files)
self.jobs = jobs
self.files = files
self.results = []
def getJob(self, msg):
# on startup the client will not have a message to send
if msg:
# recreate pickled msg
msg_dict = json.loads(msg)
self.results.append((msg_dict['result'], msg_dict['jidx']))
# if we're all done, let the client know
if len(self.jobs) == 0:
job = None
jidx = -1
encoded = ""
else:
# get new job for client to process
jidx = random.randint(0, len(self.jobs) - 1)
job = self.jobs[jidx]
del self.jobs[jidx]
# get file
with open(self.files[jidx], 'r') as f:
filecontents = f.read()
encoded = base64.b64encode(filecontents)
# create dict object to send to client
response_msg = {
"job" : job,
"index" : jidx,
"file" : encoded
}
return defer.succeed(response_msg)
# args for factory
files = ['test.txt', 'test.txt', 'test.txt']
jobs = ["4*4-5", "2**2-5", "2/9*2/3"]
application = service.Application('jobservice')
factory = JobServerFactory(jobs=jobs, files=files)
internet.TCPServer(12345, factory).setServiceParent(
service.IServiceCollection(application))
JobClient.py
from twisted.internet import reactor, protocol
from twisted.protocols.basic import Int32StringReceiver
import json
import time
from logger import JobLogger
class JobClientProtocol(Int32StringReceiver):
log = JobLogger("client.log")
def stringReceived(self, msg):
# unpack job from server
server_msg_dict = json.loads(msg)
job = server_msg_dict["job"]
index = server_msg_dict["index"]
filestring = server_msg_dict["file"]
if index == -1:
# we're done with all tasks
self.transport.loseConnection()
self.log.write("Recieved job %d from server with file '%s'" % (index, filestring))
# do something with file
# job from the server...
time.sleep(5)
result = { "a" : 1, "b" : 2, "c" : 3}
result_msg = { "result" : result, "jidx" : index }
self.log.write("Completed job %d from server with result '%s'" % (index, result))
# serialize and tell server
result_str = json.dumps(result_msg)
self.transport.write(encoded_str)
def lengthLimitExceeded(self, msg):
self.transport.loseConnection()
class JobClientFactory(protocol.ClientFactory):
def buildProtocol(self, addr):
p = JobClientProtocol()
p.factory = self
return p
reactor.connectTCP("127.0.0.1", 12345, JobClientFactory())
reactor.run()
logging.py
class JobLogger(object):
def __init__(self, filename):
self.log = open(filename, 'a')
def write(self, string):
self.log.write("%s\n" % string)
def close(self):
self.log.close()
Running, testing locally with only one client:
$ twistd -y JobServer.py -l ./jobserver.log --pidfile=./jobserver.pid
$ python JobClient.py
Problems I'm having:
The client and server .log files don't get written to reliably - sometimes not until after I kill the process.
The protocol gets stuck after the client connects and the server sends back a message. The message seemingly never gets to the client.
In general, I hope these protocols ensure that operations on either side can take any amount of time, but perhaps I didn't design that correctly.

The client and server .log files don't get written to reliably - sometimes not until after I kill the process.
If you want bytes to appear on disk in a timely manner, you may need to call flush on your file object.
The protocol gets stuck after the client connects and the server sends back a message. The message seemingly never gets to the client.
The server doesn't send int32 strings to the client: it calls transport.write directly. The client gets confused because these end up looking like extremely long int32 strings. For example, the first four bytes of "Internal server error" decode as the integer 1702129225 so if there is an error on the server and these bytes are sent to the client, the client will wait for roughly 2GB of data before proceeding.
Use Int32StringReceiver.sendString instead.