Develop Reference

Python Twisted: What Is Received with DataReceived and What Should I Probably Use Instead

I'm tooling around with a text-based adventure game in Python with Twisted and I think I've fundamentally misunderstood what comes back with Twisted's dataReceived function. I've got the following as part of my overall code;
def process(self, data):
print "Got command {}, {}".format(data, len(data))
if data == "test":
print "DEBUG: got the test command"
from twisted.internet import reactor, protocol
class MudLoop(protocol.Protocol):
def connectionMade(self):
login(self)
def dataReceived(self, data):
process(self, data)
factory = protocol.ServerFactory()
factory.protocol = MudLoop
reactor.listenTCP(12000,factory)
reactor.run()
No string sent back from dataReceived will trigger the process function's test for it and do something else, e.g. if a connected client enters 'test', no console output is received as per the DEBUG instruction saying it got the test command. I've got a couple of theories as to why, the first is that what's being passed isn't a string, the second is that while it's a string it's got invisible characters (carriage return and new line?) appended to it that are making it fail the test for what was entered - this seems supported by my test for the process function being called adding the length of the command in the console on a new line.
Which of these are true? Or is it something else? And in any case should I be testing the string returned from dataReceived in this way? Researching before I posted seems to suggest that other people haven't had this problem but do run into other problems later with strings receiving in an indeterminate order from multiple clients. If I solve this is my approach wrong and I'm just going onto the next problem as a result?

This is a Twisted FAQ: Why is dataReceived called with only part of the data I called transport.write with?.

Detect the end of a raw data

Twisted has two data reception modes: a Line Mode and a Raw Mode, and we can switch between them using setRawMode() and setLineMode() functions.
the line mode detects and end of line and then calls the lineReceived() function.
From Twisted doc:
def rawDataReceived(self, data):
Override this for when raw data is received.
How can Twisted detect the end of a raw data and then call rawDataReceived() ?
EDIT:
I'll add this to to complete my question.
I'm using this Qt function to send data to the Twisted server
qint64 QIODevice::write(const QByteArray & byteArray)
I thought that using write() two times means that the Twisted server will trigger the rawDataReceived() functions two times too.
write( "raw1" );
write( "raw2" );
but data are received in one time.

You asked:
How can Twisted detect the end of a raw data and then call rawDataReceived() ?
In short, when you turn on raw your asking Twisted not to detect.
... but let me explain
When you talk about 'detecting the end of data' inside of a connection (I.E. if your not closing the connection at the end of data), your talking about an idea that is normally referred to as framing.
Framing is one of the primary issues you have to keep in mind when your doing application level networking programming, because most (networking) protocols don't guarantee data framing to the application.
Confusingly many networking protocols (of which TCP is one of the most notorious) will often but not always present data to the receiver in the same way as it is transmitted (I.E. As though it had framing, each write will cause one read to happen - but only in cases of slow-use and low-load). Because of this maybe-it-will-work-maybe-it-won't behavior the best practice is to always explicitly add/build-in some sort of framing.
The most common method to add application-level framing in TCP/Serial/Keyboard style interfaces is to use line-breaks as end-of-frame makers, which is what LineMode is about.
Turning on raw mode in Twisted is like saying 'I want to write my own framing', but I doubt thats really what your after.
Instead you probably want to look at some of the other helper protocols (netstring, prefixed-message-length) that Twisted offers that will do binary framing for you (also see SO: Fragmented data in Twisted dataRecivied by Twisted's author Glyph)

Twisted does not detect the end of the raw data. It just calls rawDataReceived as it receive data.
Following is relevant part from Twisted code. (protocols/basic.py)
def dataReceived(self, data):
"""
Protocol.dataReceived.
Translates bytes into lines, and calls lineReceived (or
rawDataReceived, depending on mode.)
"""
if self._busyReceiving:
self._buffer += data
return
try:
self._busyReceiving = True
self._buffer += data
while self._buffer and not self.paused:
if self.line_mode:
....
else:
data = self._buffer
self._buffer = b''
why = self.rawDataReceived(data) # <--------
if why:
return why
finally:
self._busyReceiving = False

How to Covert to dictionary in Python

I am working on a large scale embedded system built using Python and we are using ZeroMQ to make everything modular. I have sensor data being sent across a ZeroMQ serial port in the form of the python Dictionary as shown here:
accel_com.publish_message({"ACL_X": ACL_1_X_val})
Where accel_com is a Communicator class we built that wraps the ZeroMQ logic that publishes messages across a port. Here you can see we are sending Dictionaries across.
However, on the other side of the communication port, I have another module that grabs this data using this code:
accel_msg = san.get_last_message("sensor/accelerometer")
accel.ax = accel_msg.get('ACL_X')
accel.ay = accel_msg.get('ACL_Y')
accel.az = accel_msg.get('ACL_Z')
The problem is when I try to treat accel_msg as a Python Dictionary, I get an Error:
'NoneType' object does not have a method 'get()'.
So my guess is the dictionary is not going across the wire correctly. I am not very familiar with Python so I am not sure how to solve this problem.

Expanding on #JoranBeasley's comment:
accel_msg is sometimes None, such as while it's waiting for a message. The solution is to skip over None messages
while True: # waiting indefinitely for messages
accel_msg = san.get_last_message("sensor/accelerometer")
if accel_msg: # or more explicitly, if accel_msg is not None:
accel.ax = accel_msg.get('ACL_X')
accel.ay = accel_msg.get('ACL_Y')
accel.az = accel_msg.get('ACL_Z')
break # if you only want one message. otherwise remove this
else:
print accel_msg # which is almost certainly None

Non-blocking socket in Python?

Is it me, or can I not find a good tutorial on non-blocking sockets in python?
I'm not sure how to exactly work the .recv and the .send in it. According to the python docs, (my understanding of it, at least) the recv'ed or send'ed data might be only partial data. So does that mean I have to somehow concatenate the data while recv and make sure all data sends through in send. If so, how? An example would be much appreciated.

It doesn't really matter if your socket is in non-blocking mode or not, recv/send work pretty much the same; the only difference is that non-blocking socket throws 'Resource temporarily unavailable' error instead of waiting for data/socket.
recv method returns numbers of bytes received, which is told to be less or equal to the passed bufsize. If you want to receive exactly size bytes, you should do something similar to the following code:
def recvall(sock, size):
data = ''
while len(data) < size:
d = sock.recv(size - len(data))
if not d:
# Connection closed by remote host, do what best for you
return None
data += d
return data
This is important to remember, that in blocking mode you have to do exactly the same. (The number of bytes passed to application layer is for example limited by recv buffer size in the OS.)
send method returns number of bytes sent, which is told to be less or equal to the length of passed string. If you want to ensure the whole message was sent, you should do something similar to the following code:
def sendall(sock, data):
while data:
sent = sock.send(data)
data = data[sent:]
You can use sock.sendall directly, but (according to the documentation) on error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent.
The sockets in Python follow the BSD socket API and behave in the similar way to c-style sockets (the difference is, for example, they throw exception instead of returning error code). You should be happy with any socket tutorial on the web and manpages.

Keep bytes you want to send in a buffer. (A list of byte-strings would be best, since you don't have to concatenate them.) Use the fcntl.fcntl function to set the socket in non-blocking mode:
import fcntl, os
fcntl.fcntl(mysocket, fcntl.F_SETFL, os.O_NONBLOCK)
Then select.select will tell you when it is OK to read and write to the socket. (Writing when it is not OK will give you the EAGAIN error in non-blocking mode.) When you write, check the return value to see how many bytes were actually written. Eliminate that many bytes from your buffer. If you use the list-of-strings approach, you only need to try writing the first string each time.
If you read the empty string, your socket has closed.

How can I create a non-http proxy with Twisted

How can I create a non-http proxy with Twisted. Instead I would like to do it for the Terraria protocol which is made entirely of binary data. I see that they have a built-in proxy for HTTP connections, but this application needs to act more like an entry point which is forwarded to a set server (almost like a BNC on IRC).
I can't figure out how to read the data off of one connection and send it to the other connection.
I have already tried using a socket for this task, but the blocking recv and send methods do not work well as two connections need to be live at the same time.

There are several different ways to create proxies in Twisted. The basic technique is built on peering, by taking two different protocols, on two different ports, and somehow gluing them together so that they can exchange data with each other.
The simplest proxy is a port-forwarder. Twisted ships with a port-forwarder implementation, see http://twistedmatrix.com/documents/current/api/twisted.protocols.portforward.html for the (underdocumented) classes ProxyClient and ProxyServer, although the actual source at http://twistedmatrix.com/trac/browser/tags/releases/twisted-11.0.0/twisted/protocols/portforward.py might be more useful to read through. From there, we can see the basic technique of proxying in Twisted:
def dataReceived(self, data):
self.peer.transport.write(data)
When a proxying protocol receives data, it puts it out to the peer on the other side. That's it! Quite simple. Of course, you'll usually need some extra setup... Let's look at a couple of proxies I've written before.
This is a proxy for Darklight, a little peer-to-peer system I wrote. It is talking to a backend server, and it wants to only proxy data if the data doesn't match a predefined header. You can see that it uses ProxyClientFactory and endpoints (fancy ClientCreator, basically) to start proxying, and when it receives data, it has an opportunity to examine it before continuing, either to keep proxying or to switch protocols.
class DarkServerProtocol(Protocol):
"""
Shim protocol for servers.
"""
peer = None
buf = ""
def __init__(self, endpoint):
self.endpoint = endpoint
print "Protocol created..."
def challenge(self, challenge):
log.msg("Challenged: %s" % challenge)
# ...omitted for brevity...
return is_valid(challenge)
def connectionMade(self):
pcf = ProxyClientFactory()
pcf.setServer(self)
d = self.endpoint.connect(pcf)
d.addErrback(lambda failure: self.transport.loseConnection())
self.transport.pauseProducing()
def setPeer(self, peer):
# Our proxy passthrough has succeeded, so we will be seeing data
# coming through shortly.
log.msg("Established passthrough")
self.peer = peer
def dataReceived(self, data):
self.buf += data
# Examine whether we have received a challenge.
if self.challenge(self.buf):
# Excellent; change protocol.
p = DarkAMP()
p.factory = self.factory
self.transport.protocol = p
p.makeConnection(self.transport)
elif self.peer:
# Well, go ahead and send it through.
self.peer.transport.write(data)
This is a rather complex chunk of code which takes two StatefulProtocols and glues them together rather forcefully. This is from a VNC proxy (https://code.osuosl.org/projects/twisted-vncauthproxy to be precise), which needs its protocols to do a lot of pre-authentication stuff before they are ready to be joined. This kind of proxy is the worst case; for speed, you don't want to interact with the data going over the proxy, but you need to do some setup beforehand.
def start_proxying(result):
"""
Callback to start proxies.
"""
log.msg("Starting proxy")
client_result, server_result = result
success = True
client_success, client = client_result
server_success, server = server_result
if not client_success:
success = False
log.err("Had issues on client side...")
log.err(client)
if not server_success:
success = False
log.err("Had issues on server side...")
log.err(server)
if not success:
log.err("Had issues connecting, disconnecting both sides")
if not isinstance(client, Failure):
client.transport.loseConnection()
if not isinstance(server, Failure):
server.transport.loseConnection()
return
server.dataReceived = client.transport.write
client.dataReceived = server.transport.write
# Replay last bits of stuff in the pipe, if there's anything left.
data = server._sful_data[1].read()
if data:
client.transport.write(data)
data = client._sful_data[1].read()
if data:
server.transport.write(data)
server.transport.resumeProducing()
client.transport.resumeProducing()
log.msg("Proxying started!")
So, now that I've explained that...
I also wrote Bravo. As in, http://www.bravoserver.org/. So I know a bit about Minecraft, and thus about Terraria. You will probably want to parse the packets coming through your proxy on both sides, so your actual proxying might start out looking like this, but it will quickly evolve as you begin to understand the data you're proxying. Hopefully this is enough to get you started!