I am making an iPhone application that communicates to a Twisted socket and it works great when I have one message to send. However, my issue is I need to send many different bits of information to the app. Here is my code.
if numrows == 1:
#Did login
msg = "%s: Login Credentials Success" % _UDID
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
for result in results:
for i in range(1, 6):
msg = "%s:L%d;%s" % (_UDID, i, result[i])
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
else:
msg = "%s: Login Credentials Failed" % _UDID
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
cursor.close()
database.close()
#print msg
#for c in self.factory.clients:
#c.message(msg)
def message(self, message):
self.transport.write(message)
Say I were to send just the first msg, and every other msg didn't exist along with the print and for methods below each msg, the message Login Credentials Success would be sent to the app. But if put in the rest like how you are seeing it, nothing goes though because it sends it all at once, even with putting a time.sleep in the code.
The app checks the response every .05 seconds or less. Even though that the login credentials is on the top, it doesn't go through because there is more info that is being sent afterwards, but without all the info after the credentials message, it would go through.
I am desperate in finding the answer to this. I've tried everything I can think of. The app is not the issue, it's the Python.
Thanks.
At a risk of offending by contradicting, you may want to reexamine the claim that your app is not the problem. It sounds like you are expecting to have complete control over the content of each outgoing TCP packet, and that your app is depending on packet boundaries to determine the message boundaries. This isn't a very good approach to networking in general; some network intermediaries may split up (fragment) or even combine packets along the way, which would confuse your app. Remember that TCP is not a packet protocol: it is a stream protocol, so you're only really guaranteed that the octets you sent will be received in order, if at all.
A common approach to dealing with messages of varying size is to prefix each message with an X-bit big-endian size field stating how large the following message is. The receiving end of the communication reads X bits first, then reads 'size' octets after that to get the full message (blocking until that point, and leaving any additional information in the buffer for the next message handler to get).
I wouldn't mind helping more with your Twisted code, but it may very well be that it's already written properly. At least, I recommend not depending on trying to make Twisted flush network write buffers immediately. While it may help make your code appear to work right, right now, it will really just be hiding problems that will surface later.
Your issue appears to be that Twisted buffers the data you write to it.
Apparently, there is no easy way to force the data to be sent on its own without refactoring a great deal of your code. See Twisted transport.write and Using Twisted's twisted.web classes, how do I flush my outgoing buffers?.
Without knowing what your code looks like before the snipped you pasted and, according to the accepted answer of the last link:
define the wait function just as it is in the accepted answer
make sure that your class inherits from http.Request
decorate your method message with defer.inlineCallbacks
add yield wait(N) (for a value of N that you have to test and determine) after calls to write on the message method
I don't have enough experience with Twisted to know which of those steps are needed and which are just curft from the code of the original answer that doesn't apply to your code.
It may be possible (and easier) though, to re-write the iPhone application to accept the kind of messages that get sent when they include multiple writes in a single message.
Related
i have tried to receive data form my connection by this code on python 2.7 :
server = socket(AF_INET, SOCK_STREAM)
server.bind(('0.0.0.0', 21))
server.listen(1)
client , addr = server.accept()
data = client.recv(2048)
When i have to print or send to another connection my data it's working , however i want to add those lines :
if(data == "/disconnect") :
<disconnect blha blha... you know >
else :
<print the data and send it back blha blha... >
( i have checked without that if statement and " disconnect blha blha .. " works nicely )
it's just pass the code so when my client requests to disconnect the request is sent to the server as the "original" messages .. (the sever don't kick him)
what should i do ? Thanks !
You have two problems, and you need to fix both of them.
First, a TCP socket is just a stream of bytes. When you do a recv, you're not going to get exactly one message sent by send from the other side, you're going to get whatever's in the buffer at the moment—which could be two messages, or half a message, or anything else. When you're just testing with localhost connections on a computer that isn't heavily loaded, on many platforms, it will do what you're hoping for >99% of the time—but that just makes the problem hard to debug, it doesn't fix it. And as soon as you try to access the same code over the internet, it'll start failing most of the time instead of rarely.
Fortunately, the client appears to be sending messages as text, without any embedded newlines, with a \r\n Windows-style end-of-line between each message. This is a perfectly good protocol; you just have to write the code to handle that protocol on the receive side.
The second problem is that, even if you happen to get exactly one message send, that message includes the \r\n end-of-line. And '/disconnect\r\n' == '/disconnect' is of course going to be false. So, as part of your protocol handler, you need to strip off the newlines.
As it happens, you could solve both problems by using the makefile method to give you a file object that you can iterate, or readline on, etc., just like you do with a file that you open from disk, which you probably already know how to handle.
But it's worth learning how to do this stuff, so I'll show you how to do it manually. The key is that you keep a buffer around, add each recv onto that buffer, and then split it into lines, put any remainder back on the buffer, and process each line as a message. There are more elegant/concise ways to write this, but let's keep it dead simple:
buf = ''
while True:
data = client.recv(2048)
buf += data
lines = buf.split('\r\n')
buf = lines.pop()
for line in lines:
# line is now a single message, with the newline stripped
if line == "/disconnect":
# do disconnect stuff
else:
# do normal message stuff
That's all you need to get the basics working. But in a real server, you also need some code to handle two other conditions—because clients don't always shut down cleanly. For example, if a client gets disconnected from the internet before it can send a /disconnect message, you don't want to keep spinning and reading nothing forever, you want to treat it as a disconnect.
if not data: means the client has done a clean (at the TCP level) shutdown. So, you need to disconnect and break out of the receive loop.
Depending on your design, it may be legal to shutdown only the send side and wait for a final reply from the server, so you want to make sure you've finished sending whatever you have. (This is common in many internet protocols.)
It may even be legal to not send a final newline before shutting down; if you want to support this, you should check if buf: and if so, treat buf as one last command. (This is not common in many protocol—but is a common bug in clients, so, e.g., many web servers will handle it.)
try:/except Exception as e: will catch all kinds of errors. These errors mean the socket is no longer usable (or that there's a serious error in your code, of course), so you want to handle this by throwing away the connection and breaking out of the receive loop, without first sending any final response or reading any final message.
It's almost always worth logging that e in some way (maybe just print 'Error from', addr, repr(e)), so if you're getting unexpected exceptions you have something to debug.
The log output of my python program (using the builtin logging module, but occurs even when using simple prints) is partially messed up, as you can see in the following image. Note the first line, first word still being correct and then it gets mixed up:
I tried to visualize the situation where this happens:
Basically in my main thread/program I start a simple socketserver.TCPServer to listen for incoming messages. That server runs on its own thread (QtCore.QThread) so my program is not blocked. If some other application sends a message the request handler of the TCPServer will simply forward the message to the main thread using a QtCore.SIGNAL like:
self.emit(QtCore.SIGNAL('received(const QString)'), receivedMessage)
The program then does some parsing and computation with that message and logs those, thereby producing the gibberish seen above. At some point the logging returns back to working normally.
I am not sure if this is related to sockets or threading or both, but I guess it may be a common issue and therefore I am thankful for any hints why this occurs.
I think I have located the problem:
When the external application wants to send a message it will always create a new client socket, connect to the server, send the message and then close the client socket.
The sock.close() does not seem to close immediately, the docs say I should call sock.shutdown(how) first, but unfortunately this did not help as well. I can use a small time.sleep(0.5) after the close to fix the logging issue, but instead I did something like this:
def ensure_closed(self):
while True:
try:
self.sock.recv(1024)
except:
break
def close_connection(self):
self.sock.close()
self.ensure_closed()
# Continue with other stuff.
# Now the logging behaves normally.
There might be better ways to do it.
Here's a simple question and I'm surprised I haven't come across a similar one already.
I would like two processes to send strings (messages) to each other with send() and receive() functions. Here's a basic example:
# Process 1
# ... deal with sockets, connect to process 2 ...
msg = 'An arbitrarily long string\nMaybe with line breaks'
conn.send(msg)
msg = conn.receive()
if process1(msg):
conn.send('ok')
else:
conn.send('nok')
and
# Process 2
# ... deal with sockets, connect to process 1 ...
msg = conn.receive()
conn.send(process2(msg))
msg = conn.receive()
if msg == 'ok':
print('Success')
elif msg == 'nok':
print('Failure')
else:
print('Protocol error')
I know it is quite easy with bare stream sockets, but that's still cumbersome and error prone (do several conn.recv() inside a loop and check for size, like HTTP or end of stream marker, like SMTP, etc).
By the way, it doesn't necessarily need to use sockets, as long as messages of any size can be reliably carried through the network in an efficient manner.
Am I doing something wrong? Isn't there a simple library (Twisted AMP doesn't look simple) doing exactly that? I've been searching the Internet for a few hours without success :)
You can use ZeroMQ, there is an excellent Python binding called pyzmq.
It is a library for writing all kinds of distributed software, based on the
concept of message queues.
The project got a lot of hype lately, and you will find numerous examples and
tutorials on the web.
I tried using a socket for 2 sends. The first one succeeds and the next one does not.
From the http://docs.python.org/howto/sockets.html
it would appear that multiple sends should be allowed. For Better or worse, I don't really need to read from the socket.
I have used twisted, but for the present purpose, I would like to stick to a socket, if I can help it(partly because I am using it within an application already using twisted to communicate.. this is a seperate connection).
"When the connect completes, the socket s can be used to send in a request for the text of the page. The same socket will read the reply, and then be destroyed. That’s right, destroyed. Client sockets are normally only used for one exchange (or a small set of sequential exchanges)."
return value for the send that succeeds = 35
return value for the send that FAILS = 32
code with some minor editing to remove any business logic.
self._commandSock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def sendPrereqs(self,id,prereqs):
self._commandSock.connect(self._commandConnection)
#parse prereqs
temp = prereqs.split(',')
for pair in temp:
tup = pair.partition(':')
try:
command = 'some command'
logging.info('sending command: ' + command)
ret = self._commandSock.send(command)
if ret == None:
logging.info('send called successfully: ' + command)
else:
logging.info('socket returned non-None: ' + str(ret))
except:
print 'Unexpected Exception ', sys.exc_info()[0]()
print sys.exc_info()
#logging.info('Unexpected Exception '+ str(sys.exc_info()[0]()))
#logging.info(' ' + str(sys.exc_info()))
self._commandSock.close()`
return value for the send that succeeds = 35 return value for the send that FAILS = 32
Documentation says that successful send should return None.
No it doesn't. Documentation says:
Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data. For further information on this concept, consult the Socket Programming HOWTO.
You still haven't explained what you mean by "FAILS". The send call is returning successfully, and it's almost certainly placed 32 bytes into the socket write buffer.
If the only reason you think it's failing is that it returns the correct value, then the answer is obviously that it's not failing.
If something else is going wrong, there are all kinds of things that could be wrong at a higher level. One likely one is this: The server (especially if it was coded by someone who doesn't understand sockets well) is coded to expect one recv() of 35 bytes, and one recv() of 32 bytes. But it's actually getting a single recv() of 67 bytes, and waiting forever for the second, which never comes. There is no rule guaranteeing that each send() on a stream (TCP) socket corresponds to one recv() on the other side. You need to create some kind of stream protocol that demarcates the separate messages.
Meanwhile, the quote you're referring to is irrelevant. It's describing how client sockets are used by simple web browsers: They make a connection, do one send, receive all the data, then destroy the connection. I can see why it misled you, and you may want to file a documentation bug to get it improved. But for now, just ignore it.
If you want to make absolutely sure that the client is sending the right data, and the problem is in the server, there are two easy ways to do that:
Use netcat as a trivial substitute server (e.g., nc -kl 6000, replacing the "6000" with the actual port) and making sure it logs what you think the server should be seeing.
Use Wireshark to watch the connection between the client and server.
Once you've verified that the problem is on the server side, you need to debug the server. If you need help with that, that's probably best done in a new question, where you can post the server code instead of the client, and explain (with a link here) that you're sure the client is sending the right information.
The documentation is only referring to a common scenario. You can call send, sendall, and sendto on all sockets as often as you want - as long as the socket is not closed.
Note that these methods return the number of bytes sent, 32 and 35 simply mean you sent 32 bytes the first and 35 bytes the second time.
The fact that socket.send returns without an exception means that the data got handed to the operating system, but not that it actually reached the endpoint (or has been read correctly by an application there).
If I set up a pipeline which distributes load across a cluster, I would like to log where the messages get sent. This is what I have in mind (python):
import zmq
context = zmq.Context()
socket = context.socket(zmq.DOWNSTREAM)
socket.connect("tcp://127.0.0.1:5000")
socket.connect("tcp://127.0.0.1:6000")
msg = "Hello World\0"
connection_string = socket.send(msg)
# should print "Sent message to tcp://127.0.0.1:5000"
print "Sent message to", connection_string
But I cant find anything that talks about this. Any help at all is appreciated.
It think you'll want to use the X(REP|REQ) sockets somewhere in the topology for this. You might want to checkout the monitered queue device:
http://github.com/zeromq/pyzmq/blob/master/zmq/devices.pyx
And for general info have a look at the diagram and explanations of the patterns here too:
http://ipython.scipy.org/doc/nightly/html/development/parallel_connections.html
Also note that the DOWNSTREAM socket type is now called PUSH (and the corresponding UPSTREAM has changed to PULL).
Perhaps you could include the address of the recipient within the reply and log it then?
My 0.02 cents...
Well i believe since ZMQ routes in fair queue manner, that would be round robin. Hence messages would be sent to recepients in order. However, writing code under that assumption would not be the right thing since the underlying routing logic might change in future versions.
For PUSH/PULL, i am not sure, however for REQ/REP, i guess the REP side can send the reply back in an envolope with the first field as its address. That way the REQ can read twice on the socket and get the responder address as well as the data.