I tried using a socket for 2 sends. The first one succeeds and the next one does not.
From the http://docs.python.org/howto/sockets.html
it would appear that multiple sends should be allowed. For Better or worse, I don't really need to read from the socket.
I have used twisted, but for the present purpose, I would like to stick to a socket, if I can help it(partly because I am using it within an application already using twisted to communicate.. this is a seperate connection).
"When the connect completes, the socket s can be used to send in a request for the text of the page. The same socket will read the reply, and then be destroyed. That’s right, destroyed. Client sockets are normally only used for one exchange (or a small set of sequential exchanges)."
return value for the send that succeeds = 35
return value for the send that FAILS = 32
code with some minor editing to remove any business logic.
self._commandSock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def sendPrereqs(self,id,prereqs):
self._commandSock.connect(self._commandConnection)
#parse prereqs
temp = prereqs.split(',')
for pair in temp:
tup = pair.partition(':')
try:
command = 'some command'
logging.info('sending command: ' + command)
ret = self._commandSock.send(command)
if ret == None:
logging.info('send called successfully: ' + command)
else:
logging.info('socket returned non-None: ' + str(ret))
except:
print 'Unexpected Exception ', sys.exc_info()[0]()
print sys.exc_info()
#logging.info('Unexpected Exception '+ str(sys.exc_info()[0]()))
#logging.info(' ' + str(sys.exc_info()))
self._commandSock.close()`
return value for the send that succeeds = 35 return value for the send that FAILS = 32
Documentation says that successful send should return None.
No it doesn't. Documentation says:
Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data. For further information on this concept, consult the Socket Programming HOWTO.
You still haven't explained what you mean by "FAILS". The send call is returning successfully, and it's almost certainly placed 32 bytes into the socket write buffer.
If the only reason you think it's failing is that it returns the correct value, then the answer is obviously that it's not failing.
If something else is going wrong, there are all kinds of things that could be wrong at a higher level. One likely one is this: The server (especially if it was coded by someone who doesn't understand sockets well) is coded to expect one recv() of 35 bytes, and one recv() of 32 bytes. But it's actually getting a single recv() of 67 bytes, and waiting forever for the second, which never comes. There is no rule guaranteeing that each send() on a stream (TCP) socket corresponds to one recv() on the other side. You need to create some kind of stream protocol that demarcates the separate messages.
Meanwhile, the quote you're referring to is irrelevant. It's describing how client sockets are used by simple web browsers: They make a connection, do one send, receive all the data, then destroy the connection. I can see why it misled you, and you may want to file a documentation bug to get it improved. But for now, just ignore it.
If you want to make absolutely sure that the client is sending the right data, and the problem is in the server, there are two easy ways to do that:
Use netcat as a trivial substitute server (e.g., nc -kl 6000, replacing the "6000" with the actual port) and making sure it logs what you think the server should be seeing.
Use Wireshark to watch the connection between the client and server.
Once you've verified that the problem is on the server side, you need to debug the server. If you need help with that, that's probably best done in a new question, where you can post the server code instead of the client, and explain (with a link here) that you're sure the client is sending the right information.
The documentation is only referring to a common scenario. You can call send, sendall, and sendto on all sockets as often as you want - as long as the socket is not closed.
Note that these methods return the number of bytes sent, 32 and 35 simply mean you sent 32 bytes the first and 35 bytes the second time.
The fact that socket.send returns without an exception means that the data got handed to the operating system, but not that it actually reached the endpoint (or has been read correctly by an application there).
Related
i have tried to receive data form my connection by this code on python 2.7 :
server = socket(AF_INET, SOCK_STREAM)
server.bind(('0.0.0.0', 21))
server.listen(1)
client , addr = server.accept()
data = client.recv(2048)
When i have to print or send to another connection my data it's working , however i want to add those lines :
if(data == "/disconnect") :
<disconnect blha blha... you know >
else :
<print the data and send it back blha blha... >
( i have checked without that if statement and " disconnect blha blha .. " works nicely )
it's just pass the code so when my client requests to disconnect the request is sent to the server as the "original" messages .. (the sever don't kick him)
what should i do ? Thanks !
You have two problems, and you need to fix both of them.
First, a TCP socket is just a stream of bytes. When you do a recv, you're not going to get exactly one message sent by send from the other side, you're going to get whatever's in the buffer at the moment—which could be two messages, or half a message, or anything else. When you're just testing with localhost connections on a computer that isn't heavily loaded, on many platforms, it will do what you're hoping for >99% of the time—but that just makes the problem hard to debug, it doesn't fix it. And as soon as you try to access the same code over the internet, it'll start failing most of the time instead of rarely.
Fortunately, the client appears to be sending messages as text, without any embedded newlines, with a \r\n Windows-style end-of-line between each message. This is a perfectly good protocol; you just have to write the code to handle that protocol on the receive side.
The second problem is that, even if you happen to get exactly one message send, that message includes the \r\n end-of-line. And '/disconnect\r\n' == '/disconnect' is of course going to be false. So, as part of your protocol handler, you need to strip off the newlines.
As it happens, you could solve both problems by using the makefile method to give you a file object that you can iterate, or readline on, etc., just like you do with a file that you open from disk, which you probably already know how to handle.
But it's worth learning how to do this stuff, so I'll show you how to do it manually. The key is that you keep a buffer around, add each recv onto that buffer, and then split it into lines, put any remainder back on the buffer, and process each line as a message. There are more elegant/concise ways to write this, but let's keep it dead simple:
buf = ''
while True:
data = client.recv(2048)
buf += data
lines = buf.split('\r\n')
buf = lines.pop()
for line in lines:
# line is now a single message, with the newline stripped
if line == "/disconnect":
# do disconnect stuff
else:
# do normal message stuff
That's all you need to get the basics working. But in a real server, you also need some code to handle two other conditions—because clients don't always shut down cleanly. For example, if a client gets disconnected from the internet before it can send a /disconnect message, you don't want to keep spinning and reading nothing forever, you want to treat it as a disconnect.
if not data: means the client has done a clean (at the TCP level) shutdown. So, you need to disconnect and break out of the receive loop.
Depending on your design, it may be legal to shutdown only the send side and wait for a final reply from the server, so you want to make sure you've finished sending whatever you have. (This is common in many internet protocols.)
It may even be legal to not send a final newline before shutting down; if you want to support this, you should check if buf: and if so, treat buf as one last command. (This is not common in many protocol—but is a common bug in clients, so, e.g., many web servers will handle it.)
try:/except Exception as e: will catch all kinds of errors. These errors mean the socket is no longer usable (or that there's a serious error in your code, of course), so you want to handle this by throwing away the connection and breaking out of the receive loop, without first sending any final response or reading any final message.
It's almost always worth logging that e in some way (maybe just print 'Error from', addr, repr(e)), so if you're getting unexpected exceptions you have something to debug.
I use python socket to send data to server, and the code like:
When I close the server, it will send the data twice, and then, it will go to the "except" code. If I set the SEND_INTERVAL too long, it will be a disaster. So, how to get the error immediately when the server is closed or downtime?
Nothing happens immediatly over the network. That's one thing.
Secondly the underlying OS will detect broken connections (and Python gets that info in the form of an exception), but usually this takes time. And that's why you still send messages even though the connection is already dead. But since OS operates on network layer (as opposed to the application layer) then there's an issue: the connection may be dead but OS may never detect this. For example this will happen when the server is dead but behind alive proxy.
Thirdly the most reliable way to know that a server is alive is when it sends something back to the client. So you should always .recv() (with timeout) after .sendall() call and the server should always .sendall() after .recv() (the request-response pattern, even when the response is a simple "I received message"). If you can't modify the server side and (in worst case) if the server doesn't send anything back to the client then there's no reliable way to know this.
That's why you need some form of framing/correctness protocol. Simple .sendall() won't do.
I have a problem with receiving data from server to client. I have the following client-side function that attempts to receive data from the server. The data sent by the server using the socket.sendall (data) function is greater than buff_size so I need a loop to read all the data.
def receiveAll (sock):
data = ""
buff_size = 4096
while True:
part = sock.recv (buff_size)
data + = part
if part <buff_size:
break;
return data
The problem that occurs to me is that after the first iteration (read the first 4096mb), in the second the program is blocked waiting for the other data in part = sock.recv (buff_size). How do I have to do so that recv() can continue reading the other missing data? Thank you.
Your interpretation is wrong. Your code reads all the data that it get from the server. It just doesn't know that it should stop listening for incoming data. It doesn't know that the server sent everything it had.
First of all note that these lines
if part <buff_size:
break;
are very wrong. First of all you are comparing a string to int (in Python3.x that would throw an exception). But even if you meant if len(part) <buff_size: then this is still wrong. Because first of all there might be a lag in the middle of streaming and you will only read a piece smaller then buff_size. Your code will stop there.
Also if your server sends a content of the size being a multiple of buff_size then the if part will never be satisfied and it will hang on .recv() forever.
Side note: don't use semicolons ;. It's Python.
There are several solutions to your problem but none of them can be used correctly without modyfing the server side.
As a client you have to know when to stop reading. But the only way to know it is if the server does something special and you will understand it. This is called a communication protocol. You have to add a meaning to data you send/receive.
For example if you use HTTP, then a server sends this header Content-Length: 12345 before body so now as a client you know that you only need to read 12345 bytes (your buffer doesn't have to be as big, but with that info you will know how many times you have to loop before reading it all).
Some binary protocols may send the size of the content in first 2 or 4 bytes for example. This can be easily interpreted on the client side as well.
Easier solution is this: simply make server close the connection after he sends all the data. Then you will only need to add check if not part: break in your code.
I am making an iPhone application that communicates to a Twisted socket and it works great when I have one message to send. However, my issue is I need to send many different bits of information to the app. Here is my code.
if numrows == 1:
#Did login
msg = "%s: Login Credentials Success" % _UDID
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
for result in results:
for i in range(1, 6):
msg = "%s:L%d;%s" % (_UDID, i, result[i])
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
else:
msg = "%s: Login Credentials Failed" % _UDID
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
cursor.close()
database.close()
#print msg
#for c in self.factory.clients:
#c.message(msg)
def message(self, message):
self.transport.write(message)
Say I were to send just the first msg, and every other msg didn't exist along with the print and for methods below each msg, the message Login Credentials Success would be sent to the app. But if put in the rest like how you are seeing it, nothing goes though because it sends it all at once, even with putting a time.sleep in the code.
The app checks the response every .05 seconds or less. Even though that the login credentials is on the top, it doesn't go through because there is more info that is being sent afterwards, but without all the info after the credentials message, it would go through.
I am desperate in finding the answer to this. I've tried everything I can think of. The app is not the issue, it's the Python.
Thanks.
At a risk of offending by contradicting, you may want to reexamine the claim that your app is not the problem. It sounds like you are expecting to have complete control over the content of each outgoing TCP packet, and that your app is depending on packet boundaries to determine the message boundaries. This isn't a very good approach to networking in general; some network intermediaries may split up (fragment) or even combine packets along the way, which would confuse your app. Remember that TCP is not a packet protocol: it is a stream protocol, so you're only really guaranteed that the octets you sent will be received in order, if at all.
A common approach to dealing with messages of varying size is to prefix each message with an X-bit big-endian size field stating how large the following message is. The receiving end of the communication reads X bits first, then reads 'size' octets after that to get the full message (blocking until that point, and leaving any additional information in the buffer for the next message handler to get).
I wouldn't mind helping more with your Twisted code, but it may very well be that it's already written properly. At least, I recommend not depending on trying to make Twisted flush network write buffers immediately. While it may help make your code appear to work right, right now, it will really just be hiding problems that will surface later.
Your issue appears to be that Twisted buffers the data you write to it.
Apparently, there is no easy way to force the data to be sent on its own without refactoring a great deal of your code. See Twisted transport.write and Using Twisted's twisted.web classes, how do I flush my outgoing buffers?.
Without knowing what your code looks like before the snipped you pasted and, according to the accepted answer of the last link:
define the wait function just as it is in the accepted answer
make sure that your class inherits from http.Request
decorate your method message with defer.inlineCallbacks
add yield wait(N) (for a value of N that you have to test and determine) after calls to write on the message method
I don't have enough experience with Twisted to know which of those steps are needed and which are just curft from the code of the original answer that doesn't apply to your code.
It may be possible (and easier) though, to re-write the iPhone application to accept the kind of messages that get sent when they include multiple writes in a single message.
When I close the socket on one end of a connection, the other end gets an error the second time it sends data, but not the first time:
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("localhost", 12345))
server.listen(1)
client = socket.create_connection(("localhost",12345))
sock, addr = server.accept()
sock.close()
client.sendall("Hello World!") # no error
client.sendall("Goodbye World!") # error happens here
I've tried setting TCP_NODELAY, using send instead of sendall, checking the fileno(), I can't find any way to get the first send to throw an error or even to detect afterwards that it failed. EDIT: calling sock.shutdown before sock.close doesn't help. EDIT #2: even adding a time.sleep after closing and before writing doesn't matter. EDIT #3: checking the byte count returned by send doesn't help, since it always returns the number of bytes in the message.
So the only solution I can come up with if I want to detect errors is to follow each sendall with a client.sendall("") which will raise an error. But this seems hackish. I'm on a Linux 2.6.x so even if a solution only worked for that OS I'd be happy.
This is expected, and how the TCP/IP APIs are implemented (so it's similar in pretty much all languages and on all operating systems)
The short story is, you cannot do anything to guarantee that a send() call returns an error directly if that send() call somehow cannot deliver data to the other end. send/write calls just delivers the data to the TCP stack, and it's up to the TCP stack to deliver it when it can.
TCP is also just a transport protocol, if you need to know if your application "messages" have reached the other end, you need to implement that yourself(some form of ACK), as part of your application protocol - there's no other free lunch.
However - if you read() from a socket, you can get notified immediatly when an error occurs, or when the other end closed the socket - you usually need to do this in some form of multiplexing event loop (that is, using select/poll or some other IO multiplexing facility).
Just note that you cannot read() from a socket to learn whether the most recent send/write succeded, Here's a few cases as of why (but it's the cases one doesn't think about that always get you)
several write() calls got buffered up due to network congestion, or because the tcp window was closed (perhaps a slow reader) and then the other end closes the socket or a hard network error occurs, thus you can't tell if if was the last write that didn't get through, or a write you did 30 seconds ago.
Network error, or firewall silently drops your packets (no ICMP replys are generated), You will have to wait until TCP times out the connection to get an error which can be many seconds, usually several minutes.
TCP is busy doing retransmission as you call send - maybe those retransmissions generate an error.(really the same as the first case)
As per the docs, try calling sock.shutdown() before the call to sock.close().