i have tried to receive data form my connection by this code on python 2.7 :
server = socket(AF_INET, SOCK_STREAM)
server.bind(('0.0.0.0', 21))
server.listen(1)
client , addr = server.accept()
data = client.recv(2048)
When i have to print or send to another connection my data it's working , however i want to add those lines :
if(data == "/disconnect") :
<disconnect blha blha... you know >
else :
<print the data and send it back blha blha... >
( i have checked without that if statement and " disconnect blha blha .. " works nicely )
it's just pass the code so when my client requests to disconnect the request is sent to the server as the "original" messages .. (the sever don't kick him)
what should i do ? Thanks !
You have two problems, and you need to fix both of them.
First, a TCP socket is just a stream of bytes. When you do a recv, you're not going to get exactly one message sent by send from the other side, you're going to get whatever's in the buffer at the moment—which could be two messages, or half a message, or anything else. When you're just testing with localhost connections on a computer that isn't heavily loaded, on many platforms, it will do what you're hoping for >99% of the time—but that just makes the problem hard to debug, it doesn't fix it. And as soon as you try to access the same code over the internet, it'll start failing most of the time instead of rarely.
Fortunately, the client appears to be sending messages as text, without any embedded newlines, with a \r\n Windows-style end-of-line between each message. This is a perfectly good protocol; you just have to write the code to handle that protocol on the receive side.
The second problem is that, even if you happen to get exactly one message send, that message includes the \r\n end-of-line. And '/disconnect\r\n' == '/disconnect' is of course going to be false. So, as part of your protocol handler, you need to strip off the newlines.
As it happens, you could solve both problems by using the makefile method to give you a file object that you can iterate, or readline on, etc., just like you do with a file that you open from disk, which you probably already know how to handle.
But it's worth learning how to do this stuff, so I'll show you how to do it manually. The key is that you keep a buffer around, add each recv onto that buffer, and then split it into lines, put any remainder back on the buffer, and process each line as a message. There are more elegant/concise ways to write this, but let's keep it dead simple:
buf = ''
while True:
data = client.recv(2048)
buf += data
lines = buf.split('\r\n')
buf = lines.pop()
for line in lines:
# line is now a single message, with the newline stripped
if line == "/disconnect":
# do disconnect stuff
else:
# do normal message stuff
That's all you need to get the basics working. But in a real server, you also need some code to handle two other conditions—because clients don't always shut down cleanly. For example, if a client gets disconnected from the internet before it can send a /disconnect message, you don't want to keep spinning and reading nothing forever, you want to treat it as a disconnect.
if not data: means the client has done a clean (at the TCP level) shutdown. So, you need to disconnect and break out of the receive loop.
Depending on your design, it may be legal to shutdown only the send side and wait for a final reply from the server, so you want to make sure you've finished sending whatever you have. (This is common in many internet protocols.)
It may even be legal to not send a final newline before shutting down; if you want to support this, you should check if buf: and if so, treat buf as one last command. (This is not common in many protocol—but is a common bug in clients, so, e.g., many web servers will handle it.)
try:/except Exception as e: will catch all kinds of errors. These errors mean the socket is no longer usable (or that there's a serious error in your code, of course), so you want to handle this by throwing away the connection and breaking out of the receive loop, without first sending any final response or reading any final message.
It's almost always worth logging that e in some way (maybe just print 'Error from', addr, repr(e)), so if you're getting unexpected exceptions you have something to debug.
Related
I have to send some serial commands from a PC to an equipment as part of a bigger application and no matter what i try, i cannot seem to get python to send the data correctly. When i send the same commands through Termite, everything works as expected. I've been trying to solve this for days and i'm at a loss as to what i could try next.
The things i did and could rule out as not the cause for this issue:
Improper socket configuration: I've made sure to use the correct baud rate, bitlength, parity and stopbits as Termite (which happen to be the default settings for the socket anyways)
Check the command's correctness and termination: Within Termite i have to append CRLF as termination, so in Python i just add \r\n at the end of the command. This seems to be identical, as i've checked both commands on the oscilloscope and they are identical, termination characters included. I've went further and also used a serial port monitor and compared the two and they're identical - no missing bits and correct termination.
I've tried adding a heading \r\n to each command (this is what the equipment expects as command termination) thinking there might be some garbage or noise when first sending some data, but to no avail.
Clearing the input buffer, maybe there could be some issue when not reading from a socket that sends some response. I dont need what the equipment sends via serial, it offers no useful feedback but i did this anyways and have gotten no results. Before each transmission, i make sure to read all the bytes available.
Making sure Windows does not close my port. This i'm not that sure of, maybe it still does it after some long time, but so far i've gotten no errors and could always write and receive data, as confirmed by the serial monitor.
Below are some excerpts of socket configuration and command sending:
try:
self.SPC = serial.Serial(port=connectionData.get("spc_com"), baudrate=115200, timeout=1)
except serial.SerialException as err:
print(f'SPC Serial Error: {err.strerror}')
return False
def SPCCommand(self, command: str):
if not command:
return
try:
self.readAndClearBuffer()
self.SPC.write(command.encode())
except serial.SerialException as err:
print(f'SPC Serial Error: {err.strerror}')
return
def readAndClearBuffer(self):
data = ''
while True:
try:
data = self.SPC.read(1024)
if not data:
return
print(data)
except serial.SerialException as err:
return
self.SPCCommand("\r\n8000011200000000")
I don't have any idea what else to try. The fact that the commands are identical on the oscilloscope, as well as the serial monitor, leaves me at a loss. Could there be a different issue? Is there anything else i could try?
I have a problem with receiving data from server to client. I have the following client-side function that attempts to receive data from the server. The data sent by the server using the socket.sendall (data) function is greater than buff_size so I need a loop to read all the data.
def receiveAll (sock):
data = ""
buff_size = 4096
while True:
part = sock.recv (buff_size)
data + = part
if part <buff_size:
break;
return data
The problem that occurs to me is that after the first iteration (read the first 4096mb), in the second the program is blocked waiting for the other data in part = sock.recv (buff_size). How do I have to do so that recv() can continue reading the other missing data? Thank you.
Your interpretation is wrong. Your code reads all the data that it get from the server. It just doesn't know that it should stop listening for incoming data. It doesn't know that the server sent everything it had.
First of all note that these lines
if part <buff_size:
break;
are very wrong. First of all you are comparing a string to int (in Python3.x that would throw an exception). But even if you meant if len(part) <buff_size: then this is still wrong. Because first of all there might be a lag in the middle of streaming and you will only read a piece smaller then buff_size. Your code will stop there.
Also if your server sends a content of the size being a multiple of buff_size then the if part will never be satisfied and it will hang on .recv() forever.
Side note: don't use semicolons ;. It's Python.
There are several solutions to your problem but none of them can be used correctly without modyfing the server side.
As a client you have to know when to stop reading. But the only way to know it is if the server does something special and you will understand it. This is called a communication protocol. You have to add a meaning to data you send/receive.
For example if you use HTTP, then a server sends this header Content-Length: 12345 before body so now as a client you know that you only need to read 12345 bytes (your buffer doesn't have to be as big, but with that info you will know how many times you have to loop before reading it all).
Some binary protocols may send the size of the content in first 2 or 4 bytes for example. This can be easily interpreted on the client side as well.
Easier solution is this: simply make server close the connection after he sends all the data. Then you will only need to add check if not part: break in your code.
I am communicating with an instrument via TCP/IP using the Python socket package.
The program sends a command to the instrument to perform an action, and then repetitively sends another "check" command until it receives a "done" reply. However, after many loops, the program hangs while waiting for a "done" reply.
I have circumvented this problem by using the recv_timeout() function below, which returns no data if the socket is hanging, then I close the connection with socket.close() and reconnect.
Is there a more elegant solution without having to reboot anything?
import socket
import time
def recv_timeout(self,timeout=0.5):
'''
code from http://code.activestate.com/recipes/408859/
'''
self.s.setblocking(0)
total_data=[];data='';begin=time.time()
while 1:There must be a way I can reboot to carry on communicating with the instrument, without having to restart.
#if you got some data, then break after wait sec
if total_data and time.time()-begin>timeout:
break
#if you got no data at all, wait a little longer
elif time.time()-begin>timeout*2:
break
try:
data=self.s.recv(8192)
if data:
total_data.append(data)
begin=time.time()
else:
time.sleep(0.1)
except:
pass
return ''.join(total_data)
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect(('555.555.55.555',23))
for action_num in range(0,1000):
socket.sendall(('performaction %s \r'%action_num).encode())
while True:
time.sleep(0.2)
socket.sendall(('checkdone \r').encode())
done = socket.recv_timeout()
if not done:
print 'communication broken...what should I do?'
socket.close()
time.sleep(60)
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect(('555.555.55.555',23))
elif done == '1':
print 'done performing action'
break
socket.close()
I have circumvented this problem by using the recv_timeout() function
below, which returns no data if the socket is hanging
Are you certain that the socket will hang forever? What about the possibility that the instrument just sometimes takes more than half a second to respond? (Note that even if the instrument's software is good at responding in a timely manner, that is no guarantee that the response data will actually get to your Python program in a timely manner. For example, if the TCP packets containing the response get dropped by the network and have to be resent, that could cause them to take more than .5 seconds to return to your program. You can force that scenario to occur by pulling the Ethernet cable out of your PC for a second or two, and then plugging it back in... you'll see that the response bytes still make it through, just a second or two later on (after the dropped packets get resent); that is, if your Python program hasn't given up on them and closed the socket already.
Is there a more elegant solution without having to reboot anything?
The elegant solution is to figure out what is happening to the reply bytes in the fault scenario, and fixing the underlying bug so that the reply bytes no longer get lost. WireShark can be very helpful in diagnosing where the fault is; for example if WireShark shows that the response bytes did enter your computer's Ethernet port, then that is a pretty good clue that the bug is in your Python program's handling of the incoming bytes(*). On the other hand if the response bytes never show up in WireShark, then there might be a bug in the instrument itself that causes it to fail to respond sometimes. Wireshark would also show you if the problem is that your Python script failed to send out the "check" command for some reason.
That said, if you really can't fix the underlying bug (e.g. because it's a bug in the instrument and you don't have the ability to upgrade the source code of the software running on the instrument) then the only thing you can do is what you are doing -- close the socket connection and reconnect. If the instrument doesn't want to respond for some reason, you can't force it to respond.
(*) One thing to do is print out the contents of the string returned by recv_timeout(). You may find that you did get a reply, but it just wasn't the '1' string you were expecting.
I tried using a socket for 2 sends. The first one succeeds and the next one does not.
From the http://docs.python.org/howto/sockets.html
it would appear that multiple sends should be allowed. For Better or worse, I don't really need to read from the socket.
I have used twisted, but for the present purpose, I would like to stick to a socket, if I can help it(partly because I am using it within an application already using twisted to communicate.. this is a seperate connection).
"When the connect completes, the socket s can be used to send in a request for the text of the page. The same socket will read the reply, and then be destroyed. That’s right, destroyed. Client sockets are normally only used for one exchange (or a small set of sequential exchanges)."
return value for the send that succeeds = 35
return value for the send that FAILS = 32
code with some minor editing to remove any business logic.
self._commandSock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def sendPrereqs(self,id,prereqs):
self._commandSock.connect(self._commandConnection)
#parse prereqs
temp = prereqs.split(',')
for pair in temp:
tup = pair.partition(':')
try:
command = 'some command'
logging.info('sending command: ' + command)
ret = self._commandSock.send(command)
if ret == None:
logging.info('send called successfully: ' + command)
else:
logging.info('socket returned non-None: ' + str(ret))
except:
print 'Unexpected Exception ', sys.exc_info()[0]()
print sys.exc_info()
#logging.info('Unexpected Exception '+ str(sys.exc_info()[0]()))
#logging.info(' ' + str(sys.exc_info()))
self._commandSock.close()`
return value for the send that succeeds = 35 return value for the send that FAILS = 32
Documentation says that successful send should return None.
No it doesn't. Documentation says:
Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data. For further information on this concept, consult the Socket Programming HOWTO.
You still haven't explained what you mean by "FAILS". The send call is returning successfully, and it's almost certainly placed 32 bytes into the socket write buffer.
If the only reason you think it's failing is that it returns the correct value, then the answer is obviously that it's not failing.
If something else is going wrong, there are all kinds of things that could be wrong at a higher level. One likely one is this: The server (especially if it was coded by someone who doesn't understand sockets well) is coded to expect one recv() of 35 bytes, and one recv() of 32 bytes. But it's actually getting a single recv() of 67 bytes, and waiting forever for the second, which never comes. There is no rule guaranteeing that each send() on a stream (TCP) socket corresponds to one recv() on the other side. You need to create some kind of stream protocol that demarcates the separate messages.
Meanwhile, the quote you're referring to is irrelevant. It's describing how client sockets are used by simple web browsers: They make a connection, do one send, receive all the data, then destroy the connection. I can see why it misled you, and you may want to file a documentation bug to get it improved. But for now, just ignore it.
If you want to make absolutely sure that the client is sending the right data, and the problem is in the server, there are two easy ways to do that:
Use netcat as a trivial substitute server (e.g., nc -kl 6000, replacing the "6000" with the actual port) and making sure it logs what you think the server should be seeing.
Use Wireshark to watch the connection between the client and server.
Once you've verified that the problem is on the server side, you need to debug the server. If you need help with that, that's probably best done in a new question, where you can post the server code instead of the client, and explain (with a link here) that you're sure the client is sending the right information.
The documentation is only referring to a common scenario. You can call send, sendall, and sendto on all sockets as often as you want - as long as the socket is not closed.
Note that these methods return the number of bytes sent, 32 and 35 simply mean you sent 32 bytes the first and 35 bytes the second time.
The fact that socket.send returns without an exception means that the data got handed to the operating system, but not that it actually reached the endpoint (or has been read correctly by an application there).
I am making an iPhone application that communicates to a Twisted socket and it works great when I have one message to send. However, my issue is I need to send many different bits of information to the app. Here is my code.
if numrows == 1:
#Did login
msg = "%s: Login Credentials Success" % _UDID
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
for result in results:
for i in range(1, 6):
msg = "%s:L%d;%s" % (_UDID, i, result[i])
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
else:
msg = "%s: Login Credentials Failed" % _UDID
print msg
for c in self.factory.clients:
c.message(msg)
time.sleep(0.5)
cursor.close()
database.close()
#print msg
#for c in self.factory.clients:
#c.message(msg)
def message(self, message):
self.transport.write(message)
Say I were to send just the first msg, and every other msg didn't exist along with the print and for methods below each msg, the message Login Credentials Success would be sent to the app. But if put in the rest like how you are seeing it, nothing goes though because it sends it all at once, even with putting a time.sleep in the code.
The app checks the response every .05 seconds or less. Even though that the login credentials is on the top, it doesn't go through because there is more info that is being sent afterwards, but without all the info after the credentials message, it would go through.
I am desperate in finding the answer to this. I've tried everything I can think of. The app is not the issue, it's the Python.
Thanks.
At a risk of offending by contradicting, you may want to reexamine the claim that your app is not the problem. It sounds like you are expecting to have complete control over the content of each outgoing TCP packet, and that your app is depending on packet boundaries to determine the message boundaries. This isn't a very good approach to networking in general; some network intermediaries may split up (fragment) or even combine packets along the way, which would confuse your app. Remember that TCP is not a packet protocol: it is a stream protocol, so you're only really guaranteed that the octets you sent will be received in order, if at all.
A common approach to dealing with messages of varying size is to prefix each message with an X-bit big-endian size field stating how large the following message is. The receiving end of the communication reads X bits first, then reads 'size' octets after that to get the full message (blocking until that point, and leaving any additional information in the buffer for the next message handler to get).
I wouldn't mind helping more with your Twisted code, but it may very well be that it's already written properly. At least, I recommend not depending on trying to make Twisted flush network write buffers immediately. While it may help make your code appear to work right, right now, it will really just be hiding problems that will surface later.
Your issue appears to be that Twisted buffers the data you write to it.
Apparently, there is no easy way to force the data to be sent on its own without refactoring a great deal of your code. See Twisted transport.write and Using Twisted's twisted.web classes, how do I flush my outgoing buffers?.
Without knowing what your code looks like before the snipped you pasted and, according to the accepted answer of the last link:
define the wait function just as it is in the accepted answer
make sure that your class inherits from http.Request
decorate your method message with defer.inlineCallbacks
add yield wait(N) (for a value of N that you have to test and determine) after calls to write on the message method
I don't have enough experience with Twisted to know which of those steps are needed and which are just curft from the code of the original answer that doesn't apply to your code.
It may be possible (and easier) though, to re-write the iPhone application to accept the kind of messages that get sent when they include multiple writes in a single message.