Develop Reference

Recvall with while loop doesn't work between two devices in python

I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)

recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...

Python breaks string after first character

My Python socket program is breaking after the first character when receiving a single line string. The socket program runs in Raspberry Pi and the client and Java. Here is my code for socket
HOST = ''
PORT = 8888
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print('Socket created')
try:
s.bind((HOST, PORT))
except socket.error as socketError:
print('socket binding failed, ', socketError)
print('Exiting...')
sys.exit(0)
print('Socket binding complete')
s.listen(1)
print('Socket listening for connection...')
conn, addr = s.accept()
print('connected to ',addr[0])
try:
while True:
data = conn.recv(1024).decode('utf-8')
print('value received',data)
except Exception as loopException:
print("Exception occurred in loop, exiting...",loopException)
finally:
s.close()
This is how I send data from java
socket = new Socket(dstAddress, Integer.parseInt(dstPort));
os = new DataOutputStream(socket.getOutputStream());
os.writeBytes("Connection-Ready to receive commands");
This is what I get in terminal
value received C
value received onnected-Ready to receive commands
Does anybody know why it is behaving like this?

You need to read in a loop until you get all of the input. You need a way to know when you have all of the input.
If the client is writing and then closing the socket, then you can read until you get to EOF. If the client isn't closing the socket, then you need some way to know how much to read. The simplest thing is to have the client send a length word before sending the characters. You read the length and then read that many characters.
To put in a delay is a hack that may work for now, but isn't reliable. If you have some slowdown in your network, your code can break.
It looks from your output like you are closing the stream on the client side even though you don't show that. So then you just need to buffer up all your data until you reach the end of the stream. This should work:
buffer = b''
try:
while True:
data = conn.recv(1024)
if not data:
break
buffer += data
except Exception as loopException:
print("Exception occurred in loop, exiting...", loopException)
finally:
s.close()
print('value received', buffer.decode('utf-8'))
I think this will work. I don't have a connection to try it against, but my IDE likes it. I'm not that familiar with Python 3 byte streams. If this doesn't quite work, you should get the idea here anyway.

Sockets are a byte stream and there is no guarantee that a recv will have all the bytes of a complete message. It is your responsibility to buffer the received data until you have a complete message.
This also means you need to provide a way to know you have a complete message. Send the length of the message before the message, or read until a newline.

Making a 'byte type object' in python 3

I currently have a server running under this code:
import socket
port = 60000
s = socket.socket()
host = socket.gethostname()
s.bind((host, port))
s.listen(1)
print ('Server listening....')
while True:
conn, addr = s.accept()
print ('Got connection from', addr)
data = conn.recv(1024)
print('Server received', repr(data))
filename='mytext.txt'
f = open(filename,'rb')
l = f.read(1024)
while (l):
conn.send(l)
print('Sent ',repr(l))
l = f.read(1024)
f.close()
print('Done sending')
conn.send('Thank you for connecting')
conn.close()
And a client running under this code:
import socket
s = socket.socket()
host = socket.gethostname()
port = 60000
s.connect((host, port))
s.send('Test')
with open('received_file', 'wb') as f:
print ('file opened')
while True:
print('receiving data...')
data = s.recv(1024)
print('data=%s', (data))
if not data:
break
# write data to a file
f.write(data)
f.close()
print('Successfully get the file')
s.close()
print('connection closed')
The problem is that the when the client attempts to send the data it says that the data sent must be a byte-type object (Error code shown here: https://gyazo.com/97ef155f6809a801b02f381670895a2b.) I have searched the internet for an answer of how to create a byte type object inside of Python 3, but I have failed to find anything that has worked.

If you’re just sending literals, you can just use a bytes literal instead of a str literal, by prefixing it with a b. Instead of this:
sock.send('Hello')
… just do this:
sock.send(b'Hello')
If you’re sending string variables, you will want to encode them for sending, and probably decode them on receiving. Instead of this:
sock.send(msg)
resp = sock.recv(4096)
… do:
sock.send(msg.encode('ascii')
resp = sock.recv(4096).decode('ascii')
If you want to send non-ASCII strings, you’ll need to pick an encoding. Unless you have a good reason to do otherwise, use UTF-8.
One more thing to keep in mind: TCP sockets are just streams of bytes, not messages, and they can be split up arbitrarily into packets. When you can recv(1024), you may only get part of what the other side sent with send. That’s probably obvious if you’re sending strings longer than 1024 bytes, but even for shorter strings, it can happen. Or, if you’re not strictly alternating sends and receives, you can get multiple sends concatenate into a single receive. And if you send hello and then world and receive them as helloworld you have no way to know that happened, or how to split them back up.
And, worse, it probably won’t happen when you’re testing on localhost on an idle machine, so things will look good during development, but then fail mysteriously all over the place once you deploy it somewhere.
So, you need some protocol that describes where one message ends and another begins. I have a blog post that explains some options, and shows how to implement them.
But if you’re just sending human-readable strings as messages, and those strings will never contain newlines, the simplest protocol is just a line for every message, the same way you’d write a text file. And look at socket.makefile: it gives you something that acts just like a file, handles the one line for every message protocol, and also automatically handles the encode and decode parts. So, instead of this:
sock.send(msg.encode('utf-8'))
resp = sock.recv(1024).decode('utf-8')
… you just do use makefile to get readable and writable file objects, then do:
wf.write(msg + '\n')
resp = next(rf)
… and then you don’t have to worry about how to buffer up the receives and split them into messages.

Change the client code line:
s.send('Test')
to:
s.send(b'Test')
Find more details about socket.send() at https://docs.python.org/3/library/socket.html.

How can I continually send data without shutdown socket connection in python

I wrote a python client to communicate with server side. Each time when I finished sanding out data, I have to call sock.shutdown(socket.SHUT_WR), otherwise the server would not do any response. But after calling sock.shutdown(socket.SHUT_WR), I have to reconnect the connection as sock.connect((HOST, PORT)), other wise I can not send data to server. So how can I keep the connection alive without close it.
My sample code as following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
sock.sendall(data)
sock.shutdown(socket.SHUT_WR)
received = sock.recv(1024)
while len(received)>0:
received = sock.recv(1024)
sock.sendall(newdata) # this would throw exception
The Server Side code as following:
def handle(self):
cur_thread = threading.current_thread()
while True:
self.data = self.rfile.read(bufsiz=100)
if not self.data:
print 'receive none!'
break
try:
data = self.data
print 'Received data, length: %d' % len(data)
self.wfile.write('get received data\n')
except Exception:
print 'exception!!'

You didn't show any server side code but I suspect it simply reads bytes until it gets none anymore.
You can't do this as you found out, because then the only way to tell the server the message is complete is by killing the connection.
Instead you'll have to add some form of framing in your protocol. Possible approaches include a designated stop character that the server recognises (such as a single newline character, or perhaps a 0-byte), sending frames in fixed sizes that your client and server agree upon, or send the frame size first as a network encoded integer followed by exactly the specified number of bytes. The server then first reads the integer and then exactly that same number of bytes from the socket.
That way you can leave the connection open and send multiple messages.

When/why to use s.shutdown(socket.SHUT_WR)?

I have just started learning python network programming. I was reading Foundations of Python Network Programming and could not understand the use of s.shutdown(socket.SHUT_WR) where s is a socket object.
Here is the code(where sys.argv[2] is the number of bytes user wants to send, which is rounded off to a multiple of 16) in which it is used:
import socket, sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = '127.0.0.1'
PORT = 1060
if sys.argv[1:] == ['server']:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while True:
print 'Listening at', s.getsockname()
sc, sockname = s.accept()
print 'Processing up to 1024 bytes at a time from', sockname
n = 0
while True:
message = sc.recv(1024)
if not message:
break
sc.sendall(message.upper()) # send it back uppercase
n += len(message)
print '\r%d bytes processed so far' % (n,),
sys.stdout.flush()
print
sc.close()
print 'Completed processing'
elif len(sys.argv) == 3 and sys.argv[1] == 'client' and sys.argv[2].isdigit():
bytes = (int(sys.argv[2]) + 15) // 16 * 16 # round up to // 16
message = 'capitalize this!' # 16-byte message to repeat over and over
print 'Sending', bytes, 'bytes of data, in chunks of 16 bytes'
s.connect((HOST, PORT))
sent = 0
while sent < bytes:
s.sendall(message)
sent += len(message)
print '\r%d bytes sent' % (sent,),
sys.stdout.flush()
print
s.shutdown(socket.SHUT_WR)
print 'Receiving all the data the server sends back'
received = 0
while True:
data = s.recv(42)
if not received:
print 'The first data received says', repr(data)
received += len(data)
if not data:
break
print '\r%d bytes received' % (received,),
s.close()
else:
print >>sys.stderr, 'usage: tcp_deadlock.py server | client <bytes>'
And this is the explanation that the author provides which I am finding hard to understand:
Second, you will see that the client makes a shutdown() call on the socket after it finishes sending its transmission. This solves an important problem: if the server is going to read forever until it sees end-of-file, then how will the client avoid having to do a full close() on the socket and thus forbid itself from doing the many recv() calls that it still needs to make to receive the server’s response? The solution is to “half-close” the socket—that is, to permanently shut down communication in one direction but without destroying the socket itself—so that the server can no longer read any data, but can still send any remaining reply back in the other direction, which will still be open.
My understanding of what it will do is that it will prevent the client application from further sending the data and thus will also prevent the server side from further attempting to read any data.
What I cant understand is that why is it used in this program and in what situations should I consider using it in my programs?

My understanding of what it will do is that it will prevent the client
application from further sending the data and thus will also prevent
the server side from further attempting to read any data.
Your understanding is correct.
What I cant understand is that why is it used in this program …
As your own statement suggests, without the client's s.shutdown(socket.SHUT_WR) the server would not quit waiting for data, but instead stick in its sc.recv(1024) forever, because there would be no connection termination request sent to the server.
Since the server then would never get to its sc.close(), the client on his part also would not quit waiting for data, but instead stick in its s.recv(42) forever, because there would be no connection termination request sent from the server.
Reading this answer to "close vs shutdown socket?" might also be enlightening.

The explanation is half-baked, it applies only to this specific code and overall I would vote with all-fours that this is bad practice.
Now to understand why is it so, you need to look at a server code. This server works by blocking execution until it receives 1024 bytes. Upon reception it processes the data (makes it upper-case) and sends it back. Now the problem is with hardcoded value of 1024. What if your string is shorter than 1024 bytes?
To resolve this you need to tell the server that - hey there is no more data coming your way, so return from message = sc.recv(1024) and you do this by shutting down the socket in one direction.
You do not want to fully close the socket, because then the server would not be able to send you the reply.