Develop Reference

One Output is not showing in python socket programming

I am writing a simple python socket program where client is sending a string to server and server says how many vowels in that string then send it back to client and client prints the output. Now, I do not know why it is not showing connection ended after sending end message. If I remove the \n from there the message appear in the same line for all except MESSAGE RECEIVED appear in the next line after Not enough vowel. Now how I can show CONNECTION ENDED also keeping all the messages in separate line.
server.py
import socket
FORMAT = 'utf-8'
PORT = 5050
HOST = socket.gethostbyname(socket.gethostname())
ADDR = (HOST,PORT)
DISCONNECTED_MESSAGE = 'End'
server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.bind(ADDR)
server.listen()
print("[LISTENING] server is listenning")
while True:
conn,addr = server.accept()
connected = True
while connected:
msg = conn.recv(2048).decode(FORMAT)
if msg == DISCONNECTED_MESSAGE:
connected = False
conn.send("CONNECTION ENDED".encode(FORMAT))
else:
vowel_count = 0
for chr in msg:
if chr in "aeiouAEIOU":
vowel_count += 1
if vowel_count == 0:
conn.send("Not enough vowels".encode(FORMAT))
elif vowel_count <= 2:
conn.send("Enough Vowel I guess\n".encode(FORMAT))
else:
conn.send("Too many vowels\n".encode(FORMAT))
conn.send("MESSAGE RECEIVED\n".encode(FORMAT))
conn.close()
client.py
import socket
PORT = 5050
HOST = socket.gethostbyname(socket.gethostname())
ADDR = (HOST,PORT)
FORMAT = 'utf-8'
DISCONNECTED_MESSAGE = 'End'
client=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
client.connect(ADDR)
def send(msg):
message = msg.encode(FORMAT)
client.send(message)
print(client.recv(2048).decode(FORMAT))
send("Nymphs cry gypsy rhythms Why Mythy nymphs cry ?Why nymphs flyby ?")
send("Nymphs cry gypsy rhythms hello Why Mythy nymphs cry ?Why nymphs flyby ?")
send("Nymphs cry gypsy rhythms Why Mythy nymphs cry ?Why nymphs flyby ? hello hello everybody good evening")
send(DISCONNECTED_MESSAGE)

The issue here is that TCP is a stream protocol, not a packet protocol. The reader does not get things in the same blocks you write. In your particular case, your server is sending two responses for each input (number of vowels and "MESSAGE RECEIVED"). Sometimes, those two messages get packed together and received in one read by the client, but sometimes, you only read the vowel evaluation by itself. Your client send function then returns, and sends its next message. It then reads "MESSAGE RECEIVED", even those the server has already queued up two more lines. By the time it gets to the end, it exits even though "CONNECTION ENDED" is still queued up.
The real solution is to have some "end of transmission" signal that your server sends when it is finished, like some unused control character. Your client should then continue to read until it gets a string that ends with your "end of transmission" signal.
The short term fix is just to have your server concatenate all the strings it wants to send into one conn.send call. Technically, TCP is allowed to break that up into smaller chunks for transmission, so it could still fail, but in practice that doesn't happen.

Recvall with while loop doesn't work between two devices in python

I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)

recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...

Python TCP sockets headers

I need to create a communication between a client and a server with TCP. But I'd like to send and work with "headers". So from the client I'd like to send a header "COMMAND1" and the server returns me something.
I have the following code:
Server
import socket
import threading
bind_ip = '0.0.0.0'
bind_port = 9998
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(5) # max backlog of connections
print ('Listening on {}:{}'.format(bind_ip, bind_port))
def handle_client_connection(client_socket):
request = client_socket.recv(1024)
print ('Received {}'.format(request))
client_socket.send('Response1!'.encode('utf-8'))
client_socket.close()
while True:
client_sock, address = server.accept()
print ('Accepted connection from {}:{}'.format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,) # without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
Client
import socket
hostname, sld, tld, port = 'www', 'integralist', 'co.uk', 80
target = '{}.{}.{}'.format(hostname, sld, tld)
# create an ipv4 (AF_INET) socket object using the tcp protocol (SOCK_STREAM)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect the client
# client.connect((target, port))
client.connect(('0.0.0.0', 9998))
# send some data (in this case a HTTP GET request)
client.send('hi'.encode('utf-8'))
# receive the response data (4096 is recommended buffer size)
response = client.recv(4096)
print (response)
Anyone knows the best way to return "Response1!" when the header is "COMMAND1" and "Response2!" when the header is "COMMAND2"?
I can't find examples on how to use headers
EDIT: It doesn't have to be "COMMAND1" or "COMMAND2" it can be a "0" or "1", or anything else.

If you want to add your own header, you just have to:
Make sure your programm finds the start of your message (like, every message beginns "!?&")
Send your own header-data just after the start-symbol of your message.
Maybe mark the end of your message with something or pass a length in your header.
Since TCP will give you a stream of data, it might come to a case, where it just gives you 2 or 3 messages at once. You have to separate these messages by yourself (e.g. by using "?!&" as start of every message).
You can always create your own protocoll as payload of another protocoll. Just as TCP is just payload from the ethernet point of view.

You can do something i have done with my program to accept such headers
Use pickle library to encode a dict headers and send it through socket.
Code will look something like this.
import pickle
def handleSocket(headers:dict):
message = pickle.dumps(headers)
socket.send(message)
For server side, you will be handling it
Gonna initialise the socket recv to 100 kb
def handleReceivedSocket(socket):
message:dict = pickle.loads(socket.recv(102400))
Another way to do this. Is sending a raw json string to the server (just change pickle.dumps,pickle.loads by json.dumps,json.loads
But it will be in raw and less secure.
Last way you can do it is uri encoding. Check w3schools

Python optimize packets in sending. How to prevent it?

Im trying to send 3 packets one after the other with python socket.
Python optimize it to one or 2 packets.
I prevented it by sleep command, but it takes too long time.
I thought to turn on the TCP urg flag, Does someone know how to do it?
or you have another solotion?
client side:
import socket
from time import sleep
IP = '127.0.0.1'
PORT = 5081
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((IP, PORT))
s.send('1'*5)
#sleep( 1)
s.send('2'*5)
#sleep( 1)
s.send('3'*5)
s.close()
server side:
import socket
IP = '0.0.0.0'
PORT = 5081
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((IP, PORT))
s.listen(1)
connection, address = s.accept()
while 1:
#Here I expected to get the 1nd value
data1 = connection.recv(BUFFER_SIZE)
#end of communication
if not data1:
break
print 'data1', data1
#Here I expected to get the 2nd value,but both inputs arrived here, 22222 and 33333
data2 = connection.recv(BUFFER_SIZE)
print 'data2', data2
#Here I expected to get the 3nd value
data3 = connection.recv(BUFFER_SIZE)
print 'data3', data3
connection.close()
thanks
Avinoam

You should not even try. TCP is a stream protocol and should be used as a stream protocol (meaning a single sequence of bytes). Even if you manage to maintain the separation of packets when you use localhost on your system, it could break if you use it between different hosts, or simply after an upgrade of the TCP/IP stack. And as soon as your packets will pass through a proxy or a software filter, anything can happen.
The correct way to separate different objects on a stream is to use an upper level protocol encoding the objects sender side and decoding them client side. An example of that is one or two bytes (in network order if more than one byte) for the size followed by the relevant bytes. Or you could imagine a text protocol with commands, headers and data, or [put whatever you want here]

When/why to use s.shutdown(socket.SHUT_WR)?

I have just started learning python network programming. I was reading Foundations of Python Network Programming and could not understand the use of s.shutdown(socket.SHUT_WR) where s is a socket object.
Here is the code(where sys.argv[2] is the number of bytes user wants to send, which is rounded off to a multiple of 16) in which it is used:
import socket, sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = '127.0.0.1'
PORT = 1060
if sys.argv[1:] == ['server']:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while True:
print 'Listening at', s.getsockname()
sc, sockname = s.accept()
print 'Processing up to 1024 bytes at a time from', sockname
n = 0
while True:
message = sc.recv(1024)
if not message:
break
sc.sendall(message.upper()) # send it back uppercase
n += len(message)
print '\r%d bytes processed so far' % (n,),
sys.stdout.flush()
print
sc.close()
print 'Completed processing'
elif len(sys.argv) == 3 and sys.argv[1] == 'client' and sys.argv[2].isdigit():
bytes = (int(sys.argv[2]) + 15) // 16 * 16 # round up to // 16
message = 'capitalize this!' # 16-byte message to repeat over and over
print 'Sending', bytes, 'bytes of data, in chunks of 16 bytes'
s.connect((HOST, PORT))
sent = 0
while sent < bytes:
s.sendall(message)
sent += len(message)
print '\r%d bytes sent' % (sent,),
sys.stdout.flush()
print
s.shutdown(socket.SHUT_WR)
print 'Receiving all the data the server sends back'
received = 0
while True:
data = s.recv(42)
if not received:
print 'The first data received says', repr(data)
received += len(data)
if not data:
break
print '\r%d bytes received' % (received,),
s.close()
else:
print >>sys.stderr, 'usage: tcp_deadlock.py server | client <bytes>'
And this is the explanation that the author provides which I am finding hard to understand:
Second, you will see that the client makes a shutdown() call on the socket after it finishes sending its transmission. This solves an important problem: if the server is going to read forever until it sees end-of-file, then how will the client avoid having to do a full close() on the socket and thus forbid itself from doing the many recv() calls that it still needs to make to receive the server’s response? The solution is to “half-close” the socket—that is, to permanently shut down communication in one direction but without destroying the socket itself—so that the server can no longer read any data, but can still send any remaining reply back in the other direction, which will still be open.
My understanding of what it will do is that it will prevent the client application from further sending the data and thus will also prevent the server side from further attempting to read any data.
What I cant understand is that why is it used in this program and in what situations should I consider using it in my programs?

My understanding of what it will do is that it will prevent the client
application from further sending the data and thus will also prevent
the server side from further attempting to read any data.
Your understanding is correct.
What I cant understand is that why is it used in this program …
As your own statement suggests, without the client's s.shutdown(socket.SHUT_WR) the server would not quit waiting for data, but instead stick in its sc.recv(1024) forever, because there would be no connection termination request sent to the server.
Since the server then would never get to its sc.close(), the client on his part also would not quit waiting for data, but instead stick in its s.recv(42) forever, because there would be no connection termination request sent from the server.
Reading this answer to "close vs shutdown socket?" might also be enlightening.

The explanation is half-baked, it applies only to this specific code and overall I would vote with all-fours that this is bad practice.
Now to understand why is it so, you need to look at a server code. This server works by blocking execution until it receives 1024 bytes. Upon reception it processes the data (makes it upper-case) and sends it back. Now the problem is with hardcoded value of 1024. What if your string is shorter than 1024 bytes?
To resolve this you need to tell the server that - hey there is no more data coming your way, so return from message = sc.recv(1024) and you do this by shutting down the socket in one direction.
You do not want to fully close the socket, because then the server would not be able to send you the reply.