I am writing a simple python socket program where client is sending a string to server and server says how many vowels in that string then send it back to client and client prints the output. Now, I do not know why it is not showing connection ended after sending end message. If I remove the \n from there the message appear in the same line for all except MESSAGE RECEIVED appear in the next line after Not enough vowel. Now how I can show CONNECTION ENDED also keeping all the messages in separate line.
server.py
import socket
FORMAT = 'utf-8'
PORT = 5050
HOST = socket.gethostbyname(socket.gethostname())
ADDR = (HOST,PORT)
DISCONNECTED_MESSAGE = 'End'
server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.bind(ADDR)
server.listen()
print("[LISTENING] server is listenning")
while True:
conn,addr = server.accept()
connected = True
while connected:
msg = conn.recv(2048).decode(FORMAT)
if msg == DISCONNECTED_MESSAGE:
connected = False
conn.send("CONNECTION ENDED".encode(FORMAT))
else:
vowel_count = 0
for chr in msg:
if chr in "aeiouAEIOU":
vowel_count += 1
if vowel_count == 0:
conn.send("Not enough vowels".encode(FORMAT))
elif vowel_count <= 2:
conn.send("Enough Vowel I guess\n".encode(FORMAT))
else:
conn.send("Too many vowels\n".encode(FORMAT))
conn.send("MESSAGE RECEIVED\n".encode(FORMAT))
conn.close()
client.py
import socket
PORT = 5050
HOST = socket.gethostbyname(socket.gethostname())
ADDR = (HOST,PORT)
FORMAT = 'utf-8'
DISCONNECTED_MESSAGE = 'End'
client=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
client.connect(ADDR)
def send(msg):
message = msg.encode(FORMAT)
client.send(message)
print(client.recv(2048).decode(FORMAT))
send("Nymphs cry gypsy rhythms Why Mythy nymphs cry ?Why nymphs flyby ?")
send("Nymphs cry gypsy rhythms hello Why Mythy nymphs cry ?Why nymphs flyby ?")
send("Nymphs cry gypsy rhythms Why Mythy nymphs cry ?Why nymphs flyby ? hello hello everybody good evening")
send(DISCONNECTED_MESSAGE)
The issue here is that TCP is a stream protocol, not a packet protocol. The reader does not get things in the same blocks you write. In your particular case, your server is sending two responses for each input (number of vowels and "MESSAGE RECEIVED"). Sometimes, those two messages get packed together and received in one read by the client, but sometimes, you only read the vowel evaluation by itself. Your client send function then returns, and sends its next message. It then reads "MESSAGE RECEIVED", even those the server has already queued up two more lines. By the time it gets to the end, it exits even though "CONNECTION ENDED" is still queued up.
The real solution is to have some "end of transmission" signal that your server sends when it is finished, like some unused control character. Your client should then continue to read until it gets a string that ends with your "end of transmission" signal.
The short term fix is just to have your server concatenate all the strings it wants to send into one conn.send call. Technically, TCP is allowed to break that up into smaller chunks for transmission, so it could still fail, but in practice that doesn't happen.
Related
I currently have a server at public IP: IP. I can use this Client Code to connect to the Server and establish a connection between the two.
Client.py
import socket
import select
import multiprocessing
class ClientAdminConnection:
# Port 80
def __init__(self):
self.ports = [80]
print("Ports: {}".format(self.ports))
self.sockets = []
self.createConnectionToServer()
# Multithread Listening
processes = []
for socket in self.sockets:
print("Creating Process")
process = multiprocessing.Process(target=self.startListeningForCommands, args=(socket,))
processes.append(process)
for process in processes:
print("Starting Process")
process.start()
def createConnectionToServer(self):
for port in self.ports:
client_admin_port = ('IP', port)
client_admin_socket = socket.socket()
# Connect to Server
client_admin_socket.connect(client_admin_port)
print("Connected to {}".format(client_admin_port))
self.sockets.append(client_admin_socket)
def startListeningForCommands(self, socket):
print("Listening")
wholeMsg = ""
while True:
data = socket.recv(1024)
if not data:
print("No data")
break
else:
dataBit = data.decode("utf-8")
print("There is Data: {}".format(dataBit))
wholeMsg += dataBit
print("Message: {}".format(wholeMsg))
I'd like to be able to send multiple messages and have the client listening after I .connect()
Currently, I can get it where I can read the data bit. But, I'd like to be able to send 1 message - see the entire thing (even if longer than 1024 bytes) and then send another message. Right now - I can only see the dataBits that come in. I never get to see wholeMsg. Generally, most of my messages are small - but I would like to use an arbitrary size and still see the entire message every time one is sent over the connection.
Current Results:
Ports: [80]
Connected to ('34.207.93.146', 80)
Creating Process
Starting Process
Listening
There is Data: Mary Had A Little Lamb
Mary Had A Little Lamb is sent from the server.
What I Expect:
Ports: [80]
Connected to ('34.207.93.146', 80)
Creating Process
Starting Process
Listening
There is Data: Mary Had A Little Lamb
Message: Mary Had A Little Lamb *THIS SECTION IS MISSING
TCP is a stream-oriented connection. It is not packet-oriented. A single transmission might be chopped into several pieces, or concatenated with other unread data. YOU have to establish a way to know when a message is finished, such as a \n. That way, the receiver keeps reading, and appending to the message buffer, until it sees the \n.
I appreciate Tim's answer, but wanted to highlight some simple concepts that I was missing as a new socket-eer in hopes of anyone else trying to understand.
def startListeningForCommands(self, socket):
print("Listening")
wholeMsg = ""
while True:
data = socket.recv(1024)
if not data:
print("No data")
break
else:
dataBit = data.decode("utf-8")
print("There is Data: {}".format(dataBit))
wholeMsg += dataBit
The above won't work in the sense that wholeMsg will never get called because
while True:
data = socket.recv(1024)
will continuously portion out segments of data into 1024 bytes IF there is data obtained. It is to the programmer's discretion to determine when a whole message has been received. This is where headers and footers come into play (to my understanding). But, there will never not be data, assuming the socket is open and listening.
For example, you would get:
Listening
"There is Data: I am the first message"
"There is Data: I am the second message"
If there was more than 1024 bytes, say 1200 bytes it would be
"There is Data: 'First 1024 bytes'"
"There is Data: 'Next 176 bytes'"
"There is Data: I am the second message"
It is up to you to determine where the start of one message is and the end of that message. How you do that... I am still researching. Just wanted to help out anyone else struggling with that concept.
I am trying simple client and sinple server , client send 2 massages and server recieve 2 massages . I am printg first massage print(data1) and second massage print(data2) and somehow it print both massages in the same line . Can you help ?
server
import socket
SERVER_IP = '0.0.0.0'
DEST_PORT = 1731
server_socket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server_socket.bind((SERVER_IP,DEST_PORT))
server_socket.listen(1)
client_socket,address=server_socket.accept()
data1 = client_socket.recv(512).decode()
print(data1)
data2 = client_socket.recv(512).decode()
print(data2)
client_socket.close()
server_socket.close()
cliend code
#client
import socket
HOST_IP = '127.0.0.1'
DEST_PORT = 1731
my_socket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
my_socket.connect((HOST_IP,DEST_PORT))
data1 = "test1"
print("client send")
my_socket.sendall(data1.encode())
my_socket.sendall("second send ".encode())
my_socket.close()
TCP sockets are a stream-oriented protocol, not a message-oriented one.
What's happening is that your client ends up sending b"test1second send " before your server reads anything from the socket.
When it does, it reads up to 512 characters – more than enough to contain your message – and again, up to 512 characters yet there are zero to read at that point and an empty line gets printed.
Depending on your actual requirements, you will need to figure out a way to delimit or encapsulate your messages; for text, a popular choice is by newline characters (though it will land you in some hot water should you need to transmit an actual newline in a message).
A better choice still is something like netstrings, where each message is preceded by its length (in ASCII for netstrings, but it could be a binary unsigned integer (4 bytes), or whatever you specify), or some sort of TLV scheme.
I have just started learning python network programming. I was reading Foundations of Python Network Programming and could not understand the use of s.shutdown(socket.SHUT_WR) where s is a socket object.
Here is the code(where sys.argv[2] is the number of bytes user wants to send, which is rounded off to a multiple of 16) in which it is used:
import socket, sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = '127.0.0.1'
PORT = 1060
if sys.argv[1:] == ['server']:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while True:
print 'Listening at', s.getsockname()
sc, sockname = s.accept()
print 'Processing up to 1024 bytes at a time from', sockname
n = 0
while True:
message = sc.recv(1024)
if not message:
break
sc.sendall(message.upper()) # send it back uppercase
n += len(message)
print '\r%d bytes processed so far' % (n,),
sys.stdout.flush()
print
sc.close()
print 'Completed processing'
elif len(sys.argv) == 3 and sys.argv[1] == 'client' and sys.argv[2].isdigit():
bytes = (int(sys.argv[2]) + 15) // 16 * 16 # round up to // 16
message = 'capitalize this!' # 16-byte message to repeat over and over
print 'Sending', bytes, 'bytes of data, in chunks of 16 bytes'
s.connect((HOST, PORT))
sent = 0
while sent < bytes:
s.sendall(message)
sent += len(message)
print '\r%d bytes sent' % (sent,),
sys.stdout.flush()
print
s.shutdown(socket.SHUT_WR)
print 'Receiving all the data the server sends back'
received = 0
while True:
data = s.recv(42)
if not received:
print 'The first data received says', repr(data)
received += len(data)
if not data:
break
print '\r%d bytes received' % (received,),
s.close()
else:
print >>sys.stderr, 'usage: tcp_deadlock.py server | client <bytes>'
And this is the explanation that the author provides which I am finding hard to understand:
Second, you will see that the client makes a shutdown() call on the socket after it finishes sending its transmission. This solves an important problem: if the server is going to read forever until it sees end-of-file, then how will the client avoid having to do a full close() on the socket and thus forbid itself from doing the many recv() calls that it still needs to make to receive the server’s response? The solution is to “half-close” the socket—that is, to permanently shut down communication in one direction but without destroying the socket itself—so that the server can no longer read any data, but can still send any remaining reply back in the other direction, which will still be open.
My understanding of what it will do is that it will prevent the client application from further sending the data and thus will also prevent the server side from further attempting to read any data.
What I cant understand is that why is it used in this program and in what situations should I consider using it in my programs?
My understanding of what it will do is that it will prevent the client
application from further sending the data and thus will also prevent
the server side from further attempting to read any data.
Your understanding is correct.
What I cant understand is that why is it used in this program …
As your own statement suggests, without the client's s.shutdown(socket.SHUT_WR) the server would not quit waiting for data, but instead stick in its sc.recv(1024) forever, because there would be no connection termination request sent to the server.
Since the server then would never get to its sc.close(), the client on his part also would not quit waiting for data, but instead stick in its s.recv(42) forever, because there would be no connection termination request sent from the server.
Reading this answer to "close vs shutdown socket?" might also be enlightening.
The explanation is half-baked, it applies only to this specific code and overall I would vote with all-fours that this is bad practice.
Now to understand why is it so, you need to look at a server code. This server works by blocking execution until it receives 1024 bytes. Upon reception it processes the data (makes it upper-case) and sends it back. Now the problem is with hardcoded value of 1024. What if your string is shorter than 1024 bytes?
To resolve this you need to tell the server that - hey there is no more data coming your way, so return from message = sc.recv(1024) and you do this by shutting down the socket in one direction.
You do not want to fully close the socket, because then the server would not be able to send you the reply.
I am working on a proof of concept using python that emulates a server/client communication using sockets to send UDP packets. I can easily do a simple client to server and back to client comms, but I am trying to introduce a "middle-man" into that communication. Conceptually the problem can be descirbed as, if "Joe" is the main client, he will send a message to "Steve" who is the middle man who will do something with that message before sending it to "Carol" who acts as the server that will process the new message and send a response back to the middle-man, "Steve". Eventually the middle-man will then send that message on elsewhere, but at the moment I am not worrying about that.
My current code looks like:
"Joe" (original client) looks like
# create dgram udp socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
except socket.error:
print ('Failed to create socket')
sys.exit()
host = 'localhost'
port = 8888
print("start comms")
while 1:
arr = ['Dog', 'cat', 'treE', 'Paul']
num = random.randrange(0,4)
#Send the string
s.sendto(arr[num].encode(), (host, port))
"Steve" (middle man) looks like
host = ''
hostRT = 'localhost'
portVM = 8888
portRT = 8752
# socket to receive from "Joe"
s1 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s1.bind((host, portVM))
# socket to send to "Carol"
s2 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
print("start comms")
while 1:
# receive from "Joe"
data = s1.recvfrom(1024)
num = data[0].decode()
addrVM = data[1]
# print data from Joe
print(num)
# add some excitement to Joe's message
num += '!!!'
# show received message address + port number
print ("message[" + addrVM[0] + ":" + str(addrVM[1]) + ']')
# Send to "Carol"
s2.sendto(num.encode(), (hostRT, portRT))
# receive from "Carol"
d = s2.recvfrom(1024)
reply = d[0].decode()
addrRT = d[1]
# show received message address + port number
print ("message[" + addrRT[0] + ":" + str(addrRT[1]) + ']')
# show Carol's response
print ('Server reply : ' + reply)
s1.close()
s2.close()
"Carol" (server) looks like
host = ''
port = 8752
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
print ("socket created")
s.bind((host, port))
print ("Socket bind complete")
while 1:
d = s.recvfrom(1024)
data = d[0].decode()
addr = d[1]
print(data)
reply = "Upper case client data = " + data.upper()
print(reply)
s.sendto(reply.encode(), addr)
print ("message[" + addr[0] + ":" + str(addr[1]) + '] - ' + data.strip())
s.close()
Currently I can receive a message from Joe but then it hangs on the sending to the server Carol. I'm completely new to socket programming so any help would be greatly appreciated.
Edit for clarification
Using Python 3.4
Joe is sending packets non stop as to emulate the real life application that this proof of concept is for. Joe will be sending packets at a rate of roughly 1 packet / 4ms, but I am only concerned with the most recent packet. However, since the average turn around time for the round trip from Steve to Carol is around 10ms, I had originally thought to cache Joe's most recent packet in a local memory location and overwrite that location until Steve is ready to send a packet to Carol once she has responded with the last packet. However, for this simple proof of concept I haven't tried to implement that. Any suggestions on that would also be helpful.
There are multiple faults that contribute to the overall failure, some of which are not apparent (i.e. it sort of works until it crashes down somewhere else).
First of all, at the moment sends packets as fast as he cans. That alone can lead to significant packet loss everywhere else (that might be a good thing, since you now have to make sure your code survives packet loss). Unless you truly want to stress the network, something like time.sleep(0.1) would be appropriate in the send loop.
More importantly, steve's socket setup is all messed up. He needs two sockets at the most, not three. The way it is currently set up, carol answers steve to the IP address and port she got the packet from (which is quite sensible), but steve reads on a distinct socket that never gets data sent to.
To make matters worse, the port steve's s3 listens on is actually the same one that carol uses! Since you are not using multicast. You can simply remove every reference to s3 from the code and use s2 exclusively.
Another problem is that you don't deal with packet loss. For example, if a packet gets lost between steve and carol, the code
# Send to "Carol"
s2.sendto(num.encode(), (hostRT, portRT))
# receive from "Carol"
d = s2.recvfrom(1024) # s3 in the original
will hang forever, since Carol does not send any new packets after the one that got lost. As mentioned before, packet loss is way more likely since joe is blasting out packets as fast as he can.
To detect packet loss, there are a few options:
Use multiple threads or processes for sending and receinv. This is going to make your program way more complex.
Switch to asynchronous / non-blocking IO, in Python with the high-level asyncore or the more low-level select.
Set and correctly handle socket timeouts. This is probably the easiest option for now, but is quite limited.
Switch to TCP if you actually need reliable communication.
Apart from the aforementioned network problems, there are also some potential problems or inaccuracies:
If you are using Python 2.x, the behavior of decode and encode on strings depends on the system configuration. Like many other potential problems, this has been fixed in Python 3.x by mandating UTF-8 (in 2.x you have to explicitly request that). In your case, that's fine as long as you only send ASCII characters.
while(1) : looks really strange in Python - why the whitespace after the argument, and why parentheses . Why not while 1: or while True:?
You can use tuple unpacking to great effect. Instead of
data = s1.recvfrom(1024)
num = data[0].decode()
addrVM = data[1]
how about:
data, addrVM = s1.recvfrom(1024)
num = data.decode('utf-8')
I attempted to find an answer for this, but most examples out there are for purely echo base Socket Servers.
Basically I have the following code:
import socket
import sys
from thread import *
HOST = '' # Symbolic name meaning all available interfaces
PORT = 8888 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print 'Socket created'
#Bind socket to local host and port
try:
s.bind((HOST, PORT))
except socket.error as msg:
print 'Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1]
sys.exit()
print 'Socket bind complete'
#Start listening on socket
s.listen(10)
print 'Socket now listening'
#Function for handling connections. This will be used to create threads
def clientthread(conn):
#Sending message to connected client
conn.send('Welcome to the server. Type something and hit enter\n') #send only takes string
#infinite loop so that function do not terminate and thread do not end.
while True:
#Receiving from client
data = conn.recv(1024)
if data == "hello":
reply = 'OK...Hello back to you'
else:
reply = '01:OK - ' + data
if not data:
break
conn.sendall(reply)
#came out of loop
conn.close()
#now keep talking with the client
while 1:
#wait to accept a connection - blocking call
conn, addr = s.accept()
print 'Connected with ' + addr[0] + ':' + str(addr[1])
#start new thread takes 1st argument as a function name to be run, second is the tuple of arguments to the function.
start_new_thread(clientthread ,(conn,))
s.close()
Everything works great until I try to use a conditional statement. I am very new to python and I am using this as a way to learn it a little better, but when the following line runs it skips the if each time.
#Receiving from client
data = conn.recv(1024)
if data == "hello":
reply = 'Why hello there!'
else:
reply = '01:OK - ' + data
if not data:
break
conn.sendall(reply)
From the telnet client I am connecting to it just echos everything I send it including the 'hello' I send it rather than the phrase.
I have a feeling that it is something simple, but I am not sure the format of the data variable.
Very close!
Telnet will send whatever EOL delimiter you gave it, along with the text. So if you type "data" and press enter, data is actually something like hello\r\n.
You can effectively ignore this whitespace by doing something like changing
data = conn.recv(1024)
to
data = conn.recv(1024).strip()
And you should be good to go.
EDIT:
As noted in the comments, the network may split up the message into multiple packets. To work around this, you can use use the socket.makefile() method and get a file-like object and then use readline() which will block until a complete line is available. For example, changing clientthread to:
def clientthread(conn):
#Sending message to connected client
conn.send('Welcome to the server. Type something and hit enter\n') #send only takes string
sfh = conn.makefile("r+b", bufsize=0)
#infinite loop so that function do not terminate and thread do not end.
while True:
#Receiving from client
data = sfh.readline().strip()
if data == "hello":
reply = 'OK...Hello back to you'
else:
reply = '01:OK - ' + data
if not data:
break
conn.sendall(reply)
#came out of loop
conn.close()
== PYTHON ==
Socket created
Socket bind complete
Socket now listening
Connected with 192.168.1.10:42749
== TELNET ==
$ telnet 192.168.1.106 8888
Trying 192.168.1.106...
Connected to 192.168.1.106.
Escape character is '^]'.
Welcome to the server. Type something and hit enter
hello
OK...Hello back to you