My Python socket program is breaking after the first character when receiving a single line string. The socket program runs in Raspberry Pi and the client and Java. Here is my code for socket
HOST = ''
PORT = 8888
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print('Socket created')
try:
s.bind((HOST, PORT))
except socket.error as socketError:
print('socket binding failed, ', socketError)
print('Exiting...')
sys.exit(0)
print('Socket binding complete')
s.listen(1)
print('Socket listening for connection...')
conn, addr = s.accept()
print('connected to ',addr[0])
try:
while True:
data = conn.recv(1024).decode('utf-8')
print('value received',data)
except Exception as loopException:
print("Exception occurred in loop, exiting...",loopException)
finally:
s.close()
This is how I send data from java
socket = new Socket(dstAddress, Integer.parseInt(dstPort));
os = new DataOutputStream(socket.getOutputStream());
os.writeBytes("Connection-Ready to receive commands");
This is what I get in terminal
value received C
value received onnected-Ready to receive commands
Does anybody know why it is behaving like this?
You need to read in a loop until you get all of the input. You need a way to know when you have all of the input.
If the client is writing and then closing the socket, then you can read until you get to EOF. If the client isn't closing the socket, then you need some way to know how much to read. The simplest thing is to have the client send a length word before sending the characters. You read the length and then read that many characters.
To put in a delay is a hack that may work for now, but isn't reliable. If you have some slowdown in your network, your code can break.
It looks from your output like you are closing the stream on the client side even though you don't show that. So then you just need to buffer up all your data until you reach the end of the stream. This should work:
buffer = b''
try:
while True:
data = conn.recv(1024)
if not data:
break
buffer += data
except Exception as loopException:
print("Exception occurred in loop, exiting...", loopException)
finally:
s.close()
print('value received', buffer.decode('utf-8'))
I think this will work. I don't have a connection to try it against, but my IDE likes it. I'm not that familiar with Python 3 byte streams. If this doesn't quite work, you should get the idea here anyway.
Sockets are a byte stream and there is no guarantee that a recv will have all the bytes of a complete message. It is your responsibility to buffer the received data until you have a complete message.
This also means you need to provide a way to know you have a complete message. Send the length of the message before the message, or read until a newline.
Related
Good evening,
I am trying to make a basic implementation of the clients-to-server model using sockets.
Basically, I want the client to retrieve a small piece of information (IE: MESSAGE key) to the server. To accomplish this, I've used threading.
When the client sends the request to receive the info, there is no error on the client side.
However, the server sends an error
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Users\Stock\AppData\Local\Continuum\anaconda3\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users\Stock\AppData\Local\Continuum\anaconda3\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "server.py", line 40, in listeningForRetriever
dataBytes = conn.recv(1024)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
After doing some research, this thread seemed pretty relevant, https://stackoverflow.com/a/49289545/6902431, but I had already implemented the suggested fixes.
server.py
import socket
import threading
from Utility import *
HOST = '127.0.0.1' # Standard loopback interface address (localhost)
PORT = 65432 # Port to listen on (non-privileged ports are > 1023)
PORT_A = 22221
PORT_B = 22222
PORT_C = 22223
bank = {'me':'you'}
def listeningForRetriever():
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((HOST, PORT_B))
s.listen()
conn, addr = s.accept()
with conn:
while True:
dataBytes = conn.recv(1024)
dataString = bytesToString(dataBytes)
try:
retrievedValueBytes = bytesToString(bank[cleanGET(dataString)])
print(f"Value: {retrievedValueBytes} has been retrieved with Key {cleanGET(dataString)}")
conn.sendall(retrievedValueBytes)
except:
print("SERVER: Errored out")
conn.sendall(stringToBytes("ERROR! Value retrieval for command: " + dataString + " doesn't exist!"))
retrieverThread = threading.Thread(target=listeningForRetriever)
retrieverThread.start()
retriever.py
#Retrieves values from server using keys
#retriever.py --key=username
import sys
import socket
from Utility import *
# total arguments
n = len(sys.argv)
print("Total arguments passed:", n)
# Arguments passed
print("\nName of Python script:", sys.argv[0])
print("\nArguments passed:", end = " ")
key = ''
value = ''
for i in range(1, n):
print(sys.argv[i], end = " ")
if '--key=' in sys.argv[i]:
data = sys.argv[i].replace('--key=','')
key = data
print()
if key == '':
print("ERROR! Invalid syntax.")
print("Try something like: Bob.py --key=username")
else:
print(f"Key: {key} {type(key)}")
#TODO: Send key to storage of server
HOST = '127.0.0.1' # The server's hostname or IP address
PORT = 22222 # The port used by the server
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(stringToBytes("GET key"+"="+key))
print("DEBUG 1")
value = s.recv(1024)
print("DEBUG 2")
print(f"Retrieved Value: {value}")
print(f"Value Retrieved: {value}")
s.close()
What am I overlooking/misunderstanding?
I could (more or less...) reproduce and fix. But I may be wrong, because your code uses an Utility module that you failed to show so I could only guess from the names what bytesToString, stringToBytes and cleanGET were supposed to do.
First there is a misunderstanding about how TCP actually behaves. It is a streaming protocol, that guarantees that all sent bytes will be received an in same order. But there is no guarantee on the packets themselves: they may be splitted and/or reassembled by the network. So you must use a higher level way to delimit a message. It does not exhibit any problem here because short packets are often left unmodified, but the protocol does not guarantee that so you could experience weird errors later. Here as you send a single message, you could shut down the socket to signal the end of the message: the receiver will see a 0 size packet at that point.
Next (and the actual cause of your problem), you loop after accepting a connection. So you get a message, successfully process it, read again on a socket that will be closed by the peer and get a 0 size byte string. It probably causes a key error when looking in the bank dict, but you silently swallow that error message (which is BAD), so you try to write on a socket closed by the peer, which causes the connection to be aborted. And finally on next recv you get the error that you see.
What you should have learned from that:
never ever use a silent try: ... except: ... that swallows any exception without letting you know what has happened. Always limit the filtered exception to the smaller expected set, or (at least and at dev time) display the caught exception to make sure you have not inadvertently caught an unwanted one.
use a higher level protocol to delimit messages on TCP
Now for the fixes:
retriever.py
...
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(stringToBytes("GET key"+"="+key))
s.shutdown(socket.SHUT_WR) # signals the end of the message
print("DEBUG 1")
...
server.py
...
s.listen()
conn, addr = s.accept()
with conn:
# read until the end of a message signaled by a 0 size packet
dataBytes = b''
while True:
b = conn.recv(1024)
if len(b) == 0:
break
dataBytes += b
dataString = bytesToString(dataBytes)
try:
retrievedValueBytes = stringToBytes(bank[cleanGET(dataString)])
print(f"Value: {retrievedValueBytes} has been retrieved with Key {cleanGET(dataString)}")
conn.sendall(retrievedValueBytes)
except Exception as e:
print(e)
print("SERVER: Errored out")
conn.sendall(stringToBytes("ERROR! Value retrieval for command: " + dataString + " doesn't exist!"))
...
Here the thread stops after one single message. If you want the server to be able to accept many clients, you can loop over accept:
s.listen()
while True:
conn, addr = s.accept()
...
Your server is coded to listen in a loop (while True / conn.recv), but your
client only listens once, then exits, which closes the connection.
So the message on the server side is actually not vague at all, it's perfectly to
the point: it's telling you that the connection has been closed. It has indeed,
by the client.
(To be very specific, your retriever code uses a socket as a context manager,
meaning that when you exit the 'with' bock, the socket gets automatically
closed - you can remove the s.close() call, BTW)
Your code works fine, you should be getting the data back (I did, but your post
doesn't say anything about that). The only thing missing is the exception
handling on the server side to gracefully handle the client disconnection.
I currently have a server running under this code:
import socket
port = 60000
s = socket.socket()
host = socket.gethostname()
s.bind((host, port))
s.listen(1)
print ('Server listening....')
while True:
conn, addr = s.accept()
print ('Got connection from', addr)
data = conn.recv(1024)
print('Server received', repr(data))
filename='mytext.txt'
f = open(filename,'rb')
l = f.read(1024)
while (l):
conn.send(l)
print('Sent ',repr(l))
l = f.read(1024)
f.close()
print('Done sending')
conn.send('Thank you for connecting')
conn.close()
And a client running under this code:
import socket
s = socket.socket()
host = socket.gethostname()
port = 60000
s.connect((host, port))
s.send('Test')
with open('received_file', 'wb') as f:
print ('file opened')
while True:
print('receiving data...')
data = s.recv(1024)
print('data=%s', (data))
if not data:
break
# write data to a file
f.write(data)
f.close()
print('Successfully get the file')
s.close()
print('connection closed')
The problem is that the when the client attempts to send the data it says that the data sent must be a byte-type object (Error code shown here: https://gyazo.com/97ef155f6809a801b02f381670895a2b.) I have searched the internet for an answer of how to create a byte type object inside of Python 3, but I have failed to find anything that has worked.
If you’re just sending literals, you can just use a bytes literal instead of a str literal, by prefixing it with a b. Instead of this:
sock.send('Hello')
… just do this:
sock.send(b'Hello')
If you’re sending string variables, you will want to encode them for sending, and probably decode them on receiving. Instead of this:
sock.send(msg)
resp = sock.recv(4096)
… do:
sock.send(msg.encode('ascii')
resp = sock.recv(4096).decode('ascii')
If you want to send non-ASCII strings, you’ll need to pick an encoding. Unless you have a good reason to do otherwise, use UTF-8.
One more thing to keep in mind: TCP sockets are just streams of bytes, not messages, and they can be split up arbitrarily into packets. When you can recv(1024), you may only get part of what the other side sent with send. That’s probably obvious if you’re sending strings longer than 1024 bytes, but even for shorter strings, it can happen. Or, if you’re not strictly alternating sends and receives, you can get multiple sends concatenate into a single receive. And if you send hello and then world and receive them as helloworld you have no way to know that happened, or how to split them back up.
And, worse, it probably won’t happen when you’re testing on localhost on an idle machine, so things will look good during development, but then fail mysteriously all over the place once you deploy it somewhere.
So, you need some protocol that describes where one message ends and another begins. I have a blog post that explains some options, and shows how to implement them.
But if you’re just sending human-readable strings as messages, and those strings will never contain newlines, the simplest protocol is just a line for every message, the same way you’d write a text file. And look at socket.makefile: it gives you something that acts just like a file, handles the one line for every message protocol, and also automatically handles the encode and decode parts. So, instead of this:
sock.send(msg.encode('utf-8'))
resp = sock.recv(1024).decode('utf-8')
… you just do use makefile to get readable and writable file objects, then do:
wf.write(msg + '\n')
resp = next(rf)
… and then you don’t have to worry about how to buffer up the receives and split them into messages.
Change the client code line:
s.send('Test')
to:
s.send(b'Test')
Find more details about socket.send() at https://docs.python.org/3/library/socket.html.
Hi I'm trying to encrypt data sent over a socket in Python (using cryptography.fernet), but whenever I run the code to send the data back, I get a cryptography.fernet.invalidtoken exception on the server end. I can decrypt the tokens with the same key locally, so I think it must be the socket. Here's the code I'm using:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
print colored("listening...","red")
s.listen(10)
conn,addr = s.accept()
print colored("connected!","red")
data = conn.recv(1024)
print data
Sorry I didn't give the encryption function or exception. I was in a hurry to get this out. Anyway, I've found the solution. The issue was that the buffer defined in conn.recv() was too small and the token received was being truncated. Increasing the buffer size to a value larger than the original 1024 fixed the issue.
I wrote a python client to communicate with server side. Each time when I finished sanding out data, I have to call sock.shutdown(socket.SHUT_WR), otherwise the server would not do any response. But after calling sock.shutdown(socket.SHUT_WR), I have to reconnect the connection as sock.connect((HOST, PORT)), other wise I can not send data to server. So how can I keep the connection alive without close it.
My sample code as following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
sock.sendall(data)
sock.shutdown(socket.SHUT_WR)
received = sock.recv(1024)
while len(received)>0:
received = sock.recv(1024)
sock.sendall(newdata) # this would throw exception
The Server Side code as following:
def handle(self):
cur_thread = threading.current_thread()
while True:
self.data = self.rfile.read(bufsiz=100)
if not self.data:
print 'receive none!'
break
try:
data = self.data
print 'Received data, length: %d' % len(data)
self.wfile.write('get received data\n')
except Exception:
print 'exception!!'
You didn't show any server side code but I suspect it simply reads bytes until it gets none anymore.
You can't do this as you found out, because then the only way to tell the server the message is complete is by killing the connection.
Instead you'll have to add some form of framing in your protocol. Possible approaches include a designated stop character that the server recognises (such as a single newline character, or perhaps a 0-byte), sending frames in fixed sizes that your client and server agree upon, or send the frame size first as a network encoded integer followed by exactly the specified number of bytes. The server then first reads the integer and then exactly that same number of bytes from the socket.
That way you can leave the connection open and send multiple messages.
I have just started learning python network programming. I was reading Foundations of Python Network Programming and could not understand the use of s.shutdown(socket.SHUT_WR) where s is a socket object.
Here is the code(where sys.argv[2] is the number of bytes user wants to send, which is rounded off to a multiple of 16) in which it is used:
import socket, sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = '127.0.0.1'
PORT = 1060
if sys.argv[1:] == ['server']:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while True:
print 'Listening at', s.getsockname()
sc, sockname = s.accept()
print 'Processing up to 1024 bytes at a time from', sockname
n = 0
while True:
message = sc.recv(1024)
if not message:
break
sc.sendall(message.upper()) # send it back uppercase
n += len(message)
print '\r%d bytes processed so far' % (n,),
sys.stdout.flush()
print
sc.close()
print 'Completed processing'
elif len(sys.argv) == 3 and sys.argv[1] == 'client' and sys.argv[2].isdigit():
bytes = (int(sys.argv[2]) + 15) // 16 * 16 # round up to // 16
message = 'capitalize this!' # 16-byte message to repeat over and over
print 'Sending', bytes, 'bytes of data, in chunks of 16 bytes'
s.connect((HOST, PORT))
sent = 0
while sent < bytes:
s.sendall(message)
sent += len(message)
print '\r%d bytes sent' % (sent,),
sys.stdout.flush()
print
s.shutdown(socket.SHUT_WR)
print 'Receiving all the data the server sends back'
received = 0
while True:
data = s.recv(42)
if not received:
print 'The first data received says', repr(data)
received += len(data)
if not data:
break
print '\r%d bytes received' % (received,),
s.close()
else:
print >>sys.stderr, 'usage: tcp_deadlock.py server | client <bytes>'
And this is the explanation that the author provides which I am finding hard to understand:
Second, you will see that the client makes a shutdown() call on the socket after it finishes sending its transmission. This solves an important problem: if the server is going to read forever until it sees end-of-file, then how will the client avoid having to do a full close() on the socket and thus forbid itself from doing the many recv() calls that it still needs to make to receive the server’s response? The solution is to “half-close” the socket—that is, to permanently shut down communication in one direction but without destroying the socket itself—so that the server can no longer read any data, but can still send any remaining reply back in the other direction, which will still be open.
My understanding of what it will do is that it will prevent the client application from further sending the data and thus will also prevent the server side from further attempting to read any data.
What I cant understand is that why is it used in this program and in what situations should I consider using it in my programs?
My understanding of what it will do is that it will prevent the client
application from further sending the data and thus will also prevent
the server side from further attempting to read any data.
Your understanding is correct.
What I cant understand is that why is it used in this program …
As your own statement suggests, without the client's s.shutdown(socket.SHUT_WR) the server would not quit waiting for data, but instead stick in its sc.recv(1024) forever, because there would be no connection termination request sent to the server.
Since the server then would never get to its sc.close(), the client on his part also would not quit waiting for data, but instead stick in its s.recv(42) forever, because there would be no connection termination request sent from the server.
Reading this answer to "close vs shutdown socket?" might also be enlightening.
The explanation is half-baked, it applies only to this specific code and overall I would vote with all-fours that this is bad practice.
Now to understand why is it so, you need to look at a server code. This server works by blocking execution until it receives 1024 bytes. Upon reception it processes the data (makes it upper-case) and sends it back. Now the problem is with hardcoded value of 1024. What if your string is shorter than 1024 bytes?
To resolve this you need to tell the server that - hey there is no more data coming your way, so return from message = sc.recv(1024) and you do this by shutting down the socket in one direction.
You do not want to fully close the socket, because then the server would not be able to send you the reply.