Develop Reference

Recvall with while loop doesn't work between two devices in python

I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)

recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...

WinError 10053. How do I keep sending data over TCP "forever"

I am currently trying to learn networking with python. I am really new to this topic so I replicated some examples from somewhere like here
I want to achieve a continous data transfer with TCP. This means I want to send data as long as some condition is met. So I slightly modified the example to this code below:
My Setup is Win10 with Python 3.8
My client.py copied and modified form above:
# Echo client program
import socket
HOST = '192.168.102.127' # The remote host
PORT = 21
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
i=0 #For counting how often the string was sent
while True: #for testing this is forever
s.sendall(b'Hello, world')
data = s.recv(1024)#
print(i)
i=i+1
print('Received', repr(data))
My server.py:
# Echo server program
import socket
HOST = '' # Symbolic name meaning all available interfaces
PORT = 21
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
with conn:
print('Connected by', addr)
while True:
data = conn.recv(1024)
if not data: break
conn.sendall(data)
The error I am getting is
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
after i=5460 (in multiple tries) on the Client side and
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host on the Server side
The longer my text message is, less messages got sent before the error.
This leads me to believe I sent the data to some sort of buffer which is (over-)written to until the error is thrown.
When looking for possible solutions I only found different implementations, which did not cover my problem or used other software.
As stated in some answers for similar questions, I disabled my firewall and stopped my antivirus, but with no noticable difference.
When looking up the error, there is also the possibilty of protocol errors but I do not expect that to be a problem.
When reading into the socket/TCP documentation, I found somewhere that TCP is not really designed for this kind of problem, but rather for
client connects to server
|
V
client sends request to server
|
V
server sends request answer
|
V
server closes connection.
Is this really true?
But I cannot believe that for every data that is sent a new socket must be connected, like in this question. This solution is also really slow.
But if this is the case, what could I use alternatively?
To illustrate the bigger picture:
I have a some other code which is giving me status data (text) at 500Hz. In Python, I am processing this data and sending the processed data to an Arduino with Ethernet shield. This data is "realtime" data, so I need the data sent to the arduino as fast as possible. Here the client is Python and the Server is the Arduino with the Ethernet module. The connection and everthing is working fine, only the continous sending of data is my problem.

Python TCP sockets headers

I need to create a communication between a client and a server with TCP. But I'd like to send and work with "headers". So from the client I'd like to send a header "COMMAND1" and the server returns me something.
I have the following code:
Server
import socket
import threading
bind_ip = '0.0.0.0'
bind_port = 9998
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(5) # max backlog of connections
print ('Listening on {}:{}'.format(bind_ip, bind_port))
def handle_client_connection(client_socket):
request = client_socket.recv(1024)
print ('Received {}'.format(request))
client_socket.send('Response1!'.encode('utf-8'))
client_socket.close()
while True:
client_sock, address = server.accept()
print ('Accepted connection from {}:{}'.format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,) # without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
Client
import socket
hostname, sld, tld, port = 'www', 'integralist', 'co.uk', 80
target = '{}.{}.{}'.format(hostname, sld, tld)
# create an ipv4 (AF_INET) socket object using the tcp protocol (SOCK_STREAM)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect the client
# client.connect((target, port))
client.connect(('0.0.0.0', 9998))
# send some data (in this case a HTTP GET request)
client.send('hi'.encode('utf-8'))
# receive the response data (4096 is recommended buffer size)
response = client.recv(4096)
print (response)
Anyone knows the best way to return "Response1!" when the header is "COMMAND1" and "Response2!" when the header is "COMMAND2"?
I can't find examples on how to use headers
EDIT: It doesn't have to be "COMMAND1" or "COMMAND2" it can be a "0" or "1", or anything else.

If you want to add your own header, you just have to:
Make sure your programm finds the start of your message (like, every message beginns "!?&")
Send your own header-data just after the start-symbol of your message.
Maybe mark the end of your message with something or pass a length in your header.
Since TCP will give you a stream of data, it might come to a case, where it just gives you 2 or 3 messages at once. You have to separate these messages by yourself (e.g. by using "?!&" as start of every message).
You can always create your own protocoll as payload of another protocoll. Just as TCP is just payload from the ethernet point of view.

You can do something i have done with my program to accept such headers
Use pickle library to encode a dict headers and send it through socket.
Code will look something like this.
import pickle
def handleSocket(headers:dict):
message = pickle.dumps(headers)
socket.send(message)
For server side, you will be handling it
Gonna initialise the socket recv to 100 kb
def handleReceivedSocket(socket):
message:dict = pickle.loads(socket.recv(102400))
Another way to do this. Is sending a raw json string to the server (just change pickle.dumps,pickle.loads by json.dumps,json.loads
But it will be in raw and less secure.
Last way you can do it is uri encoding. Check w3schools

How can I continually send data without shutdown socket connection in python

I wrote a python client to communicate with server side. Each time when I finished sanding out data, I have to call sock.shutdown(socket.SHUT_WR), otherwise the server would not do any response. But after calling sock.shutdown(socket.SHUT_WR), I have to reconnect the connection as sock.connect((HOST, PORT)), other wise I can not send data to server. So how can I keep the connection alive without close it.
My sample code as following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
sock.sendall(data)
sock.shutdown(socket.SHUT_WR)
received = sock.recv(1024)
while len(received)>0:
received = sock.recv(1024)
sock.sendall(newdata) # this would throw exception
The Server Side code as following:
def handle(self):
cur_thread = threading.current_thread()
while True:
self.data = self.rfile.read(bufsiz=100)
if not self.data:
print 'receive none!'
break
try:
data = self.data
print 'Received data, length: %d' % len(data)
self.wfile.write('get received data\n')
except Exception:
print 'exception!!'

You didn't show any server side code but I suspect it simply reads bytes until it gets none anymore.
You can't do this as you found out, because then the only way to tell the server the message is complete is by killing the connection.
Instead you'll have to add some form of framing in your protocol. Possible approaches include a designated stop character that the server recognises (such as a single newline character, or perhaps a 0-byte), sending frames in fixed sizes that your client and server agree upon, or send the frame size first as a network encoded integer followed by exactly the specified number of bytes. The server then first reads the integer and then exactly that same number of bytes from the socket.
That way you can leave the connection open and send multiple messages.

Receive all of the data when using python socket

I am using socket to connect to a simple server to receive some data:
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
host = "X.X.X.X"
port = Y
s.connect((host,port))
data = s.recv(512)
print data
Not all of the expected received data is received; some of it is cut off. So any operations that I need to perform on the received data is thrown off.
I've tried using s.settimeout(1) and putting the socket in non-blocking mode. Strangely, it works just fine in ipython.

Even if 512 bytes are sent from the server to the client, if they're sent through multiple packets, and that one packet has TCP Push flag set, the kernel will stop buffering and hand over the data to userland, ie, your application.
You should loop on data = s.recv(512) and print data. I'd even remove the '512', which is useless.
while True:
if not data: break
data = s.recv()
print data