I would like to send the location of a moving point to a server via TCP with the socket module. That point location is updated at each iteration of a for loop and is sent in the form of a tuple (x, y) that has been serialized with pickle dumps methods.
Problem:
On the server side, it seems that I only get to receive the location from the first iteration of that loop. As if all the following updated positions had been skipped or lost in the process.
I can’t say for sure what is the reason behind this behavior but my bet is that I am not correctly setting things on the server side. I suspect the data to be sent entirely but not processed adequately on reception due to some mistakes that I am probably doing with the socket module (I am completely new to the world of network interfaces).
Code:
--client side--
#Python3.7
import socket
import pickle
import math
HOST = "127.0.0.1"
PORT = 12000
den = 20
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(1000):
i = step%den
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = pickle.dumps((x, y), protocol=0)
sock.sendall(data)
--server side--
#Jython2.7
import pickle
import socket
HOST = "127.0.0.1"
PORT = 12000
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
while True:
connection, address = s.accept()
if connection:
data = connection.recv(4096)
print(pickle.loads(data)) # <-- only print once (first location)
You need to put connection, address = s.accept() outside the while loop otherwise your server will wait for a new connection every time.
You also have an issue with the way your are receiving data. connection.recv(4096) will return any amount of bytes between 0 and 4096 not every time a complete "data" message is received. To handle this you could send a header before sending you json indicating how much data should be received
By adding a header, you will make sure the data messages you are sending will be received properly.
The header in this example is a four bytes int indicating the size of data.
Server
import pickle
import socket
import struct
HEADER_SIZE = 4
HOST = "127.0.0.1"
PORT = 12000
def receive(nb_bytes, conn):
# Ensure that exactly the desired amount of bytes is received
received = bytearray()
while len(received) < nb_bytes:
received += conn.recv(nb_bytes - len(received))
return received
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
connection, address = s.accept()
while True:
# receive header
header = receive(HEADER_SIZE, connection)
data_size = struct.unpack(">i", header)[0]
# receive data
data = receive(data_size, connection)
print(pickle.loads(data))
Client
import socket
import pickle
import math
HEADER_SIZE = 4
HOST = "127.0.0.1"
PORT = 12000
den = 20
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(1000):
i = step%den
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = pickle.dumps((x, y), protocol=0)
# compute header by taking the byte representation of the int
header = len(data).to_bytes(HEADER_SIZE, byteorder ='big')
sock.sendall(header + data)
Hope it helps
Related
I made a very simple server and client program to test. I need to send 3 different lists from server and client must recieve them seperately. Could you please help me ?
Here is server.py
import socket
import pickle
am0=['AQ-20', 'A3000', 'AQ-26', 'A5000', 'AQ-33', 'A5000pro', 'AQ-33pro']
am1=['A10000Pro', 'AQ-43', 'AX-48', 'AX-58', 'AX-68']
am2=['Material', 'nan', 'Steel', 'Stainless S.', 'Stainless S. 1.403']
am00=pickle.dumps(am0)
am01=pickle.dumps(am1)
am02=pickle.dumps(am2)
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.bind(('127.0.0.1', 1))
buffer_size = 1024
mysocket.listen()
(client, (ip,port)) = mysocket.accept()
client.send(am00)
#----------------HOW CAN I ADD THEM:----------------
#client.send(am01)
#client.send(am02)
mysocket.close()
here is client.py
import socket
import pickle
host = '127.0.0.1'
port = 1
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
#----------------HOW CAN I RECEIVE THEM SEPERATELY ?----------------
data1 = pickle.loads(s.recv(buffer_size))
print(data1)
the pickle protocol includes the length of the pickled buffer and you can use pickle.Unpickler to load one pickled object at a time. The problem is that it wants a file-like object, not a socket. Fortunately, sockets can make themselves look like files using socket.makefile. There are caveats, so reading the referenced doc is worthwhile.
Update your server to write each pickled object and then change the client to
import socket
import pickle
host = '127.0.0.1'
port = 8899
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
unpickler = pickle.Unpickler(s.makefile("rb"))
print(unpickler.load())
print(unpickler.load())
print(unpickler.load())
Here is a more universal method that also works with non pickled data (eg. receiving data from c socket), you first send in the data len and then the actual data in series to the socket, note that the while loop is there because you are not guaranteed to receive the specify number of bytes in a single call, if the data hasn't arrived yet you will receive less. struct.pack packs data you provided to the function into binary data in the order provided by first argument, 'H' represents unsigned short which is 2 bytes long (max integer you can pack into that is 65535)
Server:
import socket
import pickle, struct
am0=['AQ-20', 'A3000', 'AQ-26', 'A5000', 'AQ-33', 'A5000pro', 'AQ-33pro']
am1=['A10000Pro', 'AQ-43', 'AX-48', 'AX-58', 'AX-68']
am2=['Material', 'nan', 'Steel', 'Stainless S.', 'Stainless S. 1.403']
am00=pickle.dumps(am0)
am00 = struct.pack("H", len(am00)) + am00
am01=pickle.dumps(am1)
am01 = struct.pack("H",len(am01)) +am01
am02=pickle.dumps(am2)
am02 = struct.pack("H", len(am02)) + am02
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.bind(('127.0.0.1', 1))
buffer_size = 1024
mysocket.listen()
(client, (ip,port)) = mysocket.accept()
client.send(am00)
#----------------HOW CAN I ADD THEM:----------------
client.send(am01)
client.send(am02)
mysocket.close()
Client:
import socket
import pickle, struct
host = '127.0.0.1'
port = 1
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
#----------------HOW CAN I RECEIVE THEM SEPERATELY ?---------------
def recv_all(sock:socket.socket):
data_len = 0
buffer = b''
while len(buffer) < 2:
buffer += sock.recv(2-len(buffer))
data_len = struct.unpack("H", buffer)[0]
buffer = b''
while len(buffer) < data_len:
buffer += sock.recv(data_len-len(buffer))
return buffer
data1 = pickle.loads(recv_all(s))
print(data1)
data1 = pickle.loads(recv_all(s))
print(data1)
data1 = pickle.loads(recv_all(s))
print(data1)
I would like to read one by one the objects coming from a TCP stream, preferably using the MessagePack library.
On one side I have a client that, at each iteration of a loop:
computes a point location (tuple)
packs that location and sends it through a socket
On the other side, a server that:
receive the data when client is detected
unpack that data
For now I am storing the data in a buffer on reception, then proceed to unpacking when the stream is over. My problem is that I need to unpack the tuples one by one as they are sent. In other words I would like to read the data in real time without putting it in a buffer.
Provided it is possible, how could I achieve this using MessagePack ?
-- client side --
#Python3.7
import socket
import msgpack
import math
HOST = "127.0.0.1"
PORT = 9000
den = 40
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(den):
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = msgpack.packb((x, y), use_bin_type = True)
sock.sendall(data)
-- server side --
#Jython2.7 <-- Python 2.7 compatible only
from io import BytesIO
import msgpack
import socket
HOST = "127.0.0.1"
PORT = 9000
buf = BytesIO()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
connection, address = s.accept()
while True:
try:
data = connection.recv(1024)
buf.write(data)
except:
buf.seek(0)
unpacker = msgpack.Unpacker(buf, use_list=False, raw=False)
for unpacked in unpacker:
print(unpacked)
buf = BytesIO()
See "Stream Unpacking" section in the README:
https://github.com/msgpack/msgpack-python#streaming-unpacking
You can do like this:
unpacker = msgpack.Unpacker(use_list=False, raw=False)
while True:
data = connection.recv(1024)
if not data:
break
unpacker.feed(data)
for unpacked in unpacker:
print(unpacked)
The problem I'm having is to get a file from the server to client across devices. Everything works fine on localhost.
Lets say I want to "get ./testing.pdf" which sends the pdf from the server to the client. It sends but it is always missing bytes. Is there any problems with how I am sending the data. If so how can I fix it? I left out the code for my other functionalities since they are not used for this function.
sending a txt file with "hello" in it works perfectly
server.py
import socket, os, subprocess # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
#host = ''
port = 5000 # Reserve a port for your service.
bufsize = 4096
s.bind((host, port)) # Bind to the port
s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
while True:
userInput = c.recv(1024)
.... CODE ABOUT OTHER FUNCTIONALITY
elif userInput.split(" ")[0] == "get":
print "inputed get"
somefile = userInput.split(" ")[1]
size = os.stat(somefile).st_size
print size
c.send(str(size))
bytes = open(somefile).read()
c.send(bytes)
print c.recv(1024)
c.close()
client.py
import socket, os # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
#host = '192.168.0.18'
port = 5000 # Reserve a port for your service.
bufsize = 1
s.connect((host, port))
print s.recv(1024)
print "Welcome to the server :)"
while 1 < 2:
userInput = raw_input()
.... CODE ABOUT OTHER FUNCTIONALITY
elif userInput.split(" ")[0] == "get":
print "inputed get"
s.send(userInput)
fName = os.path.basename(userInput.split(" ")[1])
myfile = open(fName, 'w')
size = s.recv(1024)
size = int(size)
data = ""
while True:
data += s.recv(bufsize)
size -= bufsize
if size < 0: break
print 'writing file .... %d' % size
myfile = open('Testing.pdf', 'w')
myfile.write(data)
myfile.close()
s.send('success')
s.close
I can see two problems right away. I don't know if these are the problems you are having, but they are problems. Both of them relate to the fact that TCP is a byte stream, not a packet stream. That is, recv calls do not necessarily match one-for-one with the send calls.
size = s.recv(1024) It is possible that this recv could return only some of the size digits. It is also possible that this recv could return all of the size digits plus some of the data. I'll leave it for you to fix this case.
data += s.recv(bufsize) / size -= bufsize There is no guarantee that that the recv call returns bufsize bytes. It may return a buffer much smaller than bufsize. The fix for this case is simple: datum = s.recv(bufsize) / size -= len(datum) / data += datum.
I have implemented a small benchmark for socket communication via TCP and UDP in Python. Surprisingly, TCP is almost exactly double as fast as UDP.
To avoid routing effects, server and client are running on the same Unix machine, but on different threads.
Maybe the code is useful. Here is the server code:
import socket
import sys
host = 'localhost'
port = 8888
buffersize = 8
server_address = (host, port)
def start_UDP_server():
socket_UDP = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
socket_UDP.bind(server_address)
print("UDP server is running...")
while True:
data, from_address = socket_UDP.recvfrom(buffersize)
if not data: break
socket_UDP.sendto(data, from_address)
socket_UDP.close()
def start_TCP_server():
socket_TCP = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket_TCP.bind(server_address)
socket_TCP.listen(1)
print("TCP server is running...")
while True:
client, client_address = socket_TCP.accept()
while True:
data = client.recv(buffersize)
if not data: break
client.sendall(data)
client.close()
So you can run either start_TCP_server() or start_UDP_server().
On client side the code is:
import socket
import sys
import time
host = 'localhost'
port = 8888
buffersize = 8
server_address = (host, port)
client_address = (host, port+1)
N = 1000000
def benchmark_UDP():
socket_UDP = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
socket_UDP.bind(client_address)
print("Benchmark UDP...")
duration = 0.0
for i in range(0, N):
b = bytes("a"*buffersize, "utf-8")
start = time.time()
socket_UDP.sendto(b, server_address)
data, from_address = socket_UDP.recvfrom(buffersize)
duration += time.time() - start
if data != b:
print("Error: Sent and received data are bot the same")
print(duration*pow(10, 6)/N, "µs for UDP")
def benchmark_TCP():
socket_TCP = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket_TCP.connect(server_address)
print("Benchmark TCP...")
duration = 0.0
for i in range(0, N):
b = bytes("a"*buffersize, "utf-8")
start = time.time()
socket_TCP.sendall(b)
data = socket_TCP.recv(buffersize)
duration += time.time() - start
if data != b:
print("Error: Sent and received data are bot the same")
print(duration*pow(10, 6)/N, "µs for TCP")
socket_TCP.close()
Like for the server you can start the benchmark by benchmark_TCP() or benchmark_UDP().
The results are about 25 µs for TCP, and about 54 µs for UDP on Unix and even worse for Windows (about 30 µs for TCP and more than 200 µs for UDP). Why? I would expect a minimal advantage for UDP.
Your TCP socket is connected but your UDP socket is not. This means extra processing for every send/receive on the UDP socket. Call connect on each side for the UDP socket, just like you call connect/accept on the TCP socket.
Programs like iperf do this to measure accurately.
Trying to make a server that tells the client what number he/she is. For example, once you connect it should say something like "Welcome client #5" or something along those lines. Right now I'm just trying to write it so that it simply reads a line in and echos it back. Im stuck on as far as getting it to show the clients number.
import socket
import sys
host = ''
port = 37373
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(1)
while 1:
s, address = s.accept()
data = s.recv(1024)
if data:
s.send(data)
s.close()
that is
import socket
import sys
Clinet_number = 0
host = ''
port = 37373
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(10) # number of queued connections
while 1:
Client_number += 1
s, address = s.accept()
data = s.recv(1024)
if data:
s.send(str(Client_number))
s.close()