Develop Reference

Python UDP socket; still receiving data when source is stopped; why?

I am sending 32 bytes packets every 1ms to this socket. I wish to print the data after every 40 ms. And apparently the code does that. But even when I stop sending data, I still continue to see that data is being printed.
Is it holding the data in some cache? or simply the python socket has a huge delay? Why?
The code is as follows:
## Import necessary libraries
import math
import numpy as np
import socket
import struct
import time
from synchrophasor.frame import CommandFrame
from datetime import datetime
## Configure socket for Phasor data ##
UDP_IP = "10.10.114.22"
UDP_PORT = 8208 #UDP phasor values 32 bytes (V,phi,P)
sock_ph = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock_ph.bind((UDP_IP, UDP_PORT))
print("socket bound, waiting for data...")
while True:
raw = sock_ph.recv(32)
#print(raw)
mag = struct.unpack('d', raw[8:16])[0]
# print("mag =",mag,type(mag))
angle = struct.unpack('d', raw[16:24])[0]
# print("angle =",angle,type(angle))
header = struct.unpack('d', raw[0:8])[0]
# print("header =",header,type(header))
phasor = (mag, angle)
Vol_A=raw
VA = float(mag)
phi_A = float(angle)
VB = VA
phi_B = phi_A+(math.pi) * 2 / 3
VC = VA
phi_C = phi_A-(math.pi) * 2 / 3
time.sleep(1/25)
# pmu.send_data(phasors=[(VA,phi_A),(VB,phi_B),(VC,phi_C)],analog=[9.91],digital=[0x0001])
#time.sleep(1/config_rr)
print([(VA,phi_A),(VB,phi_B),(VC,phi_C),datetime.now()])

most programs don't want to discard unread datagrams so most OSs will buffer them for you. your case is somewhat unusual so you'd need to write code to handle this case. I'd change your code to do something like:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('', 8208))
# block until we read an initial packet
raw = s.recv(1024)
s.setblocking(False)
while True:
# unpack
header, mag, angle = struct.unpack('ddd', raw)
# do something with data
print(f'header={header} mag={mag} angle={angle}')
# sleep for some time
time.sleep(1/25)
# discard any packets you've received in the mean time
while True:
try:
raw = s.recv(1024)
except OSError as err:
# OS buffer is empty: we've therefore got the most recent data
if err.errno == socket.EWOULDBLOCK:
break
# something else failing, reraise the error
raise
note that Steffen Ullrich's suggestion of sending the data at the correct rate would be easier, but assumes that you have control over the sending process. the fact that you said "I am sending" suggests you do, and so would likely make a better solution

How to receive and handle multiple TCP stream sockets?

I would like to send the location of a moving point to a server via TCP with the socket module. That point location is updated at each iteration of a for loop and is sent in the form of a tuple (x, y) that has been serialized with pickle dumps methods.
Problem:
On the server side, it seems that I only get to receive the location from the first iteration of that loop. As if all the following updated positions had been skipped or lost in the process.
I can’t say for sure what is the reason behind this behavior but my bet is that I am not correctly setting things on the server side. I suspect the data to be sent entirely but not processed adequately on reception due to some mistakes that I am probably doing with the socket module (I am completely new to the world of network interfaces).
Code:
--client side--
#Python3.7
import socket
import pickle
import math
HOST = "127.0.0.1"
PORT = 12000
den = 20
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(1000):
i = step%den
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = pickle.dumps((x, y), protocol=0)
sock.sendall(data)
--server side--
#Jython2.7
import pickle
import socket
HOST = "127.0.0.1"
PORT = 12000
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
while True:
connection, address = s.accept()
if connection:
data = connection.recv(4096)
print(pickle.loads(data)) # <-- only print once (first location)

You need to put connection, address = s.accept() outside the while loop otherwise your server will wait for a new connection every time.
You also have an issue with the way your are receiving data. connection.recv(4096) will return any amount of bytes between 0 and 4096 not every time a complete "data" message is received. To handle this you could send a header before sending you json indicating how much data should be received
By adding a header, you will make sure the data messages you are sending will be received properly.
The header in this example is a four bytes int indicating the size of data.
Server
import pickle
import socket
import struct
HEADER_SIZE = 4
HOST = "127.0.0.1"
PORT = 12000
def receive(nb_bytes, conn):
# Ensure that exactly the desired amount of bytes is received
received = bytearray()
while len(received) < nb_bytes:
received += conn.recv(nb_bytes - len(received))
return received
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
connection, address = s.accept()
while True:
# receive header
header = receive(HEADER_SIZE, connection)
data_size = struct.unpack(">i", header)[0]
# receive data
data = receive(data_size, connection)
print(pickle.loads(data))
Client
import socket
import pickle
import math
HEADER_SIZE = 4
HOST = "127.0.0.1"
PORT = 12000
den = 20
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(1000):
i = step%den
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = pickle.dumps((x, y), protocol=0)
# compute header by taking the byte representation of the int
header = len(data).to_bytes(HEADER_SIZE, byteorder ='big')
sock.sendall(header + data)
Hope it helps

Socket Programming; File corrupts when transferring over multiple devices

The problem I'm having is to get a file from the server to client across devices. Everything works fine on localhost.
Lets say I want to "get ./testing.pdf" which sends the pdf from the server to the client. It sends but it is always missing bytes. Is there any problems with how I am sending the data. If so how can I fix it? I left out the code for my other functionalities since they are not used for this function.
sending a txt file with "hello" in it works perfectly
server.py
import socket, os, subprocess # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
#host = ''
port = 5000 # Reserve a port for your service.
bufsize = 4096
s.bind((host, port)) # Bind to the port
s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
while True:
userInput = c.recv(1024)
.... CODE ABOUT OTHER FUNCTIONALITY
elif userInput.split(" ")[0] == "get":
print "inputed get"
somefile = userInput.split(" ")[1]
size = os.stat(somefile).st_size
print size
c.send(str(size))
bytes = open(somefile).read()
c.send(bytes)
print c.recv(1024)
c.close()
client.py
import socket, os # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
#host = '192.168.0.18'
port = 5000 # Reserve a port for your service.
bufsize = 1
s.connect((host, port))
print s.recv(1024)
print "Welcome to the server :)"
while 1 < 2:
userInput = raw_input()
.... CODE ABOUT OTHER FUNCTIONALITY
elif userInput.split(" ")[0] == "get":
print "inputed get"
s.send(userInput)
fName = os.path.basename(userInput.split(" ")[1])
myfile = open(fName, 'w')
size = s.recv(1024)
size = int(size)
data = ""
while True:
data += s.recv(bufsize)
size -= bufsize
if size < 0: break
print 'writing file .... %d' % size
myfile = open('Testing.pdf', 'w')
myfile.write(data)
myfile.close()
s.send('success')
s.close

I can see two problems right away. I don't know if these are the problems you are having, but they are problems. Both of them relate to the fact that TCP is a byte stream, not a packet stream. That is, recv calls do not necessarily match one-for-one with the send calls.
size = s.recv(1024) It is possible that this recv could return only some of the size digits. It is also possible that this recv could return all of the size digits plus some of the data. I'll leave it for you to fix this case.
data += s.recv(bufsize) / size -= bufsize There is no guarantee that that the recv call returns bufsize bytes. It may return a buffer much smaller than bufsize. The fix for this case is simple: datum = s.recv(bufsize) / size -= len(datum) / data += datum.

how to send an array over a socket in python

I have an array kind of ([1,2,3,4,5,6],[1,2,3,4,5,6]) this. I have to send it over a STREAM/TCP socket in python. Then I have to receive the same array at the receiving end.

Sockets are byte streams, so ideal is to write your protocol (read this)
This is a basic example without protocol and you should care about buffer -> recv(). If it is too small, your data will be chopped off. That's why you should implement a protocol, if you send unknown size of data.
Client:
import socket, pickle
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
arr = ([1,2,3,4,5,6],[1,2,3,4,5,6])
data_string = pickle.dumps(arr)
s.send(data_string)
data = s.recv(4096)
data_arr = pickle.loads(data)
s.close()
print 'Received', repr(data_arr)
Server:
import socket
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(4096)
if not data: break
conn.send(data)
conn.close()

You can Serialize the object before sending to socket and at receiving end Deserialize it. Check this

I solved this problem using json (since I heard pickle is unsafe)
client:
import json
...
arr1 = [1,2,3]
arr2 = [4,5,6]
someVar = 7
data = json.dumps({"a": arr1, "b": arr2, "c": someVar})
socket.send(data.encode())
server:
import json
...
data = socket.recv(1024)
data = json.loads(data.decode())
arr = data.get("a")
var = data.get("c")
Here we deserialize the json string, using data.get("a") which you can interpret as data.a

I solved this issue by going over each item in the array, adding it to a single string, but with a significant character, such as a greek letter or some other uncommon character, then sending that string over the socket, then splitting the recieved string up back into an array on the other side, and removing all of the 'delimiter' items in the new array.
For Example, The Client side
for item in myArray:
print("item: ", item)
myArrayString= myArrayString+ str(item) + "Δ"
print(myArrayString)
myServer.send((myArrayString).encode())
and then on the Server:
files = myconnection.recv(50000)
files = files.decode()
myArray = files.split('Δ')
for myItem in myArray:
print(myItem)
print("End Of Items in Array")
hope this helps! feel free to ask if you need anything clarified! : )

IPID field unexpected increase

Intro:
I have an exercise where I need to send a file from S to D through a third party T only.
T is run on port 10000 or 11000 depending if i use UDP or TCP; i use UDP, and T's ip is given both to S and D.
T is given and all it does is echo messages it got back to the sender.
One of the fields in the header of T's echo message is called ip_id - a counter that goes up by one with each message T receives; in order to view the ip_id value i need to use raw sockets.
S/D sends a message to T and is then supposed to receive back a message. S/D needs to check that the ip and port that it got the message from matches the one it sent to (that is, if S/D sends a message to 1.1.1.1:5 it should receive a message from 1.1.1.1:5).
First I open a socket and a raw socket once
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.raw_socket = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_UDP)
I then get my port by doing:
self.s.sendto("1", (self.remote_server, UDP_PORT))
self.myPort = self.s.getsockname()[1]
UDP_PORT is 10000 and is the port to send messages to.
remote_server is the ip to send messages to.
I also have the following method:
def send_value(self, number_of_packets):
for i in range(0,number_of_packets):
self.s.sendto(BOGUS_DATA, (self.remote_server, UDP_PORT))
self.raw_socket.recvfrom(1024)
#self.raw_socket.recv(1024)
And as well as this one:
def recieve_ip_id(self):
try:
sent = 0
continueFlag = True
while continueFlag:
self.s.sendto(BOGUS_DATA, (self.remote_server, UDP_PORT))
#mypacket = self.raw_socket.recv(1024)
mypacket = self.raw_socket.recvfrom(1024)
mypacket = mypacket[0]
continueFlag = False
if (256*ord(mypacket[20])+ord(mypacket[21]) != UDP_PORT):
continueFlag = True
if (256*ord(mypacket[22])+ord(mypacket[23]) != self.myPort):
continueFlag = True
sent = sent + 1
ip_id = 256 * ord(mypacket[4]) + ord(mypacket[5])
print "packet 20 21 " + str(256*ord(mypacket[20])+ord(mypacket[21])) + " packet 4 5 " + str(256 * ord(mypacket[4]) + ord(mypacket[5])) + " packet 22 23 " + str(256*ord(mypacket[22])+ord(mypacket[23]))
print ip_id
return (ip_id, 0)
except socket.timeout:
# dummy
return (0, 0)
It works good enough in the sense that it checks correctly who it got the message from; The problem is that the ip_id wont advance as expected. after some time the programs are running they each get ip_id as if it was not shared anymore.
am i not checking something correctly, or is it something else?
Thanks.