Develop Reference

How to detect a closed client for UDP sockets?

I'm transmitting some live data from a camera to a client over ethernet where i'm using a python socket and opencv to transmit the images. It works fine and i can read the video at the client side.
The issue is that when i start the server that listens for users that want the feed and a socket stream is started, then after i close the connection at the client side the server side keeps transmitting the data. Furthermore, after closing the client side connection i can only reconnect by restarting the server.
I've though about implementing a kind of heartbeat protocol by sending a message from client to server for every 100th image i recieve to tell i'm still listening - and if not recieved then go back to the loop to listen for a new connection. The only issue is that the recvfrom is non-blocking so a direct implementation into the while loop would probably not work.
How can i detect that a/the client isn't reading the data anymore?
Here is my server side:
import cv2, imutils, socket
import numpy as np
import time
import base64
BUFF_SIZE = 65536
server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, BUFF_SIZE)
host_name = socket.gethostname()
host_ip = 'xxx.xxx.xxx.xxx'
port = 4001
socket_address = (host_ip, port)
server_socket.bind(socket_address)
print("listening at", socket_address)
vid = cv2.VideoCapture(0)
WIDTH=480
while True:
msg,client_addr = server_socket.recvfrom(BUFF_SIZE)
print(client_addr)
while (vid.isOpened()):
print("sending img")
_,frame = vid.read()
frame = imutils.resize(frame,width=WIDTH)
encoded,buffer = cv2.imencode(".jpg",frame,[cv2.IMWRITE_JPEG_QUALITY,80])
message = base64.b64encode(buffer)
print(len(message))
server_socket.sendto(message,client_addr)
key = cv2.waitKey(1) & 0xFF
if key == ord('q'):
server_socket.close()
Here is my client side:
def video_socket_stream_client():
BUFF_SIZE = 65536
client_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
client_socket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, BUFF_SIZE)
host_name = socket.gethostname()
host_ip = 'xxx.xxx.xxx.xxx'
port = 4001
client_socket.sendto("connect".encode(),(host_ip, port)) # send signal
while True:
packet,_ = client_socket.recvfrom(BUFF_SIZE)
data = base64.b64decode(packet,' /')
npdata = np.frombuffer(data,dtype=np.uint8)
frame = cv2.imdecode(npdata,1)
time.sleep(0.016)
yield(b'--frame\r\n'b'Content-Type: image/jpeg\r\n\r\n' + data + b'\r\n')

This is one of the tradeoffs with UDP vs TCP. With UDP, there is no long-term connection. The sender just blasts data and hopes someone is listening. The listener just picks up the phone and hopes someone is talking. There is no persistent state. If the sender stops sending, the listener will not hear anything. If you need to maintain a connection, then you need to use TCP.
You shouldn't ever have to restart the server. Neither side knows what the other side is doing. I wonder if you're getting bit by not using SO_REUSEADDR.

The running status of the server can be judged by the return value of getattr(socket,'_closed').True is closed state, False is running.
e.g.
import socket
ip = 'localhost'
port = 5003
ws = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ws.bind((ip, port))
ws.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
ws.listen()
# 关闭服务
# ws.close()
print("The status is as follows：")
if(getattr(ws, '_closed') == False):
print("Running")
elif(getattr(ws, '_closed') == True):
print("closed")
You can try.

You could have the client send a "goodbye" type of packet to the server just before the client exits, to let the server know the client is going away, but of course you can't rely on that packet actually making it to the server (the packet might get dropped, or the client might crash or get physically disconnected from the network or otherwise go away in an uncontrolled fashion), so in the end you'll need to implement some kind of heartbeat protocol.
In order to send() and recv() simultaneously, you can use non-blocking I/O and select(), or you can use multiple threads. I prefer the select() approach, but either way can be made to work. You might also consider upgrading your server's logic to support multiple clients simultaneously rather than just one client-at-a-time; even if you don't actually need to send to multiple clients at once, that would allow your server to handle the case where the user quits the client and then starts a new one (i.e. the server would be able to start sending packets to the new client even before it has timed-out the old/dead client).

Troubleshooting hanged UDP socket in a Python server application code

I am writing a UDP server application that serves as a back end to Teltonika FMB630 car mounted devices.
I already took care of the protocol specifics and decoding, the problem I am facing relates to the UDP socket used.
My UDP server has to send an acknowledgement to the client device upon receiving a message (that is the protocol), however, if I send those ACKs, the server socket stops receiving data after a while.
The server's UDP socket object is passed to an concurrent.futures.ThreadPoolExecutor that fires a function (send_ack) that sends the ACK, however this is not the issue because I tried calling send_ack in the main thread, after receiving data and the same issue occurs.
I suspect the problem is the remote device somehow breaks the connection or the ISP or MNO doesn't route the reply packet (this is a GPRS device) and then the socket.send() method that is used to send the acknowledge, somehow freezes other socket operations, specifically recvfrom_into called in the main thread loop.
I wrote two scripts to illustrate the situation:
udp_test_echo.py :
#!/usr/env/bin python
import socket
import concurrent.futures
def send_ack(sock, addr, ack):
print("Sending ACK to {}".format(addr))
sock.connect(addr)
print("connected to {}".format(addr))
sock.send(ack)
print("ACK sent to {}".format(addr))
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(("127.0.0.1", 1337))
data = bytearray([0] * 10)
executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)
while True:
print("listening")
nbytes, address = s.recvfrom_into(data)
print("Socket Data received {} bytes Address {}".format(nbytes, address))
print("Data received: ", data, " Echoing back to client")
executor.submit(send_ack, s, address, data[:nbytes])
udp_test_client.py:
#!/usr/env/bin python
import socket
import time
import random
def get_random_bytes():
return bytearray([random.randint(0,255) for b in range(10)])
ip = "127.0.0.1"
port = 1337
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((ip, port))
while True:
stuff_to_send = get_random_bytes()
print("Sending stuff", stuff_to_send)
s.sendall(stuff_to_send)
print("reply: ", s.recvfrom(10))
time.sleep(0.1)
Running udp_test_echo.py in one terminal and udp_test_client.py in another, we see normal operation but if you Ctrl+C the test client and re run it, you will see that the server doesn't respond until it is restarted.
Is there a way to timeout a specific sending operation from a specific call to socket.send() method without affecting other calls ? (I want my socket.recvfrom_into call to block on the main thread)
If I settimeout on the entire socket object, I am going to have to deal with many exceptions while waiting for data in the main thread and I don't like to have to rely on exceptions for proper program operation.

The culprit was the socket.connect() call in send_ack, when being called on the server's socket object it causes the socket to no longer be bound and listen on the port specified in the start of the program.
Instead the send_ack function was changed to be:
def send_ack(sock, addr, ack):
print("Sending ACK to {}".format(addr))
sock.sendto(ack, addr)
print("ACK sent to {}".format(addr))
socket.sendto(data, address) uses the existing connection instead of starting a new one.

Broadcasting to subnet address using pyzmq

I am trying to use broadcasting packets to subnet address, I successfully tried to do that using socket's broadcast option, But I recently started learning ZeroMQ so I would like to use it to broadcast the packets to the subnet. I used zmq.PUB, zmq.SUB but at the subscriber side, the packets are undelivered because I use subnet address. If I use IP address of the machine then it works, but that's not what I want.
Is there any option for broadcasting using ZMQ?
Here is the code I tried so far:
Publisher:
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://192.X.X.255:9999") # Note.
while True:
socket.send_string('hello')
time.sleep(1)
Subscriber:
context = zmq.Context()
sub=context.socket(zmq.SUB) # Note.
sub.setsockopt(zmq.SUBSCRIBE, b"") # Note.
sub.connect('tcp://192.x.x.255:9999') -> publishing only to subnet
while True:
print(sub.recv())
We can do the broadcasting of packets using a regular socket, for example, using:
sock.setsockopt(socket.SOL_SOCKET,socket.SO_BROADCAST,1)
But I want to replace this in a way I do it with ZMQ. Does the ZMQ really have a broadcast discovery in a different way or we should use the same code above as we do for regular broadcasting?

Suppose you have three Machines (M1, M2, M3) with three different IP addresses with the same subnet and a defined port. We want to publish a message (from M1) to each subscribers (M1, M2), therefore we would have the following code snippet:
Publisher (Machine1):
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:9999") # Note.
while True:
socket.send_string('hello-all')
time.sleep(1)
Subscriber (Machine2):
context = zmq.Context()
sub=context.socket(zmq.SUB)
sub.setsockopt(zmq.SUBSCRIBE, b"")
sub.connect('tcp://Machine1_IP:9999') # Note
while True:
print(sub.recv())
Subscriber (Machine3):
context = zmq.Context()
sub=context.socket(zmq.SUB)
sub.setsockopt(zmq.SUBSCRIBE, b"")
sub.connect('tcp://Machine1_IP:9999') # Note
while True:
print(sub.recv())

UDP Client sends ping once a second, and also prints anything sent to it?

Good afternoon everyone reading this, I am new to programming with sockets, as well as new to asynchronous coding (I feel async may be part of the solution to my problem), so forgive me for any silly mistakes I make.
To start, I have a UDP Echo server that acts as a game server. Anytime it gets a ping sent to it, it adds the source ip and port to a list of "connected clients", and sends that exact ping out to everyone on the list, excluding the sender. This works fairly well, because it reacts upon receiving a message, so it can always just listen. The problem with the client however, is that I need to be constantly sending pings, while also listening.
This is currently what my client looks like:
import socket
from time import sleep
from contextlib import contextmanager
UDP_IP_ADDRESS = "127.0.0.1"
UDP_PORT_NO = 14004
Message = b"Hello World, From Client B"
#contextmanager
def socket_ctx():
""" Context Manager for the socket. Makes sure the socket will close regardless of why it exited."""
my_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Assign IP address and a RANDOM available port number to socket
my_socket.bind(('127.0.0.1', 0))
try:
# Let the rest of the app use the socket and wait for it to finish
yield my_socket
finally:
my_socket.close()
def send_data(client_sock):
client_sock.sendto(Message, (UDP_IP_ADDRESS, UDP_PORT_NO))
def listen(client_sock):
print(client_sock.recvfrom(100))
with socket_ctx() as sock:
while True:
send_data(sock)
listen(sock)
sleep(2)
Currently, it sends a ping once, then just idles as it presumably is listening. If it does happen to get a ping back, say, another client send a ping to the server, and the server sent the ping to this client, it hears it, prints it, and starts the loop again. The issue is, without another client sending something to jolt this one out of the listen, it doesn't send it's pings.
I think async might be my solution, but I would have no clue how to go about that. Does anyone have a solution for this problem?

Here's how I would implement a server with "receive and handle incoming UDP sockets, plus do some packet-sending once per second" behavior. Note that this uses the select() function to multiplex the two tasks, rather than asynchronous I/O; hopefully that is okay.
import socket
import select
import time
UDP_IP_ADDRESS = "127.0.0.1"
UDP_PORT_NO = 14004
Message = b"Hello World, From Client B"
udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_socket.bind(('127.0.0.1', 0))
print "UDP socket is listening for incoming packets on port", udp_socket.getsockname()[1]
# When we want to send the next periodic-ping-message out
nextPingTime = time.time()
while True:
secondsUntilNextPing = nextPingTime - time.time();
if (secondsUntilNextPing < 0):
secondsUntilNextPing = 0
# select() won't return until udp_socket has some data
# ready-for-read, OR until secondsUntilNextPing seconds
# have passed, whichever comes first
inReady, outReady, exReady = select.select([udp_socket], [], [], secondsUntilNextPing)
if (udp_socket in inReady):
# There's an incoming UDP packet ready to receive!
print(udp_socket.recvfrom(100))
now = time.time()
if (now >= nextPingTime):
# Time to send out the next ping!
print "Sending out scheduled ping at time ", now
udp_socket.sendto(Message, (UDP_IP_ADDRESS, UDP_PORT_NO))
nextPingTime = now + 1.0 # we'll do it again in another second

Python broadcasting message to all clients in a socket

I have made a simple chat server using threads like the following:
#-*- coding:utf-8 -*-
import _thread as thread
import time
import socket
def now():
return time.asctime(time.localtime())
def handleclient(connection, ADDR):
sod = str(ADDR)
msg = sod+"joined the chat"
msg2 = msg.encode("utf-8")
connection.sendall(msg2)
while True:
recieved = connection.recv(1024)
adsf = recieved.decode("utf-8")
print(now(),"(%s):%s" % (ADDR, recieved))
output = "%s:%s"%(ADDR, recieved.decode("utf-8"))
message = output.encode("utf-8")
connection.sendall(message)
if __name__ == "__main__":
addr = ("", 8080)
r =socket.socket()
print("socket object created at", now())
r.bind(addr)
r.listen(5)
while True:
print("Waiting for clients...")
connection, ADDR = r.accept()
print("We have connection from ", ADDR)
thread.start_new_thread(handleclient, (connection, ADDR))
However, it looks like the sendall isnt working and sending the message to only the person who sent it. How can I make it send it to all clients?

There is nothing like what you're trying to do, because as pointed out in the commends, sendall() means "definitely send all my bytes and keep trying until you have," not "send these bytes to lots of clients."
You will want to use either UDP multicast (if you're on a relatively reliable network which supports it, such as a LAN or corporate WAN), or you will simply need to send explicitly to every connected client. The other alternative is peer-to-peer: send to several clients and instruct those clients to send to more clients until all clients are taken care of. Obviously this requires more coding. :)

You may have a look at Zero MQ, which provides high-level facilities over sockets by implementing several patterns ( publish/subscribe , push/pull, etc...).