Making pynng and socket talk to each other - python

TL;DR
I spin up a server using pynng, then a client from Python Standard Library socket will try to send messages to it.
The problem is that client can send the message, but server is oblivious to it. Therefore, it doesn't work.
Am I missing something? Some low-level protocol setting? Some termination character?
The reason why I'm doing this is that I will build a Python script that uses pynng to act as a server. Then a non-Python program (which I assume has knowledge of basic TCP protocols) will try to talk with this Python server. Thus I am using the IMHO most primitive socket library I could operate, the socket module in the standard library.
The details
I will present code snippets as I discuss, but I will show the full minimal code example at the end.
I am trying to spin up a server using pynng
def server():
with pynng.Pair0(listen=f'tcp://{HOST:s}:{PORT:d}', recv_timeout=10000) as s:
print("Server running")
data = s.recv() # Blocks forever here
print(data)
Then, client that looks like this will try to connect to it:
def client():
with socket.create_connection(address=(HOST, PORT), timeout=5) as s:
print("Client connected")
s.sendall(b'Hello world')
print("Client sent message")
I put them all together in using threading:
def main():
srv = threading.Thread(target=server)
cli = threading.Thread(target=client)
srv.start()
cli.start()
srv.join()
cli.join()
Minimum working code
All told, this is the minimum working code:
import socket
import pynng
import threading
HOST = "127.0.0.1"
PORT = 65432
def main():
srv = threading.Thread(target=server)
cli = threading.Thread(target=client)
srv.start()
cli.start()
srv.join()
cli.join()
def server():
with pynng.Pair0(listen=f'tcp://{HOST:s}:{PORT:d}', recv_timeout=10000) as s:
print("Server running")
data = s.recv() # Blocks forever here
print("Message received")
print(data)
def client():
with socket.create_connection(address=(HOST, PORT), timeout=5) as s:
print("Client connected")
s.sendall(b'Hello world')
print("Client sent message")
if __name__ == "__main__":
main()
Then I run this in the terminal
$ python main.py
It seems that the server is unable to recv messages, and the recv attempt thus times out at 10000ms.
Server running
Client connected
Client sent message
Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/kmonisit/miniconda3/envs/engg/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/home/kmonisit/miniconda3/envs/engg/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "main.py", line 39, in server
data = s.recv() # Blocks forever here
File "/home/kmonisit/miniconda3/envs/engg/lib/python3.8/site-packages/pynng/nng.py", line 454, in recv
check_err(ret)
File "/home/kmonisit/miniconda3/envs/engg/lib/python3.8/site-packages/pynng/exceptions.py", line 201, in check_err
raise exc(string, err)
pynng.exceptions.Timeout: Timed out

pynng is based on Nanomsg Next Generation, which is an implementation of the Scalability Protocols. The scalability protocols work on many different transports, including tcp, but bare sockets are not compatible. However, with a little bit of prayer and elbow grease, they can be made compatible. Which is too say, you can implement the scalability protocols in pure Python if need be.
First, we need to know what the wire format is; thankfully that is documented in an RFC in the original nanomsg repository. An implementation of a Pair0 client is here:
class Pair0:
"""A poor implementation of the Pair0 protocol"""
def __init__(self, host, port, timeout=None):
self._sock = socket.create_connection(address=(host, port), timeout=timeout)
# https://github.com/nanomsg/nanomsg/blob/master/rfc/sp-tcp-mapping-01.txt
# upon making a connection, both ends are required to send this header
self._sock.send(b'\x00SP\x00\x00\x10\x00\x00')
print(self._sock.recv(8))
def send(self, data):
# messages are simply "length + payload". Length is 64-bit in network byte
# order.
packed = struct.pack('!Q', len(data))
self._sock.sendall(packed + data)
def recv(self):
size_bytes = self._sock.recv(8)
(size,) = struct.unpack('!Q', size_bytes)
received = 0
parts = []
while received < size:
data = self._sock.recv(size - received)
received += len(data)
parts.append(data)
return b''.join(parts)
And integrated into your test program:
import socket
import struct
import pynng
import threading
import time
HOST = "127.0.0.1"
PORT = 65432
def main():
srv = threading.Thread(target=server)
srv.start()
# sleep to give the server time to bind to the address
time.sleep(0.1)
_client = Pair0(HOST, PORT, 1)
_client.send(b'hello pynng')
_client.send(b'hope everything is going well for you')
print(_client.recv())
print(_client.recv())
srv.join()
def server():
with pynng.Pair0(listen=f'tcp://{HOST:s}:{PORT:d}', recv_timeout=1000) as s:
print("Server running")
for _ in range(2):
data = s.recv()
print("Message received")
print(data)
s.send(b'hello bad client')
s.send(b'I hope you are doing okay')
class Pair0:
"""A poor implementation of the Pair0 protocol"""
def __init__(self, host, port, timeout=None):
self._sock = socket.create_connection(address=(host, port), timeout=timeout)
# https://github.com/nanomsg/nanomsg/blob/master/rfc/sp-tcp-mapping-01.txt
# upon making a connection, both ends are required to send this header
self._sock.send(b'\x00SP\x00\x00\x10\x00\x00')
print(self._sock.recv(8))
def send(self, data):
# messages are simply "length + payload". Length is 64-bit in network byte
# order.
packed = struct.pack('!Q', len(data))
self._sock.sendall(packed + data)
def recv(self):
size_bytes = self._sock.recv(8)
(size,) = struct.unpack('!Q', size_bytes)
received = 0
parts = []
while received < size:
data = self._sock.recv(size - received)
received += len(data)
parts.append(data)
return b''.join(parts)
if __name__ == "__main__":
main()
Now, this is nowhere near as robust as the implementation in pynng (which relies on the underlying nng implementation). nng does The Right Thing™ in edge conditions, including losing network, handling multiple clients, keeping track of state machines, handling SIGINT, etc. This is also an imcomplete implementation, as it does not bind, etc.
Disclaimer: I am the author of pynng.

Related

Multithreading sockets with a central relay-like server

I have previously managed to implement a client-server socket script which relays messages between a single client and the server and I'm now trying to implement a multiple-client system.
More specifically, I would like to use the server as some sort of medium between two clients which retrieves information from one client and relays it to the other. I had tried to attach and send the port number of the receiving client and then extract it from the message on the server side. After that, I would try and send it to whatever socket with that port number but I ran into some trouble (as port numbers are determined at the point of sending I believe?) so now I am simply just trying to relay the sent message back to all clients. However, the problem is that the message is only being sent to the server and not being relayed to the desired client.
I had previously tried to implement a peer-to-peer system but I ran into trouble so I decided to take a step back and do this instead.
Server.py:
import socket, _thread, threading
import tkinter as tk
SERVERPORT = 8600
HOST = 'localhost'
class Server():
def __init__(self):
self.Connected = True
self.ServerSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.ServerSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
self.ServerSocket.bind((HOST, SERVERPORT))
self.ServerSocket.listen(2)
self.Clients = []
def Listen(self):
print('Server is now running')
while self.Connected:
ClientSocket, Address = self.ServerSocket.accept()
self.Clients.append(Address)
print('\nNew user connected', Address)
t = threading.Thread(target=self.NewClient, args=(ClientSocket,
Address))
t.daemon = True
t.start()
self.Socket.close()
def NewClient(self, ClientSocket, Address):
while self.Connected:
if ClientSocket:
try:
ReceivedMsg = ClientSocket.recv(4096)
print('Message received from', Address, ':', ReceivedMsg)
self.Acknowledge(ClientSocket, Address)
if ReceivedMsg.decode('utf8').split()[-1] != 'message':
ReceiverPort = self.GetSendPort(ReceivedMsg)
self.SendToClient(ClientSocket,ReceivedMsg,ReceiverPort)
except:
print('Connection closed')
raise Exception
ClientSocket.close()
def Acknowledge(self, Socket, Address):
Socket.sendto(b'The server received your message', Address)
def GetSendPort(self, Msg):
MsgDigest = Msg.decode('utf8').split()
return int(MsgDigest[-1])
def SendToClient(self, Socket, Msg, Port):
Addr = (HOST, Msg)
for Client in self.Clients:
Socket.sendto(Msg, Client)
def NewThread(Func, *args):
if len(args) == 1:
t = threading.Thread(target=Func, args=(args,))
elif len(args) > 1:
t = threading.Thread(target=Func, args=args)
else:
t = threading.Thread(target=Func)
t.daemon = True
t.start()
t.join()
Host = Server()
NewThread(Host.Listen)
And the Client(.py):
import socket, threading
import tkinter as tk
Username = 'Ernest'
PORT = 8601
OtherPORT = 8602
SERVERPORT = 8600
HOST = '127.0.0.1'
class Client():
def __init__(self, Username):
self.Connected, self.Username = False, Username
self.Socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def Connect(self):
print('Trying to connect')
try:
self.Socket.connect((HOST, SERVERPORT))
self.Connected = True
print(self.Username, 'connected to server')
Msg = MsgUI(self.Username)
Msg.Display()
except Exception:
print('Could not connect to server')
raise Exception
def SendMsg(self):
if self.Connected:
Msg = '{} sent you a message {}'.format(self.Username, OtherPORT)
self.Socket.sendall(bytes(Msg, encoding='utf8'))
self.GetResponse()
def GetResponse(self, *args):
AckMsg = '\n{} received the message'.format(self.Username)
NMsg = '\n{} did not receive the message'.format(self.Username)
if self.Connected:
Msg = self.Socket.recv(4096)
print(Msg)
if Msg:
self.Socket.sendall(bytes(AckMsg, encoding='utf8'))
else:
self.Socket.sendall(bytes(NMsg, encoding='utf8'))
class MsgUI():
def __init__(self, Username):
self.Username = Username
self.entry = tk.Entry(win)
self.sendbtn = tk.Button(win, text='send', command=Peer.SendMsg)
def Display(self):
self.entry.grid()
self.sendbtn.grid()
win.mainloop()
win = tk.Tk()
Peer = Client(Username)
Peer.Connect()
I want a message to be sent whenever the user presses the send button in the tkinter window, but at the same time, it is continually 'listening' to see if it received any messages.
I also previously tried to run the GetResponse method in the Client in another thread and instead of if self.Connected I used while self.Connected and it still didn't work.
UPDATE
After some helpful comments, I have edited the two files as such:
The server now holds the two sockets for each client which is run first. The server file is imported into the client file as a module. Each client file is then run and each client runs a function in the server file, requesting to use the socket. If the request is allowed (i.e. no error was thrown), the socket is connected, added to a set of clients stored in the server file and then returned to the client file. The client then uses this socket to send and receive messages.
Server.py
import socket, _thread, threading
import tkinter as tk
SERVERPORT = 8600
HOST = 'localhost'
class Server():
def __init__(self):
self.Connected = True
self.ServerSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.ServerSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
self.ServerSocket.bind((HOST, SERVERPORT))
self.ServerSocket.listen(2)
self.Clients = {}
def ConnectClient(self, Username, Port):
Socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.Clients[Username] = [Socket, Port, False]
try:
self.Clients[Username][0].connect((HOST, SERVERPORT))
self.Clients[Username][2] = True
print('Opened port for user', Username)
return Socket
except Exception:
print('Could not open port for user', Username)
raise Exception
def Listen(self):
print('Server is now running')
while self.Connected:
ClientSocket, Address = self.ServerSocket.accept()
print('\nNew user connected', Address)
t = threading.Thread(target=self.NewClient, args=(ClientSocket,
Address))
t.daemon = True
t.start()
self.Socket.close()
def NewClient(self, ClientSocket, Address):
while self.Connected:
if ClientSocket:
try:
ReceivedMsg = ClientSocket.recv(4096)
if b'attempting to connect to the server' in ReceivedMsg:
ClientSocket.send(b'You are now connected to the server')
else:
print('Message received from', Address, ':',ReceivedMsg)
#self.Acknowledge(ClientSocket, Address)
ReceiverPort = self.GetSendPort(ReceivedMsg)
if ReceiverPort != None:
self.SendToClient(ClientSocket,ReceivedMsg,
ReceiverPort)
except:
print('Connection closed')
raise Exception
ClientSocket.close()
def Acknowledge(self, Socket, Address):
Socket.sendto(b'The server received your message', Address)
def GetSendPort(self, Msg):
MsgDigest = Msg.decode('utf8').split()
try:
Port = int(MsgDigest[-1])
except ValueError:
Port = None
return Port
def SendToClient(self, Socket, Msg, Port):
Addr = (HOST, Port)
Receiver = None
for Client, Vars in self.Clients.items():
if Vars[1] == Port:
Receiver = Client
self.Clients[Receiver][0].sendto(Msg, Addr)
def NewThread(Func, *args):
if len(args) == 1:
t = threading.Thread(target=Func, args=(args,))
elif len(args) > 1:
t = threading.Thread(target=Func, args=args)
else:
t = threading.Thread(target=Func)
t.daemon = True
t.start()
t.join()
Host = Server()
if __name__ == '__main__':
NewThread(Host.Listen)
And Client.py
import socket, threading, Server
import tkinter as tk
Username = 'Ernest'
PORT = 8601
OtherPORT = 8602
SERVERPORT = 8600
HOST = '127.0.0.1'
class Client():
def __init__(self, Username):
self.Connected, self.Username = False, Username
def Connect(self):
print('Requesting to connect to server')
try:
self.Socket = Server.Host.ConnectClient(self.Username, PORT)
self.Connected = Server.Host.Clients[self.Username][2]
Msg = '{} is attempting to connect to the server'.format(self.Username)
self.Socket.sendall(bytes(Msg, encoding='utf8'))
ReceivedMsg = self.Socket.recv(4096)
print(ReceivedMsg)
Msg = MsgUI(self.Username)
Msg.Display()
except Exception:
print('Could not connect to server')
raise Exception
def SendMsg(self):
try:
if self.Connected:
Msg = '{} sent you a message {}'.format(self.Username,OtherPORT)
self.Socket.sendall(bytes(Msg, encoding='utf8'))
self.GetResponse()
except Exception:
print('Connection closed')
raise Exception
def GetResponse(self, *args):
AckMsg = '\n{} received the message'.format(self.Username)
NMsg = '\n{} did not receive the message'.format(self.Username)
if self.Connected:
Msg = self.Socket.recv(4096)
print(Msg)
if Msg:
self.Socket.sendall(bytes(AckMsg, encoding='utf8'))
else:
self.Socket.sendall(bytes(NMsg, encoding='utf8'))
class MsgUI():
def __init__(self, Username):
self.Username = Username
self.entry = tk.Entry(win)
self.sendbtn = tk.Button(win, text='send', command=Peer.SendMsg)
def Display(self):
self.entry.grid()
self.sendbtn.grid()
win.mainloop()
win = tk.Tk()
Peer = Client(Username)
Peer.Connect()
Now the problem is more of a python and scope problem. When trying to relay the message back to the client, I was getting a KeyError as the Clients dictionary was still empty. When making the function call to the server in the client file, it's clear that the update to the dictionary happens in the client file rather than the server file - which is in a different instance. I need a method of changing the contents of the Clients dictionary that is called to action by the client file but takes effect in the server file.
Are you committed to multithreading? Threads don't run concurrently in python ( due to the GIL), and while they are one way to handle concurrent operations, they aren't the only way and usually they're not the best way, unless they're the only way. Consider this code, which doesn't handle failure cases well, but seems to work as a starting point.
import socket, select, Queue
svrsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
svrsock.setblocking(0)
svrsock.bind(('', 17654))
svrsock.listen(16)
client_queues = {}
write_ready=[] # we'll update this for clients only that have things in the queue
while client_queues.keys() + [svrsock] :
readable, writable, exceptional = select.select(client_queues.keys() + [svrsock] , write_ready, [])
for rd in readable:
if rd is svrsock: # reading listening socket == accepting connection
conn, addr = svrsock.accept()
print("Connection from {}".format(addr))
conn.setblocking(0)
client_queues[conn] = Queue.Queue()
else:
data = rd.recv(1024)
if data:
# TODO: send to all queues
print("Message from {}".format(rd.getpeername()))
for sock, q in client_queues.iteritems():
q.put("From {}: {}".format( rd.getpeername(), data))
if sock not in write_ready:
write_ready.append(sock)
for rw in writable:
try:
data = client_queues[rw].get_nowait()
rw.send(data)
except Queue.Empty:
write_ready.remove(rw)
continue
The concept is pretty simple. The server accepts connections; each connection (socket) is associated with a queue of pending messages. Each socket that's ready for reading is read from, and its message is added to each client's queue. The recipient client is added into the write_ready list of clients with data pending, if it's not already in there. Then each socket that's ready for writing has its next queued message written to it. If there are no more messages, the recipient is removed from the write_ready list.
This is very easy to orchestrate if you don't use multithreading because all coordination is inherent in the order of the application. With threads it would be more difficult and a lot more code, but probably not more performance due to the gil.
The secret to handling multiple I/O streams concurrently without multithreading is select. In principle it's pretty easy; we pass select() a list of possible sockets for reading, another list of possible sockets for writing, and a final list that for this simplified demo I completely ignore . The results of the select call will include one or more sockets that are actually ready for reading or writing, which allows me to block until one or more sockets are ready for activity. I then process all the sockets ready for activity every pass ( but they've already been filtered down to just those which wouldn't block).
There's a ton still to be done here. I don't cleanup after myself, don't track closed connections, don't handle any exceptions, and so on. but without having to worry about threading and concurrency guarantees, it's pretty easy to start addressing these deficiencies.
Here it is "in action". Here for the client side I use netcat, which is perfect for layer 3 testing without layer 4+ protocols ( in other words, raw tcp so to speak). It simply opens a socket to the given destination and port and sends its stdin through the socket and sends its socket data to stdout, which makes it perfect for demoing this server app!
I also wanted to point out, coupling code between server and client is inadvisable because you won't be able to roll out changes to either without breaking the other. It's ideal to have a "contract" so to speak between server and client and maintain it. Even if you implement the behavior of server and client in the same code base, you should use the tcp communications contract to drive your implementation, not code sharing. Just my 2 cents, but once you start sharing code you often start coupling server/client versions in ways you didn't anticipate.
the server:
$ python ./svr.py
Connection from ('127.0.0.1', 52059)
Connection from ('127.0.0.1', 52061)
Message from ('127.0.0.1', 52061)
Message from ('127.0.0.1', 52059)
Message from ('127.0.0.1', 52059)
First client ( 52059):
$ nc localhost 17654
hello
From ('127.0.0.1', 52061): hello
From ('127.0.0.1', 52059): hello
From ('127.0.0.1', 52059): hello
Second client:
$ nc localhost 17654
From ('127.0.0.1', 52061): hello
hello
From ('127.0.0.1', 52059): hello
hello
From ('127.0.0.1', 52059): hello
If you need more convincing on why select is way more compelling than concurrent execution, consider this: Apache is based on a threading model, in other words, the connections each get a worker thread . nginx is based on a select model, so you can see how much faster that can potentially be. Not to say that nginx is inherently better, as Apache benefits from the threading model because of its heavy use of modules to extend capabilities ( mod_php for example), whereas nginx doesn't have this limitation and can handle all requests from any thread. But the raw performance of nginx is typically considered far higher and far more efficient, and a big reason for this is that it avoids almost all the cpu context switches inherent in apache. It's a valid approach!
A word on scaling. Obviously, this wouldn't scale forever. Neither would a threading model; eventually you run out of threads. A more distributed and high throughput system would likely use a Pub/Sub mechanism of some kind, offloading the client connection tracking and message queueing from the server to a pub/sub data tier and allowing connections to be restored and queued data to be sent, as well as adding multiple servers behind a load balancer. Just throwing it out there. You might be pleasantly surprised how well select can scale ( cpu is so much faster than network anyway that it's likely not the bottleneck).

Socket Server with multiply Clients

I just started programming Python.
My goal is to built a digital Picture Frame with three Screens. Therefore I use 3 Raspis, one for each Monitor.
For the communication of these Raspis I need to program a server and a Client.
For a first test I want to built a server which is able to send and receive messages to/from multiple clients.
So I started with a few socket tutorials an created the following program.
Server Class (TcpServer.py)
class TcpServer:
clients = []
serverIsRunning = 0
port = 0
def __init__(self, port):
self.port = port
self.serverIsRunning = 0
self.serverRunning = 0
def startServer (self):
print("start Server...")
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind(("", self.port))
self.server.listen(1)
self.serverRunning = 1
while self.serverRunning:
read, write, oob = select.select([self.server] + self.clients, [], [])
for sock in read:
if sock is self.server:
client, addr = self.server.accept()
self.clients.append(client)
print ("+++ Client ", addr[0], " verbunden")
else:
nachricht = sock.recv(1024)
ip = sock.getpeername()[0]
if nachricht:
print (ip, nachricht)
else:
print ("+++ Verbindung zu ", ip , " beendet")
sock.close()
self.clients.remove(sock)
for c in self.clients:
c.close()
self.clients.remove(c)
self.server.close()
def send(self, message):
message = message.encode()
self.server.send(message)
Client class (TcpClient.py)
import socket
class TcpClient:
def __init__(self, ip, port):
self.serverAdress = (ip, port)
self.connected = 0
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(self.serverAdress)
print ("connectet to ", self.serverAdress)
def send(self, message):
message = message.encode()
self.connection.send(message)
Server:
import threading
import TcpServer
tcpServer = TcpServer.TcpServer(50000)
threadTcpServer = threading.Thread(target = tcpServer.startServer)
threadTcpServer.start()
while True:
tcpServer.send(input("Nachricht eingeben: "))
Client:
import threading
import TcpClient
tcpClient = TcpClient.TcpClient("192.168.178.49", 50000)
while True:
tcpClient.send(input("Nachricht eingeben: "))
I can send messages from the Client to the server, but when I want to send a Message from the server to the client it generates the following error:
BrokenPipeError: [Errno 32] Broken pipe
I assume it is because the server thread blocks the socket while waiting of a incoming message. But I have no idea how to handle this.
How can I program a server who can send and receive messages? Can you recommend a tutorial? I didn't found a tutorial who describes a solution for my problem.
Edit:
Now I tried to solve the problem with the socketserver library, but I still can't solve may problem.
here is my new code for the server:
import socketserver
import threading
import time
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
threadTcpServer = threading.Thread(target = server.serve_forever)
threadTcpServer.start()
print("server started")
time.sleep(10)
print("sending Data")
server.request.sendall("Server is sending...")
it generates the error:
AttributeError: 'TCPServer' object has no attribute 'request'
My goal is to write a server with a thread who receives Data and still be able to send data from a other thread.
Is this even possible with only one socket?
You should use the provided socketserver rather than writing all the handling of sockets and select etc.
There are multiple problems with your code -
1 - The server is trying to write to the listening socket!! The client communication socket is the one that you get from the accept() call and that is the one you have to use for reading and writing.
2 - The client is sending the data and completing immediately, but it should really wait for getting a response. Otherwise, the python / OS will close the client socket as soon as the program completes and it will mostly be before the server gets a chance to respond.
I believe with the Handler code you are able to receive the data sent by the client on the server and are also able to send some data back from the Handler to the client? You must have understood that the server cannot send any data back unless there is a client connected to it?
Now, to send data to the client (or clients) from "another" thread, you will need a way to make the handler objects or the client sockets (available inside the Handler object as self.request) available to the "another" thread.
One way is to override the def __init__(self, request, client_address, server): method and save this object's reference in a global list. Remember to do the below as the last line of the overridden init -
# BaseRequestHandler __init__ must be the last statement as all request processing happens in this method
socketserver.BaseRequestHandler.__init__(self, request, client_address, server)
Once you have all the client handlers in the global list, you can easily write to all the clients from any thread as per your needs. You must read about synchronization (Locks) and understand that using same object / socket from multiple threads can create some logical / data issues with your application.
Another thing that you have to worry about and code for is cleaning up this global list whenever a client closes the connection.

UDP server with asyncio

I am trying to receive UDP packets in a python asyncio loop. I am very new at asyncio so I'm probably doing something wrong, as the callbacks never get called:
import asyncio
class DiscoveryProtocol(asyncio.DatagramProtocol):
def __init__(self):
super().__init__()
def connection_made(self, transport):
self.transport = transport
def datagram_received(self, data, addr):
print(data)
def start_discovery():
loop = asyncio.get_event_loop()
t = loop.create_datagram_endpoint(DiscoveryProtocol,local_addr=('',5006))
loop.run_until_complete(t)
loop.run_forever()
I can receive packets with plain old sockets (without asyncio).
What am I doing wrong?
No accepted answer so this seems to have atrophied, but it comes up in searches. If someone gets here and wants a final solution, the following code snippet illustrates a fully functional UDP server. The write_messages() function is just a test method. It reads a log file with whatever you want in it, and publishes each line as a Syslog message to UDP port 514. Running this as a script illustrates the server listening and printing whatever it drains from syslog. update the SyslogProtocol with whatever formatting/processing needs you have.
import socket
import asyncio
import os, random
HOST, PORT = 'localhost', 514
def send_test_message(message: 'Message to send to UDP port 514') -> None:
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.sendto(message.encode(), (HOST, PORT))
async def write_messages() -> "Continuously write messages to UDP port 514":
dir_path = os.path.dirname(os.path.realpath(__file__))
fp = open(os.path.join(dir_path, "tests/example.log"))
print("writing")
for line in fp.readlines():
await asyncio.sleep(random.uniform(0.1, 3.0))
send_test_message(line)
class SyslogProtocol(asyncio.DatagramProtocol):
def __init__(self):
super().__init__()
def connection_made(self, transport) -> "Used by asyncio":
self.transport = transport
def datagram_received(self, data, addr) -> "Main entrypoint for processing message":
# Here is where you would push message to whatever methods/classes you want.
print(f"Received Syslog message: {data}")
if __name__ == '__main__':
loop = asyncio.get_event_loop()
t = loop.create_datagram_endpoint(SyslogProtocol, local_addr=('0.0.0.0', PORT))
loop.run_until_complete(t) # Server starts listening
loop.run_until_complete(write_messages()) # Start writing messages (or running tests)
loop.run_forever()

Python sockets; sending from client receiving on server

I am trying to send messages on TCP/IP all on host machine. This is working, although for some reason the socket needs to be re-instantiated for every new message on the client side only. For example here is a basic client that sends three separate messages:
import socket
host = '127.0.0.1'
class Client:
def __init__(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self):
self.sock.connect((host,12347))
def send(self,message):
self.sock.sendall(message)
def close(self):
self.sock.close()
if __name__ == "__main__":
message1 = "I am message 1"
message2 = "I am message 2"
message3 = "I am message 3"
#exp = Client()
#exp.connect()
for i in range(0,3):
try:
exp = Client()
exp.connect()
if i == 0:
txt = message1
elif i == 1:
txt = message2
elif i == 2:
txt = message3
exp.send(txt)
exp.close()
print i
exp.send(txt)
except:
pass
and the server that receives:
#!/usr/bin/env python
import socket
class communication:
def __init__(self):
try:
host = '127.0.0.1'
self.Server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.Server.bind((host,12347))
self.Server.listen(1)
finally:
print "setup finished"
def recieve(self):
(connection, client_address) = self.Server.accept()
data = connection.recv(128)
return data
def close(self):
self.server.close()
if __name__ == "__main__":
exp = communication()
while True:
try:
(connection,client_address) = exp.Server.accept()
message = connection.recv(128)
finally:
print message
if message == "I am message 3":
exp.close()
You see how I re-call the Client class in each iteration of the for loop. This seems to be necessary for sending messages 2 and 3. If the socket is instantiated only once at the start of the main code along with the connect() function, then the server hangs on the recv() after the first message has been sent.
I can't understand why this is happening and the socket only needs to be setup once on the server side. I am doing something wrong, or is this normal?
Thanks!
It's even worse than you think. Take a look at your server code. exp.Server.accept() accepts a connection from the client, but connection.receive() ignores that connection completely and does a second self.Server.accept(). You ignore half of your connections!
Next, your server only does a single receive.... Even if you tried to send more messages on the connection, the server would ignore them.
But you can't just add a recv loop. Your client and server need some way to mark message boundaries so the server knows how to pull them out. Some text based systems use a new line. Others send a message size or fixed size header that the server can read. HTTP for example uses a combination of new lines and data count.
If you want to learn sockets from the ground up just know that they are complicated and you'll need to study. There are lots of ways to build a server and you'll need to understand the trade-offs. Otherwise, there are many frameworks from XMLRPC to zeromq that do some of the heavy lifting for you.

Python Game Server

I'm completely lost trying to create a UDP server/client for my game in python. I'm new to the language and only have limited experience with networking. Right now, the server runs, but doesn't seem to be getting any messages from the client.
Server:
class GameServer:
class GameServerUDPHandler(socketserver.BaseRequestHandler):
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
print("{} wrote:".format(self.client_address[0]))
print(data)
socket.sendto(data.upper(), self.client_address)
def __init__(self, port):
self.server = socketserver.UDPServer(("localhost", port), self.GameServerUDPHandler)
def start_server(self):
self.server.serve_forever(
Client:
import socket
import sys
class GameClient:
def __init__(self, port, host):
self.port = port
self.host = host
self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def register(self):
self.socket.sendto(bytes("register\n", "utf-8"), (self.host, self.port))
self.numberID = int(self.socket.recv(1024))
print("Received: {}".format(self.numberID))
-Main/Start of program
import gameserver
import gameclient
if __name__ == "__main__":
server = gameserver.GameServer(1300)
server.start_server()
client = gameclient.GameClient(1300, "localhost")
client.register()
NOTE: I'm most likely to multiple things wrong and may be violating several best practices in the language. I really have no clue.
The problem is that some of these calls are blocking. In particular, the serve_forever() method will run forever, so you need to put that on a separate thread if you want the rest of your program to continue:
import threading
if __name__ == "__main__":
server = GameServer(1300)
server_thread = threading.Thread(target=lambda: server.start_server())
server_thread.start()
time.sleep(1) # Give it time to start up; not production quality code of course
client = GameClient(1300, "localhost")
client.register()
socket.recv() is also a blocking call but that might be okay in this case.
Seems like this library isn't asynchronous so your first call to serve_forever will not return and your client never gets started. You can create a new thread to launch the server on or split your client and server into seperate processes.

Categories

Resources