I've started Python a few times ago and now, I'm currently creating a socket server. I already have the server functioning with multiple threads with multiple clients (Hurray !) But I'm looking for functionality I can't call (i don't even know if it exists) I would like to create a kind of channel where client can send different type of message.
An example I create a channel INFO and if the server received this type of socket it just does a print
I create another channel DEBUG where I can send custom command which the server will execute
etc
In a non-programming language it will do this:
def socketDebug(command):
run command
def socketInfo(input):
print input
if socket == socketDebug:
socketDebug(socket.rcv)
else:
if socket == socketInfo:
socketInfo(socket.rcv)
I hope I'm clear.
Here is a quite simple implementation of a Channel class. It creates a socket, to accept
connections from clients and to send messages. It is also a client itself,
receiving messages from other Channel instances (in separate processes for example).
The communication is done in two threads, which is pretty bad (I would use async io). when
a message is received, it calls the registered function in the receiving thread which
can cause some threading issues.
Each Channel instance creates its own sockets, but it would be much more scalable to
have channel "topics" multiplexed by a single instance.
Some existing libraries provide a "channel" functionality, like nanomsg.
The code here is for educational purposes, if it can help...
import socket
import threading
class ChannelThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.clients = []
self.chan_sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.chan_sock.bind(('',0))
_, self.port = self.chan_sock.getsockname()
self.chan_sock.listen(5)
self.daemon=True
self.start()
def run(self):
while True:
new_client = self.chan_sock.accept()
if not new_client:
break
self.clients.append(new_client)
def sendall(self, msg):
for client in self.clients:
client[0].sendall(msg)
class Channel(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.daemon = True
self.channel_thread = ChannelThread()
def public_address(self):
return "tcp://%s:%d" % (socket.gethostname(), self.channel_thread.port)
def register(self, channel_address, update_callback):
host, s_port = channel_address.split("//")[-1].split(":")
port = int(s_port)
self.peer_chan_sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.peer_chan_sock.connect((host, port))
self._callback = update_callback
self.start()
def deal_with_message(self, msg):
self._callback(msg)
def run(self):
data = ""
while True:
new_data = self.peer_chan_sock.recv(1024)
if not new_data:
# connection reset by peer
break
data += new_data
msgs = data.split("\n\n")
if msgs[-1]:
data = msgs.pop()
for msg in msgs:
self.deal_with_message(msg)
def send_value(self, channel_value):
self.channel_thread.sendall("%s\n\n" % channel_value)
Usage:
In process A:
c = Channel()
c.public_address()
In process B:
def msg_received(msg):
print "received:", msg
c = Channel()
c.register("public_address_string_returned_in_process_A", msg_received)
In process A:
c.send_value("HELLO")
In process B:
received: HELLO
Related
I have a python Twisted server application interfacing with a legacy client application and each client has been assigned a specific port on which to connect to the server. So I have set up listeners on all those ports on the server and it is working beautifully, but I need to build in some safeguards to disallow more than one client connecting on the same server port. There are too many things on the client application that break when another client is connected to the same port, and I can't update that application right now. I gotta live with how it has been operating.
I know that I could build in some logic to the connectionMade() function to see if someone already exists on that port and if so, close this new connection. But I'd rather there be a way to reject it to begin with, so the client is not even allowed to connect. Then the client will know they made a mistake and they can alter which port they are trying to connect on.
Here is a stripped down version of my server code if that helps.
from twisted.internet.protocol import Factory
from twisted.internet.protocol import Protocol
from twisted.internet import reactor
from twisted.internet import task
import time
class MyServerTasks():
def someFunction(msg):
#Do stuff
def someOtherFunction(msg):
#Do other stuff
class MyServer(Protocol):
def __init__(self, users):
self.users = users
self.name = None
def connectionMade(self):
#Depending on which port is connected, go do stuff
def connectionLost(self, reason):
#Update dictionaries and other global info
def dataReceived(self, line):
t = time.strftime('%Y-%m-%d %H:%M:%S')
d = self.transport.getHost()
print("{} Received message from {}:{}...{}".format(t, d.host, d.port, line)) #debug
self.handle_GOTDATA(line)
def handle_GOTDATA(self, msg):
#Parse the received data string and do stuff based on the message.
#For example:
if "99" in msg:
MyServerTasks.someFunction(msg)
class MyServerFactory(Factory):
def __init__(self):
self.users = {} # maps user names to Chat instances
def buildProtocol(self, *args, **kwargs):
protocol = MyServer(self.users)
protocol.factory = self
protocol.factory.clients = []
return protocol
reactor.listenTCP(50010, MyServerFactory())
reactor.listenTCP(50011, MyServerFactory())
reactor.listenTCP(50012, MyServerFactory())
reactor.listenTCP(50013, MyServerFactory())
reactor.run()
When a client connected to the server, twisted use the factory to create
a protocol (by calling its buildProtocol method) instance to handle the client request.
therefore you can maintain a counter of connected client in your MyServerFactory,
if the counter has reached maximum allowed connected client you can return None
instead of creating new protocol for that client.
Twisted will close the client connection if the factory doesn't return a protocol from
its buildProtocol method.
you can see here
class MyServer(Protocol):
def __init__(self, users):
self.users = users
self.name = None
def connectionMade(self):
#Depending on which port is connected, go do stuff
def connectionLost(self, reason):
#Update dictionaries and other global info
self.factory.counter -= 1
def dataReceived(self, line):
t = time.strftime('%Y-%m-%d %H:%M:%S')
d = self.transport.getHost()
print("{} Received message from {}:{}...{}".format(t, d.host, d.port, line)) #debug
self.handle_GOTDATA(line)
def handle_GOTDATA(self, msg):
#Parse the received data string and do stuff based on the message.
#For example:
if "99" in msg:
MyServerTasks.someFunction(msg)
class MyServerFactory(Factory):
MAX_CLIENT = 2
def __init__(self):
self.users = {} # maps user names to Chat instances
self.counter = 0
def buildProtocol(self, *args, **kwargs):
if self.counter == self.MAX_CLIENT:
return None
self.counter += 1
protocol = MyServer(self.users)
protocol.factory = self
protocol.factory.clients = []
return protocol
I want to implement a streaming server which sends and endless stream of data to all connected clients. Multiple clients should be able to connect and disconnect from the server in order to process the data in different ways.
Each client is served by a dedicated ClientThread, which sub-classes Thread and contains a queue of the data to be sent to the client (necessary, since clients might process data at different speeds and because bursts of data can occur which the clients might be unable to handle).
The program listens to incoming client connections via a seperate ClientHandlerThread. Whenever a client connects, the ClientHandlerThread spawns a ClientThread and adds it to a list.
As a dummy example, the main Thread increments an integer each second and pushes it to all ClientThread queues through ClientHandlerThread.push_item().
Every 10 increments, the number of items in the client queues is printed.
Now to my questions:
When a client disconnects, the Thread terminates and no more data is send, however, the ClientThread object remains in the ClientHandlerThreads list of clients and items are continuously pushed to its queue.
I'm therefore looking for either (1) a way to delete the ClientThread object from the list whenever a client disconnects, (2) a better way to monitor the ClientThreads than a list or (3) a different (better) architecture to archive my goal.
Many thanks!
Server
import socket
import time
from threading import Thread
from queue import Queue
class ClientThread(Thread):
def __init__(self, conn, client_addr):
Thread.__init__(self)
self.queue = Queue()
self.conn = conn
self.client_addr = client_addr
def run(self):
print('Client connected')
while True:
try:
self.conn.sendall(self.queue.get().encode('utf-8'))
time.sleep(1)
except BrokenPipeError:
print('Client disconnected')
break
class ClientHandlerThread(Thread):
def __init__(self):
Thread.__init__(self, daemon = True)
self.clients = list()
def push_item(self, item):
for client in self.clients:
client.queue.put(str(i))
def run(self):
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
s.bind('./socket')
s.listen()
i = 1
while True:
conn, client_addr = s.accept()
client = ClientThread(conn, client_addr)
client.start()
self.clients.append(client)
i += 1
if __name__ == '__main__':
client_handler = ClientHandlerThread()
client_handler.start()
i = 1
while True:
client_handler.push_item(str(i))
if i % 10 == 0:
print('[' + ', '.join([str(client.queue.qsize()) for client in client_handler.clients]) + ']')
i += 1
time.sleep(1)
Client:
import socket
if __name__ == '__main__':
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
s.connect('./socket')
print('Connected to server')
while True:
data = s.recv(1024)
if not data:
print('Disconnected from server')
break
print(data.decode('utf-8'))
Note You should probably read up on things like aiohttp for much more scalable versions to your answer.
For your question, you can make a few changes to achieve this:
First, change ClientThread's constructor:
class ClientThread(Thread):
def __init__(self, client_handler, conn, client_addr):
self.client_handler = client_handler
self.running = True
...
When the handler creates the object, it should pass self to it as client_handler.
In the run method, use
def run(self):
while True:
...
self.running = False
self.client_handler.purge()
That is, it marks itself as not running, and calls the purge method of handler. This can be written as
class ClientHandlerThread(Thread):
...
def purge(self):
self.clients = [c for c in self.clients if c.running]
I'm working on a Python based multi-client voice chat application (learning project). While voice transfer between 2 clients works like a charm and the server is also able to handle multiple connections, problems start as soon as 2 persons talk simultaneously. Whenever a client sends audio data (e.g. "AAAA"), while another client is sending "BBBB" the third clients receives something like "ABABABAB".
I've already tried to spawn multiple threads for each user and let a Mutex take control over the clients[] array (the latter one was either implemented wrong or a stupid idea since it doesn't change anything)
I've also included RMS (only send data while somebody is speaking) in order to stop the constant data stream from every connected client, which took a bit of heat from the server and helped a little bit without solving the real problem.
from socket import *
from threading import Thread, Lock
class entry_thread (Thread):
def __init__(self, host, port):
Thread.__init__(self)
adress_voice = (host, port)
self.voice_socket = socket(AF_INET, SOCK_STREAM)
self.voice_socket.bind(adress_voice)
self.voice_socket.listen(10)
def run(self):
while True:
sock_v, data_v = self.voice_socket.accept()
thread = handle_client_thread(sock_v, data_v)
thread.start()
class handle_client_thread (Thread):
def __init__(self, vsock, vdata):
global clients
Thread.__init__(self)
self.chunk = 1024
self.vsock = vsock
self.vdata = vdata
self.name = self.vsock.recv(1024).decode()
clients[self.name] = [self.vdata, self.vsock]
def run(self):
global clients
while True:
try:
audio_data = self.vsock.recv(self.chunk)
for x, client in clients.items():
if thread_safe:
mutex.acquire()
try:
if (client[0][1] != self.vdata[1]): #if receipient
# != sender
client[1].send(audio_data)
finally:
if thread_safe:
mutex.release()
except:
break
mutex = Lock()
thread_safe = False
clients = {}
server = entry_thread('', 20003)
server.start()
server.join()
Solved the issue by putting the whole for-loop into the mutex, while giving each user an individual audio channel within the client.
I am trying to create a multi-threaded TCP server in python.
When a new client is accepted, it goes to a new created thread.
However, when earlier clients send data, it looks like the newly created thread intercept them, so that it look like from server side that only newer clients are speaking!
Here is my code:
Nbre = 1
class ClientThread(threading.Thread):
def __init__(self, channel, connection):
global Nbre
Nbre = Nbre + 1
print("This is thread "+str(Nbre)+" speaking")
self.channel = channel
self.connection = connection
threading.Thread.__init__(self)
def run(self):
print(connection[0]+':'+str(connection[1])+'<'+str(Nbre)'> Connected!')
try:
while True:
data = self.channel.recv(1024)
if data:
print(connection[0]+':'+str(connection[1])+'<'+str(Nbre)+'> '+str(data.strip('\n')))
else:
break
finally:
print(connection[0]+':'+str(connection[1])+'<'+str(Nbre)+'> Exited!')
self.channel.close()
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind(('', 4747))
server.listen(0)
while True:
channel, connection = server.accept()
ClientThread(channel, connection).start()
Here is what I got when launched and with telnet clients sending "Hello" for the first client, "Bonjour" for the second one :
$ python simple_thread.py
This is thread 2 speaking # new connection
127.0.0.1:33925> Connected!
127.0.0.1:33925<2> hello
This is thread 3 speaking # new connection
127.0.0.1:33926> Connected!
127.0.0.1:33926<3> Bonjour # last connected says "Bonjour" (ok here)
127.0.0.1:33926<3> hello # first connected re-send "hello" but in thread 3?!
Why is secondly sent "hello" not poping from thread 2? And how to make it happened, so that I can reply from server side to the appropriate client?
The good news is, it probably works, but your logging is broken. Here you use Nbre, which is the number of threads, not the number of the current thread:
print(connection[0]+':'+str(connection[1])+'<'+str(Nbre)+'> '+str(data.strip('\n')))
So to make this work, store the current thread's number on the thread object, and use that instead. Something like this:
def __init__(self, channel, connection):
global Nbre
Nbre = Nbre + 1
self.number = Nbre
There is a similar problem with the connection object. The logging is using connection instead of self.connection. Normally you'd get an error, but because the while loop at the bottom creates a global connection variable, that one gets picked up instead. So use self.connection in the logging instead.
For your own sanity, I'd also recommend extracting the logging into a function, and using string.format:
def log(self, message):
print('{}:{}<{}> {}'.format(self.connection[0], str(self.connection[1]), self.number, message)
So you can just write self.log('Thread started') instead of repeating the log format each time.
I have a problem concerning threading in python (2.7.8). The problem resides in a python "chat" code written (I'll include the code to my threading class and my client class (different python files), since I think the problem is in one of those two, and not in the server code). When I run the Client.py file, I am able to communicate with another client (running the same python code) through a server, but the problem is that I have to refresh the .send_payload(msg) command in order to receive the message that the other client has sent (or simply by pressing enter in the chat, and hence sending "" as message). I want to know if it is possible to receive messages without "refreshing" the chat, somehow through threading.
class MessageReceiver(Thread):
def __init(self,client,connection):
Thread.__init__(self)
self.daemon = True
self.client = client
self.connection = connection
self.stop = False
def run(self):
while not self.stop:
data = self.connection.recv(8192)
if not data:
break
else:
self.client.receive_message(data)
pass
class Client:
def __init__(self, host, server_port):
self.host = host
self.server_port = server_port
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.run()
def run(self):
self.connection.connect((self.host, self.server_port))
self.thread = MessageReceiver(self, self.connection)
self.thread.start()
while True:
text = raw_input()
if(text.find("login") == 0):
data={"request":"login","content":text[text.find("login")+6:]}
self.send_payload(data)
if(text.find("names") == 0):
data={"request":"names","content":""}
self.send_payload(data)
else:
data={"request":"msg","content":text}
self.send_payload(data)
def disconnect(self):
self.thread.stop = True
self.connection.close()
print("Disconnected")
def receive_message(self, message):
print("Msg: " + message)
def send_payload(self, data):
self.connection.send(json.dumps(data))
print("Payload sent!")