Develop Reference

Using one socket in UDP chat using threading

I am working on UDP chat which should be listening and being able to send message any time using only one socket. Example, I will have the chat program done, I will open it first time, then second time and I must be able to communicate over UDP from both programs, simply each program has only one opened socket.
My two threads are for listening, which is deamon thread, because I want it to listen to new messages nonstop, and my other is sending the messages, which is just like a normal thread.
First of all, my problem is that it looks like my threads are blocking each other, because if I run the program, I only get output from the first thread I start.
Second problem is that I am not sure if my sending function or the entire class is written properly, or if there is something missing or incorrect.
Thanks in advance. Btw, I am new into python and I am using python 3, just to make it clear.
import socket
import threading
import logging
import time
from sys import byteorder
class Sending():
def __init__(self, name, tHost, tPort):
self.name = name
self.host = tHost
self.port = tPort
def set_name(self, name):
self.name = name
def send(self, name, tHost, tPort, msgType, dgramSize):
logging.debug('Starting send run')
message = input('Enter message: ')
data = bytearray()
data.extend( (name.encode('utf-8'), message.encode('utf-8'), msgType.to_bytes(1, byteorder = 'little')) )
#data.extend(message.encode(encoding='utf_8'))
self.sock.sendto(bytearray(data), (tHost, tPort))
def run(self):
th2 = threading.Thread(name = 'send', target=self.send('username', 'localhost', 8001, 1, 1400))
th2.start()
class Receiving():
def __init__(self, host, port):
self.host = host
self.port = port
def create_socket(self, host, port):
logging.debug('Starting socket')
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((host, port))
#print ('socket ready')
time.sleep(5)
while True:
data, addr = sock.recvfrom(1500)
print('Prijata:' + data + addr)
def run(self):
th1 = threading.Thread(name = 'rec', target=self.create_socket('localhost', 8000))
th1.setDaemon(True)
th1.start()
if __name__ == '__main__':
#print ('running')
rec = Receiving('localhost', 8000)
send = Sending('username', 'localhost', 8001)
send.run()
rec.run()

Congrats on your introduction to Python! It looks like you're using Python 3, and in future questions it's helpful if you are explicit about which version you're using because there are minor but program-breaking incompatibilities in some code (including this code!).
I found a few errors in your program:
The most major issue - as Trevor Barnwell says, you're not calling threading.Thread quite correctly. The target= argument needs to be a callable object (i.e. function), but in this case it should just be a reference to the function. If you add brackets to the function, self.create_socket(host, port) as you have above, it actually runs the function immediately. As Trevor explained, your Sending.send() method was called early, but additionally there was a similar bug in Receiving. Because Receiving.create_socket() creates an infinite loop, it never returns program execution. While the console output looks correct to the user, the actual program execution has never made it to running the listener in a separate thread.
bytearray.extend() takes an iterable of ints, what you're passing right now is a tuple of byte objects.
In Sending.send() you call self.sock, but you never assign self.sock a value, so it fails.
Sending.run() only runs Sending.send() one time. After completing input for the user, it immediately exits, because the program has finished.
If you're looking for an in-depth, project based introduction to Python appropriate for an experienced programmer (including an exercise very similar to this question on basic sockets, and another on threading), I highly recommend you check out Wesley Chun's "Core Python Applications Programming". The most recent edition (3rd) has a lot of Python 2 code, but it's easily portable to Python 3 with some minor work on the reader's part.
I tried to modify your code as little as possible to get it working, here it is:
import socket
import threading
import logging
import time
class Sending():
def __init__(self, name, tHost, tPort, target):
self.name = name
self.host = tHost
self.port = tPort
self.target_port = target
self.sock = self.create_socket()
def create_socket(self):
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((self.host, self.port))
return sock
def set_name(self, name):
self.name = name
def send_loop(self):
while True:
logging.debug('Starting send run')
message = input('Enter message: ')
data = bytearray()
data.extend(message.encode('utf-8'))
self.sock.sendto(bytearray(data), (self.host, self.target_port))
def run(self):
th2 = threading.Thread(name='send', target=self.send_loop)
th2.start()
class Receiving():
def __init__(self, host, port):
self.host = host
self.port = port
def create_socket(self):
logging.debug('Starting socket')
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((self.host, self.port))
print ('socket ready')
time.sleep(5)
while True:
data, addr = sock.recvfrom(1500)
print('\nPrijata:' + data.decode('utf-8') + str(addr))
def run(self):
th1 = threading.Thread(name='rec', target=self.create_socket)
print("Made it here")
th1.daemon = True
th1.start()
return
if __name__ == '__main__':
print('running')
rec = Receiving('localhost', 8000)
send = Sending('username', 'localhost', 8001, 8000)
rec.run()
send.run()

The threads are not blocking each other. send is called before a thread is even created.
th2 = threading.Thread(name = 'send', target=self.send('username', 'localhost', 8001, 1, 1400))
This line makes a call to send at:
self.send('username', 'localhost', 8001, 1, 1400)
I think you meant to do this:
th2 = threading.Thread(
target=self.send
args=('username', 'localhost', 8001, 1, 1400))
That way a thread will start that calls send on the next line.
Two other things:
You will want to loop in your functions because the thread terminates once the function does.
I think you mean raw_input instead of input

How to implement multi-threading into a Python WSGI server

Just to clarify up front that this is just a learning project and I have no intention of using this in production. There are several very good Python application servers out there already. But I am trying to learn more about concurrency, so I set out to write one of the things (I thought) I knew.
Also, because I wanted to "closer to the metal" so I started out with just Socket and want to keep it that way.
Below is the important parts of what I have so far. self.iq is a Queue object (inbound_queue) which then does nothing really, but puts the request (which includes the socket object) into the outbound_queue and then a Consumer object takes the request from the outbound_queue and passes it to the ResponseHandler. This seems to work fine with just me hitting it but I am concerned that I am opening myself up to a race condition with a naive implementation. Specifically assigning things to the ServerClass object that are request specific.
So the question is: Is there a better way to do this, or does my Queue usage prevent two threads from picking up and operating on the same object? Should I be encapsulating things like the WSGI environment stuff into a separate object that can be also passed into the queue? Doing stuff like that gets tricky in trying to write a server that is WSGI compliant because of the need to pass in callback functions.
class Consumer(threading.Thread):
def __init__(self, out_queue, server):
threading.Thread.__init__(self)
self.out_queue = out_queue
self.server = server
def run(self):
while True:
item = self.out_queue.get()
self.server.ResponseHandler(self.server, item)
self.out_queue.task_done()
class QueueConsumerServer(object):
methods_allowed = ['get', 'post', 'put', 'patch', 'delete', 'options', 'upgrade']
def __init__(self, host, port, application):
self.host = host
self.port = port
self.application = application
self.iq = Queue.Queue()
self.oq = Queue.Queue()
self.socket = socket.socket()
self.socket.bind((self.host, self.port))
#<snip of lots of assigning stuff to environ>
self.environ = environ
headers_set = []
headers_sent = []
for i in xrange(3):
thr = Producer(self.iq, self.oq)
thr.daemon = True
thr.start()
for i in xrange(3):
thr = Consumer(self.oq, self)
thr.daemon = True
thr.start()
def handle_request(self):
self.socket.listen(1)
try:
while True:
cli, addr = self.socket.accept()
data = cli.recv(1024)
request_data = self.parse_request_data(data)
req.path = request_data[1]
req.cli = cli
self.iq.put(req)
return
except Exception, ex:
print 'e', ex,
sys.exit(1)
finally:
sys.stdout.flush()
self.socket.close()

Python queues are thread safe, so there is no race in your code as written.
Regarding a better way to approach this, your code will probably hit the GIL before too long. I would suggest looking to into multiprocessing.

send a variable to a TCPHandler in python

I'm confused as to how to send a variable to a TCPHandler using SocketServer.TCPServer in python..
HOST, PORT = hosts[0], args.port
server = SocketServer.TCPServer((HOST, PORT), METCPHandler)
server.serve_forever()
Which calls:
class METCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
print "{} wrote:".format(self.client_address[0])
r = MExpressHandler(self.data, False)
But I want to pass a debug boolean to MExpressHandler.. so
HOST, PORT = hosts[0], args.port
server = SocketServer.TCPServer((HOST, PORT), METCPHandler(debug))
server.serve_forever()
Fails. Whats the correct way of doing this? Do I have to recreate a whole TCPHandler over-ridding __init__?

Trust your instincts, the correct way is indeed to subclass TCPServer and override the __init__ method, but Python makes this very easy!
import SocketServer
class DebugTCPServer(SocketServer.TCPServer):
def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True, debug=True):
self.debug = debug
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass, bind_and_activate=True)
class DebugMETCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
# self.server is an instance of the DebugTCPServer
DEBUG = self.server.debug
self.data = self.request.recv(1024).strip()
if DEBUG:
print "{} wrote:".format(self.client_address[0])
r = MExpressHandler(self.data, False)
server = DebugTCPServer((HOST, PORT), DebugMETCPHandler, debug=True)
or since we specified debug=True as the default:
server = DebugTCPServer((HOST, PORT), DebugMETCPHandler)

As I proposed in this post already, it is possible to do it without sub-classing the TCPServer. It is in fact more concise and generic.
You can give parameters to your handler this way:
class METCPHandler(SocketServer.BaseRequestHandler):
def __init__(self, debug):
self.debug = debug
def __call__(self, request, client_address, server):
h = METCPHandler(self.debug)
SocketServer.StreamRequestHandler.__init__(h, request, client_address, server)
You can now give an instance of your handler to the TCPServer:
SocketServer.TCPServer((HOST, PORT), METCPHandler(True))
The TCPServer normally creates a new instance of METCPHandler per request but in this case, the __call__ method will be called instead of the constructor (it is already an instance.)
In the call method, I explicitly make a copy of the current METCPHandler and pass it to the super constructor to conform to the original logic of "one handler instance per request".
It is worth having a look at the SocketServer module to understand what happens here: https://github.com/python/cpython/blob/2.7/Lib/SocketServer.py

Sending message from one server to another in Twisted

I'm a complete Twisted AND Python noob, so my apologies if any of my terminology is wrong or anything I've done is silly. Nonetheless....
I've implemented my servers in the following way:
def makeServer(application, port):
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
tempServer = internet.TCPServer(port, factory)
tempServer.setServiceParent(application)
return tempServer
application = service.Application("chatserver")
server1 = makeServer(application, port=1025)
server2 = makeServer(application, port=1026)
server3 = makeServer(application, port=1027)
Note that MyChat is an event handling class that has a "receiveMessage" action:
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.transport.write(message + '\n')
I want server1 to be able to pass messages to server2. Rather, I want server1 to be treated as a client of server2. If server1 receives the message "hi" then I want it to send that same exact message to server2. The only thing server1 needs to be able to do is to send the message it received from its client to server2.
How can I accomplish this?
NOTE: You can totally change the way I'm implementing my server if it helps.

Different parts of your application can interact with each other using method calls.
Send a message to server2 really just means Call a method on one of the objects related to server2.
For example, in MyChat, you might have:
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.transport.write(message + '\n')
for server in self.factory.otherServers:
server.otherServerMessage(self, line)
This supposes a couple things:
You add a new otherServers attribute to your factory. Its contents are objects related to the other listening servers you have set up. These might be factory objects or protocol objects. It depends on what's most convenient based on what you intend to do with the message.
You give those related objects a new method, otherServerMessage, to handle messages delivered this way. If you were to deliver the messages directly to MyChat.lineReceived (which you easily could, if you wanted) then I would expect you to end up with infinite recursion; having a different method lets you differentiate between messages received from a client and messages received from another server.

You will probably need to implement a separate client. It is possible that an object can be both a client and a server, but I doubt it will be worth it and you are likely to run into trouble.
I suggest that the server instantiates a client object, which you connect to the 'next' server. The client can for example be an instance variable on the server.
Example:
class MyChat(LineReceiver):
def connectionMade(self):
print "Proxy: connected"
factory = protocol.ClientFactory()
class Proxy(protocol.Protocol):
def relayMessage(self, msg):
self.transport.write(msg)
factory.protocol = Proxy
point = TCP4ClientEndpoint(reactor, "localhost", 1025)
conn = point.connect(factory)
conn.addCallback(self.hasConnection)
def hasConnection(self, client):
print "Proxy: Connected to relay", client
self.client = client
def lineReceived(self, line):
print "Proxy: received", repr(line)
self.client.transport.write(line+"\n")
class MyEcho(LineReceiver):
def lineReceived(self, line):
print "Echo: received", repr(line)
factory = protocol.ServerFactory()
factory.protocol = MyChat
reactor.listenTCP(1024, factory)
factory = protocol.ServerFactory()
factory.protocol = MyEcho
reactor.listenTCP(1025, factory)

You need just declare clients inside your server, like this:
factory = SomeClientFactory('ws://127.0.0.1')
connectWS(factory)
and in your Client Class:
class SomeClient(WebSocketClientProtocol):
def __init__(self):
pass
def sendCommand(self):
self.sendMessage('A message to another server')
def onOpen(self):
self.sendCommand()
def onClose(self, wasClean, code, reason):
print(reason)
def onMessage(self, payload, isBinary):
print('A answer from another server')
class SomeClientFactory(WebSocketClientFactory):
def __init__(self, url):
WebSocketClientFactory.__init__(self,url)
self.proto = DeltaClient()
self.proto.factory = self
def buildProtocol(self, addr):
return self.proto
Tip: use a "Controller" class to manage that instances of clients inside your servers.

python asynchat state

I've just started working with the basics of python socket networking. As an exercise in understanding, I've been trying to hash out a basic server that will ask it's client for a file type, and upon receiving a string of the extension, ask for the actual file. I've found numerous tutorials online that use the asyncore library, specifically asynchat to setup this kind of call and response functionality.
The most basic one I've been following can be found here (I've copied it)
http://effbot.org/librarybook/asynchat.htm
import asyncore, asynchat
import os, socket, string
PORT = 8000
class HTTPChannel(asynchat.async_chat):
def __init__(self, server, sock, addr):
asynchat.async_chat.__init__(self, sock)
self.set_terminator("\r\n")
self.request = None
self.data = ""
self.shutdown = 0
def collect_incoming_data(self, data):
self.data = self.data + data
def found_terminator(self):
if not self.request:
# got the request line
self.request = string.split(self.data, None, 2)
if len(self.request) != 3:
self.shutdown = 1
else:
self.push("HTTP/1.0 200 OK\r\n")
self.push("Content-type: text/html\r\n")
self.push("\r\n")
self.data = self.data + "\r\n"
self.set_terminator("\r\n\r\n") # look for end of headers
else:
# return payload.
self.push("<html><body><pre>\r\n")
self.push(self.data)
self.push("</pre></body></html>\r\n")
self.close_when_done()
class HTTPServer(asyncore.dispatcher):
def __init__(self, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind(("", port))
self.listen(5)
def handle_accept(self):
conn, addr = self.accept()
HTTPChannel(self, conn, addr)
#
# try it out
s = HTTPServer(PORT)
print "serving at port", PORT, "..."
My question has to do with the handle_accept method of the HTTPServer class. If every time a request comes in, the HTTPChannel object is initialized, wouldn't it be impossible in this kind of setup to create a call and response? I was thinking one could set flags for _hastype and _hasfile in the channel object, but since the accept inits it for each individual connection, the object's state is forgotten with every inidividual request. I realize this setup is supposed to be a basic HTTPServer, but my question is, how could I edit it to setup something like what I've described? Would the server object need to inherit asynchat itself and forego dispatcher completely? The channel object would have to have some state to know that the filetype has already been sent, and then ask for the binary of the file instead. I'm very curious to know what the cleanest possible implementation of this might look like.
Thanks a ton - I'm very new to sockets. Please let me know if I haven't been clear.

Normally the connection would be kept open after it's initially created, so all the parts of the communication from the same client go to the same HTTPChannel object - accept is only called when a new connection is created.