Develop Reference

Python thread won't start using socket

i have a problem with my progam using socket and thread.
I have made a socket server who add client in a thread, but the client thread never start...
here is my code:
socket server
import socket, threading, logging, sys
from client_thread import ClientThread
class SocketServer:
CLIENTS = list()
def __init__(self, server_ip, server_port, max_connections):
try:
self.tcpsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.tcpsock.bind((server_ip, server_port))
self.tcpsock.listen(10)
logging.info('Socket server successfully started !')
except Exception as e:
logging.error(format(e))
def start(self):
from src.realmserver.core.hetwan import EmulatorState, Core
while (Core.STATE == EmulatorState.IN_RUNNING):
try:
(clientsock, (ip, port)) = self.tcpsock.accept()
new_client = threading.Thread(target=ClientThread, args=[len(self.CLIENTS), ip, port, clientsock])
self.CLIENTS.append(new_client)
new_client.start()
except Exception as e:
print format(e)
for client in self.CLIENTS:
client.join()
and client thread
import logging, string, random
class ClientThread:
def __init__(self, client_id, client_ip, client_port, socket):
self.client_id = client_id
self.client_ip = client_ip
self.client_port = client_port
self.socket = socket
logging.debug('(%d) Client join us !', client_id)
def run(self):
key = ''.join(random.choice(string.ascii_uppercase + string.ascii_lowercase + string.digits) for _ in range(32))
print self.send('HC%s' % key)
while True:
entry = self.socket.recv(4096)
entry.replace("\n", "")
if not entry:
break
else:
logging.debug('(%d) Packet received : %s', self.client_id, str(entry))
self.kill()
def send(self, packet):
return self.socket.send("%s\x00" % packet)
def kill(self):
self.socket.close()
logging.debug('(%d) Client is gone...', self.client_id)
sorry for bad indentation, it's the form, not my file.
Please help me :(
Thank you in advance (sorry for bad english i'm french....)

You have this line of code in your Server instance start function:
new_client = threading.Thread(target=ClientThread,
args=[len(self.CLIENTS), ip, port, clientsock])
The target= argument to threading.Thread needs to be a callable function. Here ClientThread is the name of the constructor function for your class ClientThread, so it is a callable function, returning an instance of that class. Note that it is not actually called yet! The args= argument is more normally a tuple, but a list actually works. These are the arguments that will be passed to the target function once it's eventually called, when you use this particular threading model. (You can also pass keyword arguments using kwargs= and a dictionary.)
What happens now is a bit tricky. Now that the two parameters (target= and args=) have been evaluated, the Python runtime creates a new instance of a threading.Thread class. This new instance is, at the moment, just a data object.
If we add a print statement/function (it's not clear whether this is py2k or py3k code) we can see the object itself:
print('new_client id is', id(new_client))
which will print something like:1
new_client id is 34367605072
Next, you add this to a list and then invoke its start:
self.CLIENTS.append(new_client)
new_client.start()
The list add is straightforward enough, but the start is pretty tricky.
The start call itself actually creates a new OS/runtime thread (whose ID is not related to the data object's ID—the raw thread ID is an internal implementation detail). This new thread starts running at its run method.2 The default run method is in fact:3
try:
if self.__target:
self.__target(*self.__args, **self.__kwargs)
finally:
# Avoid a refcycle if the thread is running a function with
# an argument that has a member that points to the thread.
del self.__target, self.__args, self.__kwargs
Since you are using a regular threading.Thread instance object, you are getting this default behavior, where new_thread.start() creates the new thread itself, which then calls the default run method, which calls its self.__target which is your ClientThread class-instance-creation function.
So now, inside the new thread, Python creates an instance of a ClientThread object, calling its __init__ with the self.__args and self.__kwargs saved in the new_thread instance (which is itself shared between the original Python, and the new thread).
This new ClientThread object executes its __init__ code and returns. This is the equivalent of having the run method read:
def run(self):
ClientThread(**saved_args)
Note that this is not:
def run(self):
tmp = ClientThread(**saved_args)
tmp.run()
That is, the run method of the ClientThread instance is never called. Only the run method of the threading.Thread instance is called. If you modify your ClientThread's __init__ method to print out its ID, you will see that this ID differs from that of the threading.Thread instance:
class ClientThread:
def __init__(self, client_id, client_ip, client_port, socket):
print('creating', id(self), 'instance')
which will print a different ID (and definitely print after the new_client id is line):
new_client id is 34367605072
creating 34367777464 instance
If you add additional prints to your run method you will see that it is never invoked.
What to do about this
You have two main options here.
You can either make your ClientThread a subclass of threading.Thread:
class ClientThread(threading.Thread):
def __init__(self, client_id, client_ip, client_port, socket):
...
threading.Thread.__init__(self)
In this case, you would create the client object yourself, rather than using threading.Thread to create it:
new_thread = ClientThread(...)
...
new_thread.start()
The .start method would be threading.Thread.start since you have not overridden that, and that method would then create the actual OS/runtime thread and then call your run method, which—since you did override it—would be your run.
Or, you can create a standard threading.Thread object, supply it with a target, and have this target invoke your object's run method, e.g.:
new_client = ClientThread(...)
new_thread = threading.Thread(target=new_client.run, ...)
...
new_thread.start()
The choice is yours: to subclass, or to use separate objects.
1The actual ID is highly implementation-dependent.
2The path by which it reaches this run function is somewhat convoluted, passing through bootstrap code that does some internal initialization, then calls self.run for you, passing no arguments. You are only promised that self.run gets entered somehow; you should not rely on the "how".
3At least, this is the code in Python 2.7 and 3.4; other implementations could vary slightly.

python asynchat: how to store information on individual connections, and how to know when a client disconnects

For fun, I'm writing a minimal IRC server with asynchat. I'm trying to clear up a few fundamentals (my specific questions follow the code). I've decided not to use anything in Twisted just so I can implement a little more myself. First, the code I have:
import asyncore,asynchat
import socket
class Connection(asynchat.async_chat):
def __init__(self, server, sock, addr):
asynchat.async_chat.__init__(self, sock)
self.set_terminator('\n')
self.data = ""
print "client connecting:",addr
# do some IRC protocol initialization stuff here
def collect_incoming_data(self, data):
self.data = self.data + data
def found_terminator(self):
print self.data
self.data = ''
class Server(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind((host, port))
self.listen(5)
def handle_accept(self):
conn, addr = self.accept()
Connection(self, conn, addr)
def handle_close(self):
self.close()
s = Server('127.0.0.1',5006)
asyncore.loop()
So, in my mind, this code structure is similar to a Twisted client factory: the Server class is initialized once and basically instantiates Connection every time a client connects. First question: is the best way to keep track of all connected clients by storing all of the Connections in a list within Server?
Also, I don't understand how I am to know when a specific client closes their connection to my socket? Connection implements asynchat (and by extension asyncore) but adding the handle_close() callback to the Connection class doesn't fire when a client disconnects. It seems to be only for when the bound socket on the server is destroyed. I don't see any methods for this purpose. This socket always stays open, whether or not clients connect, right?

to handle client side closed connections check the handle_error method, does your client issue a clean close connection?
handle_error() :Called when an exception is raised and not otherwise handled. The default version prints a condensed traceback.
hope it helps.

how to send data with twisted protocol via factory

I'm writing a client implementing a custom protocol, and have a factory for it. My problem is the following: my client has bi-dir communication, and sometimes I want to tell it "send this data". But all I have is the factory object:
class MyFactory(ClientFactory):
protocol = MyProtocol
def __init__(self, recv_callback):
self.recv_callback = recv_callback
def send_message(self, msg):
self.protocol.send_message(msg)
So I create a factory and have a factory object, I don't the protocol object. When send_message above is called I get an error because self.protocol is just a class, not an object.
How can I do this? Should I also expose the protocol for connection in addition to the factory?
Thanks

You have access to all of the objects you want. The factory is responsible for creating protocol instances, so if you want to keep the protocol instance around where the factory can use it, override buildProtocol and save the instance:
class MyFactory(ClientFactory):
protocol = MyProtocol
...
def buildProtocol(self, address):
proto = ClientFactory.buildProtocol(self, address)
self.connectedProtocol = proto
return proto
However, this approach is lacking in one important feature. It does not make it easy to tell when buildProtocol has been called and connectedProtocol has been set. If you try to use this attribute naively:
factory = MyFactory()
reactor.connectTCP(host, port, factory)
factory.connectedProtocol.send_message(...)
The code will fail with an AttributeError because the connection has not yet actually been set up. Since Twisted is event driven, you need to make sure to use this code by responding to an event that says the connection has been set up.
You might do this by firing a callback when the protocol is constructed instead of just setting an attribute. Twisted actually has a helper factory which does something like this already:
from twisted.internet.protocol import ClientCreator
cc = ClientCreator(reactor, MyProtocol)
whenConnected = cc.connectTCP(host, port)
# Or the equivalent with endpoints
# from twisted.internet.endpoints import TCP4ClientEndpoint
# from twisted.internet.protocol import ClientFactory
# endpoint = TCP4ClientEndpoint(reactor, host, port)
# factory = ClientFactory()
# factory.protocol = MyProtocol
# whenConnected = endpoint.connect(factory)
def cbConnected(connectedProtocol):
connectedProtocol.send_message(...)
def ebConnectError(reason):
# Connection attempt failed, perhaps retry
...
whenConnected.addCallbacks(cbConnected, ebConnectError)
You could also save the reference to connectedProtocol in the cbConnected callback so that you can continue to use it later on. You might also start whatever other operations want to use the connected protocol in cbConnected, so that they don't try to use the connection before it is actually available.

Casting sockets to subtypes

I'm trying to create my own subclass of socket.socket that will be able to handle custom messages. So far my code looks like this:
self._sockets.append(s)
logging.debug("Waiting for incoming connections on port %d" % (port))
while not self.shutdown:
inputready,outputready,exceptready = select(self._sockets,[],[])
print "Select returned"
for i in inputready:
if s == i:
# handle the server socket
client, address = s.accept()
self._sockets.append(client)
print "%r , %r" % (client, address)
else:
# handle all other sockets
s.handleMessage()
so as you can see I'm either acceptin new connections or if it returned from another socket it'll call handleMessage on that socket. Now the problem is that off course socket.accept() will return a socket.socket and not my subclass which implements the handleMessage function.
What would be the easiest way to get my custom class instead of the default socket.socket?

From your description it appears that you are making a message handler that has-a socket (or sockets). When designing classes has-a indicates composition and delegation while is-a can indicate inheritance.
So it is not appropriate to inherit from socket.socket, and your code is already looking a bit hybrid. Something like this really coarse pseudo-code is probably best suited to the task:
class MyMessageHandler(object):
def __init__(self):
self.sockets = [...]
def wait(self):
debug('waiting...')
i, o, e = select(...)

How difficult is it to setup a dictionary mapping socket descriptors to your socket wrapper objects?

How to deliver instance of object to instance of SocketServer.BaseRequestHandler?

This is problem.
My primary work is : deliver "s" object to "handle" method in TestRequestHandler class.
My first step was : deliver "s" object through "point" method to TestServer class, but here im stuck. How to deliver "s" object to TestRequestHandler? Some suggestions?
import threading
import SocketServer
from socket import *
class TestRequestHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server):
SocketServer.BaseRequestHandler.__init__(self, request, client_address, server)
return
def setup(self):
return SocketServer.BaseRequestHandler.setup(self)
def handle(self):
data = self.request.recv(1024)
if (data):
self.request.send(data)
print data
def finish(self):
return SocketServer.BaseRequestHandler.finish(self)
class TestServer(SocketServer.TCPServer):
def __init__(self, server_address, handler_class=TestRequestHandler):
print "__init__"
SocketServer.TCPServer.__init__(self, server_address, handler_class)
return
def point(self,obj):
self.obj = obj
print "point"
def server_activate(self):
SocketServer.TCPServer.server_activate(self)
return
def serve_forever(self):
print "serve_forever"
while True:
self.handle_request()
return
def handle_request(self):
return SocketServer.TCPServer.handle_request(self)
if __name__ == '__main__':
s = socket(AF_INET, SOCK_STREAM)
address = ('localhost', 6666)
server = TestServer(address, TestRequestHandler)
server.point(s)
t = threading.Thread(target=server.serve_forever())
t.setDaemon(True)
t.start()

If I understand correctly, I think you perhaps are misunderstanding how the module works. You are already specifying an address of 'localhost:6666' for the server to bind on.
When you start the server via your call to serve_forever(), this is going to cause the server to start listening to a socket on localhost:6666.
According to the documentation, that socket is passed to your RequestHandler as the 'request' object. When data is received on the socket, your 'handle' method should be able to recv/send from/to that object using the documented socket API.
If you want a further abstraction, it looks like your RequestHandler can extend from StreamRequestHandler and read/write to the socket using file-like objects instead.
The point is, there is no need for you to create an additional socket and then try to force your server to use the new one instead. Part of the value of the SocketServer module is that it manages the lifecycle of the socket for you.
On the flip side, if you want to test your server from a client's perspective, then you would want to create a socket that you can read/write your client requests on. But you would never pass this socket to your server, per se. You would probably do this in a completely separate process and test your server via IPC over the socket.
Edit based on new information
To get server A to open a socket to server B when server A receives data one solution is to simply open a socket from inside your RequestHandler. That said, there are likely some other design concerns that you will need to address based on the requirements of your service.
For example, you may want to use a simple connection pool that say opens a few sockets to server B that server A can use like a resource. There may already be some libraries in Python that help with this.
Given your current design, your RequestHandler has access to the server as a member variable so you could do something like this:
class TestServer(SocketServer.TCPServer):
def point (self, socketB):
self.socketB = socketB # hold serverB socket
class TestRequestHandler(SocketServer.BaseRequestHandler):
def handle(self):
data = self.request.recv(1024)
if (data):
self.request.send(data)
print data
self.server.socketB ... # Do whatever with the socketB
But like I said, it may be better for you to have some sort of connection pool or other object that manages your server B socket such that your server A handler can just acquire/release the socket as incoming requests are handled.
This way you can better deal with conditions where server B breaks the socket. Your current design wouldn't be able to handle broken sockets very easily. Just some thoughts...

If the value of s is set once, and not reinitialized - you could make it a class variable as opposed to an instance variable of TestServer, and then have the handler retrieve it via a class method of TestServer in the handler's constructor.
eg: TestServer._mySocket = s

Ok, my main task is this. Construction of the listening server (A-server - localhost, 6666) which during start will open "hard" connection to the different server (B-server - localhost, 7777).
When the customer send data to the A-server this (A-server) sends data (having that hard connection to the B-server) to B-server, the answer receives from the B-server to A-server and answer sends to the customer.
Then again : the customer sends data, A-server receives them, then sends to the B-server, the answer receives data from the B-server and A-server send data to the customer.
And so round and round. The connection to the B-server is closes just when the server A will stop.
All above is the test of making this.