As far as I know twisted is asynchronous and event driven and someone told me their is no need for timeout. I have to build a server application which will be connected to more than 100 clients which are embedded machines sending data to server every 2 minutes and each packet or data will be of size 238 - 1500 bytes. Thus is real life case tcp will be breaking data into multiple packets so is their any need to implement timeout or twisted will handle such situation. Any advise since I am new twisted. I have following code for my server without timeout. At the end of timeout I just want to discard packet if full packet is not received while connection remains alive.
class Server(LineReceiver):
def connectionMade(self):
self.factory.clients.append(self)
self.setRawMode()
self._peer = self.transport.getPeer()
print 'Connected Client', self._peer
def connectionLost(self, reason):
self.factory.clients.remove(self)
print 'Lost connection from', self._peer
def rawDataReceived(self, data):
inputArray = [ord(inp) for inp in data]
#do something
def main():
"""This runs the protocol on port 8000"""
factory = protocol.ServerFactory()
factory.protocol = Server
factory.clients = []
reactor.listenTCP(8000,factory)
reactor.run()
As #Ashish Nitin Patil suggested, just cut the connection to implement the timeout:
from twisted.internet import reactor
# ...
def connectionMade(self):
# ... your code
# cancel connection in 2 minutes
reactor.callLater(120, self.transport.loseConnection)
Or
At the end of timeout I just want to discard packet if full packet is not received while connection remains alive.
If you don't want to cancel the connection on timeout then:
from time import time as timer
def connectionMade(self):
# ... your code
self.endtime = timer() + 120 # timeout in 2 minutes
def dataReceived(self, data):
if timer() > self.endtime: # timeout
if not self.have_we_received_full_packet()
return # do nothing (discard data, the connection remains alive)
else:
# timeout happened but we have a full packet, now what?
inputArray = bytearray(data)
#do something
Related
I just started programming Python.
My goal is to built a digital Picture Frame with three Screens. Therefore I use 3 Raspis, one for each Monitor.
For the communication of these Raspis I need to program a server and a Client.
For a first test I want to built a server which is able to send and receive messages to/from multiple clients.
So I started with a few socket tutorials an created the following program.
Server Class (TcpServer.py)
class TcpServer:
clients = []
serverIsRunning = 0
port = 0
def __init__(self, port):
self.port = port
self.serverIsRunning = 0
self.serverRunning = 0
def startServer (self):
print("start Server...")
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind(("", self.port))
self.server.listen(1)
self.serverRunning = 1
while self.serverRunning:
read, write, oob = select.select([self.server] + self.clients, [], [])
for sock in read:
if sock is self.server:
client, addr = self.server.accept()
self.clients.append(client)
print ("+++ Client ", addr[0], " verbunden")
else:
nachricht = sock.recv(1024)
ip = sock.getpeername()[0]
if nachricht:
print (ip, nachricht)
else:
print ("+++ Verbindung zu ", ip , " beendet")
sock.close()
self.clients.remove(sock)
for c in self.clients:
c.close()
self.clients.remove(c)
self.server.close()
def send(self, message):
message = message.encode()
self.server.send(message)
Client class (TcpClient.py)
import socket
class TcpClient:
def __init__(self, ip, port):
self.serverAdress = (ip, port)
self.connected = 0
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(self.serverAdress)
print ("connectet to ", self.serverAdress)
def send(self, message):
message = message.encode()
self.connection.send(message)
Server:
import threading
import TcpServer
tcpServer = TcpServer.TcpServer(50000)
threadTcpServer = threading.Thread(target = tcpServer.startServer)
threadTcpServer.start()
while True:
tcpServer.send(input("Nachricht eingeben: "))
Client:
import threading
import TcpClient
tcpClient = TcpClient.TcpClient("192.168.178.49", 50000)
while True:
tcpClient.send(input("Nachricht eingeben: "))
I can send messages from the Client to the server, but when I want to send a Message from the server to the client it generates the following error:
BrokenPipeError: [Errno 32] Broken pipe
I assume it is because the server thread blocks the socket while waiting of a incoming message. But I have no idea how to handle this.
How can I program a server who can send and receive messages? Can you recommend a tutorial? I didn't found a tutorial who describes a solution for my problem.
Edit:
Now I tried to solve the problem with the socketserver library, but I still can't solve may problem.
here is my new code for the server:
import socketserver
import threading
import time
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
threadTcpServer = threading.Thread(target = server.serve_forever)
threadTcpServer.start()
print("server started")
time.sleep(10)
print("sending Data")
server.request.sendall("Server is sending...")
it generates the error:
AttributeError: 'TCPServer' object has no attribute 'request'
My goal is to write a server with a thread who receives Data and still be able to send data from a other thread.
Is this even possible with only one socket?
You should use the provided socketserver rather than writing all the handling of sockets and select etc.
There are multiple problems with your code -
1 - The server is trying to write to the listening socket!! The client communication socket is the one that you get from the accept() call and that is the one you have to use for reading and writing.
2 - The client is sending the data and completing immediately, but it should really wait for getting a response. Otherwise, the python / OS will close the client socket as soon as the program completes and it will mostly be before the server gets a chance to respond.
I believe with the Handler code you are able to receive the data sent by the client on the server and are also able to send some data back from the Handler to the client? You must have understood that the server cannot send any data back unless there is a client connected to it?
Now, to send data to the client (or clients) from "another" thread, you will need a way to make the handler objects or the client sockets (available inside the Handler object as self.request) available to the "another" thread.
One way is to override the def __init__(self, request, client_address, server): method and save this object's reference in a global list. Remember to do the below as the last line of the overridden init -
# BaseRequestHandler __init__ must be the last statement as all request processing happens in this method
socketserver.BaseRequestHandler.__init__(self, request, client_address, server)
Once you have all the client handlers in the global list, you can easily write to all the clients from any thread as per your needs. You must read about synchronization (Locks) and understand that using same object / socket from multiple threads can create some logical / data issues with your application.
Another thing that you have to worry about and code for is cleaning up this global list whenever a client closes the connection.
I am writing a broker-less, balanced, client-worker service written in python with ZeroMQ.
The clients acquire a worker's address, establish a connection ( zmq.REQ / zmq.REP ), send single request, receive a single response and then disconnect.
I have chosen a broker-less architecture because the amount of a data that needs to get transferred between the clients and workers is relatively large, despite there only being a single REQ/REP pair per connection, and using a broker as a 'middle man' would create a bottleneck.
While testing the system, I noticed that the communication between the clients and workers was halting randomly, only sometimes resuming after a couple of seconds (often several minutes).
I narrowed down the issue to the .connect() / .disconnect() of clients to workers.
I have written two small python scripts that reproduce the bug.
import zmq
class Site:
def __init__(self):
ctx = zmq.Context()
self.pair_socket = ctx.socket(zmq.REQ)
self.num = 0
def __del__(self):
print "closed"
def run_site(self):
print "running..."
while True:
self.pair_socket.connect('tcp://127.0.0.1:5555')
print 'connected'
self.pair_socket.send_pyobj(self.num)
print 'sent', self.num
print self.pair_socket.recv_pyobj()
self.pair_socket.disconnect('tcp://127.0.0.1:5555')
print 'disconnected'
self.num += 1
s = Site()
s.run_site()
and
import zmq
class Server:
def __init__(self):
ctx = zmq.Context()
self.pair_socket = ctx.socket(zmq.REP)
self.pair_socket.bind('tcp://127.0.0.1:5555')
def __del__(self):
print " closed"
def run_server(self):
print "running..."
while True:
x = self.pair_socket.recv_pyobj()
print x
self.pair_socket.send_pyobj(x)
s = Server()
s.run_server()
I don't think the issue is related to memory or gc as I have tried disabling gc - without much affect.
I have tried using zmq.LINGER as described here: Zeromq with python hangs if connecting to invalid socket
What could cause these randoms freezes?
The REP socket is synchronous by definition. So your server can only serve one request at a time, rest of them will just fill up the buffer and get lost at some point.
To fix the root cause, you need to use the ROUTER socket instead.
class Server:
def __init__(self):
ctx = zmq.Context()
self.pair_socket = ctx.socket(zmq.ROUTER)
self.pair_socket.bind('tcp://127.0.0.1:5555')
self.poller = zmq.Poller()
self.poller.register(self.pair_socket, zmq.POLLIN)
def __del__(self):
print " closed"
def run_server(self):
print "running..."
while True:
try:
items = dict(self.poller.poll())
except KeyboardInterrupt:
break
if self.pair_socket in items:
x = self.pair_socket.recv_multipart()
print x
self.pair_socket.send_multipart(x)
I'm implementing a simple server which should print a message if nothing is received for 3 seconds.
Handler
class SingleTCPHandler(SocketServer.StreamRequestHandler):
def handle(self):
while True:
message = self.rfile.readline().strip()
print message
Server
class SimpleServer(SocketServer.TCPServer):
timeout = 3
def handle_timeout(self):
print "Timeout"
def __init__(self, server_address, RequestHandlerClass):
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass)
Here I'm extending the TCPServer for testing the timeout method.
I've set the timeout atribute to 3. According to the docs, if that time passes and no messages are sent to the client handle_timeout() is called which, in my case, just prints 'Timeout'.
BaseServer.timeout
Timeout duration, measured in seconds, or None if no timeout is desired.
If handle_request() receives no incoming requests within the
timeout period, the handle_timeout() method is called.
I start the server, and observe it's output. When i connect to it and send some messages, they are normally printed. However, if I don't send anything for 3 seconds or more, nothing happens. As if the timeout and handle_timeout() haven't been implemented.
What could be the source of this behavior?
you must not call server_forever() method for app loop.
try this one instead:
while True:
self.handle_request()
handle_timeout() works for me then.
Can you try declare the timeout at self.timeout (i.e make it a instance field instead of class variable ?)
EDIT (here is the code)
def handle_request(self):
"""Handle one request, possibly blocking.
Respects self.timeout.
"""
# Support people who used socket.settimeout() to escape
# handle_request before self.timeout was available.
timeout = self.socket.gettimeout()
if timeout is None:
timeout = self.timeout
elif self.timeout is not None:
timeout = min(timeout, self.timeout)
fd_sets = select.select([self], [], [], timeout)
if not fd_sets[0]:
self.handle_timeout()
return
self._handle_request_noblock()
Here is the document of serve_forever():
Handle requests until an explicit shutdown() request. Poll for shutdown every poll_interval seconds. Ignores self.timeout. If you need to do periodic tasks, do them in another thread
So the serve_forever() will only check if shutdown() is called every poll_interval whose default value is 0.5 seconds. And only handle_request() care about the timeout.
Here is the code for serve_forever() and handle_request().
In the end, I dropped the socketserver module and went directly with socket module, in which timeout worked.
TIMEOUT = 3
HOST = '192.0.0.202'
PORT = 2000
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while 1:
conn, addr = s.accept()
conn.settimeout(TIMEOUT)
while 1:
try:
data = conn.recv(1024)
#Do things
except socket.timeout:
#Timeout occurred, do things
if not data or P=='end':
print 'Connection lost. Listening for a new controller.'
break
conn.close()
First of all, what do you mean by "[the server] should print a message if nothing is received for 3 seconds."?
Do you mean that the server should ...
shutdown if it hadn't any new requests in certain period?
close a connection if it isn't finished in a certain period?
In the first case you can use BaseServer.timeout but you would also have to use BaseServer.handle_request() instead of BaseServer.server_forever().
In the second case you should have set the timeout for the SingleTCPHandler:
class SingleTCPHandler(SocketServer.StreamRequestHandler):
timeout = 3
def handle(self):
while True:
message = self.rfile.readline().strip()
print message
For people who want to use their own implementation of BaseRequestHandler:
class MyRequestHandler(SocketServer.BaseRequestHandler):
def handle(self):
self.request.settimeout(3)
I need to connect to a server (e.g. smpp server) and send periodic data every 2 seconds, here's the code:
import asyncore, socket, threading, time
class SClient(asyncore.dispatcher):
buffer = ""
t = None
def __init__(self, host):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect( (host, 25) )
print "sending data from __init__"
self.sendCommand("data_init")
self.t = SenderThread(self)
self.t.start()
def sendCommand(self, command):
self.buffer = command
def handle_close(self):
self.close()
self.t.stop()
def handle_read(self):
print self.recv(8192)
def writable(self):
print 'asking for writable ? len='+str(len(self.buffer))
return (len(self.buffer) > 0)
def handle_write(self):
print "writing to socket"
sent = self.send(self.buffer)
self.buffer = self.buffer[sent:]
print "wrote "+str(sent)+" to socket"
class SenderThread(threading.Thread):
_stop = False
def __init__(self, client):
super(SenderThread,self).__init__()
self.client = client
def stop(self):
self._stop = True
def run(self):
counter = 0
while self._stop == False:
counter += 1
time.sleep(1)
if counter == 2:
print "sending data from thread"
self.client.sendCommand("data_thread")
counter = 0
client = SClient('127.0.0.1')
asyncore.loop()
Here's the output when running:
$ python test.py
sending data from __init__
asking for writable ? len=9
writing to socket
wrote 9 to socket
asking for writable ? len=0
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
asking for writable ? len=11
writing to socket
wrote 11 to socket
asking for writable ? len=0
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
asking for writable ? len=11
writing to socket
wrote 11 to socket
asking for writable ? len=0
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
sending data from thread
My thread is sending data to server every 2 seconds through the buffer variable but asyncore is calling writable and handle_write exactly every 1 minute, i dont understand why it doesnt fetch the buffer just after its getting populated from the thread ?
Look at the docs for the asyncore loop method.
The timeout argument sets the timeout parameter for the appropriate
select() or poll() call, measured in seconds; the default is 30
seconds.
It's only firing the handle_write every 30 secs.
I have this code
class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):
def handle(self):
_data = self.request.recv(1024)
Utils.log("Received from %s: %s" % (self.client_address, _data))
calling it with
kamcon_server = ThreadedTCPServer((HOST, 3011), ThreadedTCPRequestHandler)
server_thread = threading.Thread(target = kamcon_server.serve_forever)
server_thread.setDaemon(True)
server_thread.start()
I can connect to the host, and the server can send the data, but when the client sends something to the server, the connection is automatically closed. Why? Thank you.
Your handle() method is only calling recv() once per connection. If you want to handle multiple messages from the client, you need to loop. You should think about your protocol as well, so that you can handle request/response messages larger than 1024 bytes (e.g. parse _data and figure out if you have a complete message, buffer partial requests, etc).
For example:
def handle(self):
close = 0
while not close:
_data = self.request.recv(1024)
if not _data:
# EOF, client closed, just return
return
Utils.log("Received from %s: %s" % (self.client_address, _data))
self.request.send('got %d bytes\r\n' % len(_data))
if 'quit' in _data:
close = 1
Client session:
% telnet localhost 3011
hi
got 4 bytes
bye
got 5 bytes
telnet> quit
Try this code, it allows multiple connections to same port and not close socket until client do it:
import SocketServer
import socket, threading
class MyTCPHandler(SocketServer.BaseRequestHandler):
BUFFER_SIZE = 4096
def handle(self):
while 1:
#get input with wait if no data
data = self.request.recv(self.BUFFER_SIZE)
#suspect many more data (try to get all - without stop if no data)
if (len(data)==self.BUFFER_SIZE):
while 1:
try: #error means no more data
data += self.request.recv(self.BUFFER_SIZE, socket.MSG_DONTWAIT)
except:
break
#no data found exit loop (posible closed socket)
if (data == ""): break
#processing input
print "%s (%s) wrote: %s" % (self.client_address[0], threading.currentThread().getName(), data.strip())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
server = SocketServer.ThreadingTCPServer((HOST, PORT), MyTCPHandler)
server.serve_forever()
You can also use ForkingTCPServer instead ThreadingTCPServer.
the handle() method is called once a new TCP connection is set up, not once every time there is data available. You're supposed to handle all comminication of a TCP session in this method.
Since all you do is read one chunk of data, the connection getst closed when the handle() method returns.
Look into how TCP connections work, at the end of every 'Stream' of TCP data the connection is closed, to tell the listener that no more data is incoming in that batch. Roughly a TCP connection is as follows:
Open Connection
Send Data
Close Connection
If you're just looking at sending packets of data, try using UDP instead that way each packet will be read on arrival, provided you're listening for it.
It might be worth telling us what your planning on using this server for...
Anyway hope that helps.