I'm running a python script that creates a Tornado server, the server is run by supervisor.
I want to gracefully terminate all WebSocket client connections when a supervisorctl reload is issued (normally after a deploy).
My problem is that I'm not able to get a function called when my server is killed by supervisor, but it works when using kill with the signal or run on console and killed with Control+C.
I have tried other signals and configurations without luck.
import signal, sys
def clean_resources(signum, frame):
print "SIG: %d, clean me" % signum
sys.exit(0)
if __name__ == '__main__':
# Nicely handle closing the server
for sig in (signal.SIGINT, signal.SIGTERM):
signal.signal(sig, clean_resources)
This is my tornado_supervisor.conf
[program:tornado_server]
command = python /opt/tornado/server.py -p 8890
user = www-data
stdout_logfile = /var/log/tornado/tornado_server_sup.log
redirect_stderr = true
autorestart=true
environment=HOME='/var/www'
environment=PYTHONPATH="$PYTHONPATH:/opt/tornado/"
stopsignal = TERM
stopwaitsecs = 10
stopasgroup = true
I had similar/same problem. Only parent Tornado process got the signal, while children processes where not killed.
I made arrangement that parent process kills children manually using os.killpg(), also, children uses some delays to (possibly) finish current requests:
#will be initialized in main()
server = None
loop = None
def stop_loop():
global loop
loop.stop()
def signal_handler_child_callback():
global loop
global server
server.stop()
# allow to finish processing current requests
loop.add_timeout(time.time() + LOOP_STOP_DELAY, stop_loop)
def signal_handler(signum, frame):
global loop
global server
if loop:
#this is child process, will restrict incoming connections and stop ioloop after delay
loop.add_callback(signal_handler_child_callback)
else:
#this is master process, should restrict new incomming connections
#and send signal to child processes
server.stop()
signal.signal(signal.SIGTERM, signal.SIG_DFL)
os.killpg(0, signal.SIGTERM)
def main():
parse_command_line()
signal.signal(signal.SIGTERM, signal_handler)
# ...
tornado_app = tornado.web.Application(
[
#...
])
global server
server = tornado.httpserver.HTTPServer(tornado_app)
server.bind(options.port)
server.start(0)
global loop
loop = tornado.ioloop.IOLoop.instance()
loop.start()
if __name__ == '__main__':
main()
Related
I initially started a loop that runs a thread for each client of my server, but this graphic stopped the program. So I put this loop inside a thread, but the main thread stops after creating the sub thread, and no command in the main thread works after Thread.start().
def start():
startb.setEnabled(False)
stopb.setEnabled(True)
ev.set()
Thread(target = listen).start()
pass
def listen():
so = socket.socket()
so.bind((ip,port))
so.listen(4)
while ev.is_set():
th = client_thread(so.accept()[0], response)
th.start()
#The program will not run from now on
print('1')#to debug
print('2')#to debug
so.close()# -> This needs to be implemented
pass
I have a python process with a main thread starting parallel threads for 1 gRPC server and 1 HTTP server. I want the OS process for this application to exit if ANY of the parallel threads exits.
I think the main thread, as is coded here, would wait as long as there is a single parallel thread that is running. What do I need to do to change this so the main thread exits as soon as any parallel thread exits?
if __name__ == '__main__':
svc = MyService()
t1 = GrpcServer(svc)
t1.start()
t2 = HealthHttpServer()
t2.start()
with the servers defined as
class GrpcServer(threading.Thread):
def __init__(self, service):
super().__init__()
self.grpcServer(futures.ThreadPoolExecutor(max_workers=10))
self.grpcServer.add_insecure_port('[::]:8000')
myservice_pb2_grpc.add_MyServiceServicer_to_server(service, self.grpcServer)
def run(self):
self.grpcserver.start()
class HealthServer(Thread):
def __init__(self):
super().__init__()
def run(self):
port=2113
httpd = HTTPServer(('localhost', port), HealthHTTPRequestHandler)
httpd.serve_forever()
class HealthHTTPRequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
'''Respond to a GET request.'''
if self.path == '/healthz':
self.send_response(HTTPStatus.OK)
self.end_headers()
self.wfile.write(b'ok')
else:
self.send_response(HTTPStatus.NOT_FOUND)
self.end_headers()
The cleanest I've found so far is:
define all these threads as daemon threads
define a global threading.Event
object
add a top-level try...finally in each thread, and call that event' set()
in the finally
in the main thread wait on that event after the threads are started
If anything goes wrong in any of the threads the finally block will execute, signaling that event, unblocking the main thread which exits. All the other threads being daemon threads, the process will then exit.
I'm trying to run a python http server in the background using threading. I came across several references that do the following:
import threading
import http.server
import socket
from http.server import HTTPServer, SimpleHTTPRequestHandler
debug = True
server = http.server.ThreadingHTTPServer((socket.gethostname(), 6666), SimpleHTTPRequestHandler)
if debug:
print("Starting Server in background")
thread = threading.Thread(target = server.serve_forever)
thread.daemon = True
thread.start()
else:
print("Starting Server")
print('Starting server at http://{}:{}'.format(socket.gethostname(), 6666))
server.serve_forever()
When thread.daemon is set to True, the program will finish without starting the server (nothing running on port 6666).
And when I set thread.daemon to False, it starts the server in foreground and blocks the terminal until I kill it manually.
Any idea on how to make this work?
In both cases the server is launched in the background, in the separate thread. This means that thread.start() launches the server and python continues executing the rest of the code in the main thread.
However, there seems to be nothing else to execute in your program. Python reaches the end of the file and the main thread is done.
The OS requires all non-daemon threads to be done before the process could be finished. When thread.daemon is set to False the OS waits until the server thread exits (which will never happen, as the name serve_forever implies). When it is True the process is closed immediately after the main thread is done.
Put whatever code you want to be executed asynchronously after the thread.start() and you're done!
I have a multiproccessing tornado web server and I want to create another process that will do some things in the background.
I have a server with to following code
start_background_process
app = Application([<someurls>])
server = HTTPServer(app)
server.bind(8888)
server.start(4) # Forks multiple sub-processes
IOLoop.current().start()
def start_background_process():
process = multiprocessing.Process(target=somefunc)
process.start()
and everything is working great.
However when I try to close the server (by crtl c or send signal)
I get AssertionError: can only join a child process
I understood the cause of this problem:
when I create a process with multiprocess a call for the process join method
is registered in "atexit" and because tornado does a simple fork all its childs also call the join method of the process I created and the can't since the process is their brother and not their son?
So how can I open a process normally in tornado?
"HTTPTserver start" uses os.fork to fork the 4 sub-processes as it can be seen in its source code.
If you want your method to be executed by all the 4 sub-processes, you have to call it after the processes have been forked.
Having that in mind your code can be changed to look as below:
import multiprocessing
import tornado.web
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
# A simple external handler as an example for completion
from handlers.index import IndexHandler
def method_on_sub_process():
print("Executing in sub-process")
def start_background_process():
process = multiprocessing.Process(target=method_on_sub_process)
process.start()
def main():
app = tornado.web.Application([(r"/", IndexHandler)])
server = HTTPServer(app)
server.bind(8888)
server.start(4)
start_background_process()
IOLoop.current().start()
if __name__ == "__main__":
main()
Furthermore to keep the behavior of your program clean during any keyboard interruption , surround the instantiation of the sever by a try...except clause as below:
def main():
try:
app = tornado.web.Application([(r"/", IndexHandler)])
server = HTTPServer(app)
server.bind(8888)
server.start(4)
start_background_process()
IOLoop.current().start()
except KeyboardInterrupt:
IOLoop.instance().stop()
I have a flask controller which starts a tcpserver on demand. I am posting a simplified version of the code.
#app.route("/")
def hello():
scenario_server = MyTCPServer(('localhost', 9000), MyTCPHandler)
server_thread = threading.Thread(target=scenario_server.serve_forever)
server_thread.daemon = True
server_thread.start()
return "TCP server started"
The MyTCPServer is a normal SocketServer.TCPServer. This is an example of a simple MyTCPHandler.
class MyTCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
while True:
data = self.request.recv(1024).strip().decode('ascii')
if data == "HELLO":
self.request.sendall('COUCOU'.encode('ascii'))
else:
break
And I start my app application in the __main__ the standard way (using the built in server app.run(threaded=True))
After I launch the tcpserver, I connect to it using telnet and send HELLO
With this code and the previous scenario, if I press CTRL+C, the main thread will caught it but it won't kill the program. It seems that the thread running the TCPServer is not killed even if the thread is run with daemon = True
However If I modify my handler to change this :
data = self.request.recv(1024).strip().decode('ascii')
if data == "":
break
On CTRL+C, it properly exits the program.
Anybody could explain why ?