Develop Reference

Python deploying infinite while loop to production and handling failures

I have a script that waits for tasks from a task queue and then runs them. Something like this minimal example:
import redis
cache = redis.Redis(host='127.0.0.1', port=6379)
import time
def main():
while True:
message = cache.blpop('QUEUE', timeout=0)
work(message)
def work(message):
print(f"beginning work: {message}")
time.sleep(10)
if __name__ == "__main__":
main()
I am not using a web server because the script does not need to answer http requests. However I've confused myself a bit about how to make this script robust against errors in production.
With a web server and gunicorn, gunicorn would handle forking a process for each request. If the request causes an error then the worker dies and the request fails but the server continues to run.
How can I achieve this if I'm not running an http server? I could fork a process to perform the "work" function, but the code performing the fork would still be application code.
Is it possible to deploy a non-http server script like mine using Gunicorn? Is there something else I should be using to handle forking processes?
Or is it reasonable to fork inside the application and deploy to production?

what about this:
while True:
try:
message = cache.blpop('QUEUE', timeout=0)
work(message)
except: Exception as e :
print(e)

How to capture ctrl-c for killing a Flask python script

I have the following script which just boots up a web server serving a dynamically created website. In order to get dynamic data the script opens a file to read the data.
My concern is how can I catch CTRL-C command for killing the python script so I can close the file before script thread is killed.
I tried the following couple things but neither work:
from flask import Flask, render_template
import time
# Initialize the Flask application
app = Flask(__name__)
fileNames = {}
fileDesc = {}
for idx in range(1,4):
fileNames["name{}".format(idx)] = "./name" + str(idx) + ".txt"
fileDesc["name{}".format(idx)] = open(fileNames["name{}".format(idx)],'r')
try:
#app.route('/')
def index():
# code for reading data from files
return render_template('index.html', var1 = var1)
#app.errorhandler(Exception)
def all_exception_handler(error):
print("Closing")
for key, value in fileDesc:
val.close()
print("Files closed")
if __name__ == '__main__':
app.run(
host="192.168.0.166",
port=int("8080"),
debug=True
)
except KeyboardInterrupt:
print("Closing")
for key, value in fileDesc:
val.close()
print("Files closed")
Thanks in advance.

I am struggling with the same thing in my project. Something that did work for me was using signal to capture CTRL-C.
import sys
import signal
def handler(signal, frame):
print('CTRL-C pressed!')
sys.exit(0)
signal.signal(signal.SIGINT, handler)
signal.pause()
When this piece of code is put in the script that is running the Flask app, the CTRL-C can be captured. As of now, you have to use CTRL-C twice and then the handler is executed though. I'll investigate further and edit the answer if I find something new.
Edit 1
Okay I've done some more research and came up with some other methods, as the above is quite hack 'n slash.
In production, clean-up code such as closing databases or files is done via the #app.teardown_appcontext decorator. See this part of the tutorial.
When using the simple server, you can shut it down via exposing the werkzeug shutdown function. See this post.
Edit 2
I've tested the Werkzeug shutdown function, and it also works together with the teardown_appcontext functions. So I suggest to write your teardown functions using the decorator and writing a simple function that just does the shutdown of the werkzeug server. That way production and development code are the same.

Use atexit to handle this, from: https://stackoverflow.com/a/30739397/5782985
import atexit
#defining function to run on shutdown
def close_running_threads():
for thread in the_threads:
thread.join()
print "Threads complete, ready to finish"
#Register the function to be called on exit
atexit.register(close_running_threads)
#start your process
app.run()

How can I run two flask servers in two threads/processes on two ports under one python program?

I'm writing a python debugging library which opens a flask server in a new thread and serves information about the program it's running in. This works fine when the program being debugged isn't a web server itself. However if I try to run it concurrently with another flask server that's running in debug mode, things break. When I try to access the second server, the result alternates between the two servers.
Here's an example:
from flask.app import Flask
from threading import Thread
# app1 represents my debugging library
app1 = Flask('app1')
#app1.route('/')
def foo():
return '1'
Thread(target=lambda: app1.run(port=5001)).start()
# Cannot change code after here as I'm not the one writing it
app2 = Flask('app2')
#app2.route('/')
def bar():
return '2'
app2.run(debug=True, port=5002)
Now when I visit http://localhost:5002/ in my browser, the result may either be 1 or 2 instead of consistently being 2.
Using multiprocessing.Process instead of Thread has the same result.
How does this happen, and how can I avoid it? Is it unavoidable with flask/werkzeug/WSGI? I like flask for its simplicity and ideally would like to continue using it. If that's not possible, what's the simplest library/framework that I can use that won't interfere with any other web servers running at the same time? I'd also like to use threads instead of processes if possible.

The reloader of werkzeug (which is used in debug mode by default) creates a new process using subprocess.call, simplified it does something like:
new_environ = os.environ.copy()
new_environ['WERKZEUG_RUN_MAIN'] = 'true'
subprocess.call([sys.executable] + sys.argv, env=new_environ, close_fds=False)
This means that your script is reexecuted, which is usually fine if all it contains is an app.run(), but in your case it would restart both app1 and app2, but both now use the same port because if the OS supports it the listening port is opened in the parent process, inherited by the child and used there directly if an environment variable WERKZEUG_SERVER_FD is set.
So now you have two different apps somehow using the same socket.
You can see this better if you add some output, e.g:
from flask.app import Flask
from threading import Thread
import os
app1 = Flask('app1')
#app1.route('/')
def foo():
return '1'
def start_app1():
print("starting app1")
app1.run(port=5001)
app2 = Flask('app2')
#app2.route('/')
def bar():
return '2'
def start_app2():
print("starting app2")
app2.run(port=5002, debug=True)
if __name__ == '__main__':
print("PID:", os.getpid())
print("Werkzeug subprocess:", os.environ.get("WERKZEUG_RUN_MAIN"))
print("Inherited FD:", os.environ.get("WERKZEUG_SERVER_FD"))
Thread(target=start_app1).start()
start_app2()
This prints for example:
PID: 18860
Werkzeug subprocess: None
Inherited FD: None
starting app1
starting app2
* Running on http://127.0.0.1:5001/ (Press CTRL+C to quit)
* Running on http://127.0.0.1:5002/ (Press CTRL+C to quit)
* Restarting with inotify reloader
PID: 18864
Werkzeug subprocess: true
Inherited FD: 4
starting app1
starting app2
* Debugger is active!
If you change the startup code to
if __name__ == '__main__':
if os.environ.get("WERKZEUG_RUN_MAIN")) != 'true':
Thread(target=start_app1).start()
start_app2()
then it should work correctly, only app2 is reloaded by the reloader. However it runs in a separate process, not in a different thread, that is implied by using the debug mode.
A hack to avoid this would be to use:
if __name__ == '__main__':
os.environ["WERKZEUG_RUN_MAIN"] = 'true'
Thread(target=start_app1).start()
start_app2()
Now the reloader thinks it's already running in the subprocess and doesn't start a new one, everything runs in the same process. Reloading won't work and I don't know what other side effects that may have.

Run part of Python unit tests on remote Linux machine from a Windows host

My situation goes like this:
I have a windows-based server program and a linux-based client.
I have many tests for the linux clients which run and are required to run on local linux machines
I need to run some code from windows server machine which will send some messages to the linux clients. Then, a test shall be executed on linux client machines which verifies the effect of those messages
So a typical test case would look like this, running on windows host:
test_example_message(self):
# has to be executed locally on windows server
send_message(EXAMPLE, hosts)
# has to be executed locally on linux clients
for host in hosts:
verify_message_effect(EXAMPLE, host)
I have found that pytest-xdist is somehow able to do that.
I there any good tutorial or code example on how to use it?

My final design utilizes ssh & multiprocessing instead of xdist (in the execute_tc() function):
import multiprocessing
import test_functions
def test_example_message(self):
"""Example test case"""
# to get IPs, usernames, passwords, other required data
config = get_test_config('example_message')
# will take care of threading and executing parts
result_dict = execute_tc(config)
# to fail or not to fail. take care of proper reporting
process_results(result_dict)
def execute_tc(config):
"""Execute test code in parallel"""
# create shared results dictionary
manager = multiprocessing.Manager()
result_dict = manager.dict({})
# create processes
processes = []
for func, platform, args in config:
function = getattr(test_functions, func)
worker = multiprocessing.Process(target=function, args=[result_dict, platform, args])
worker.daemon = True
worker.start()
processes.append(worker)
for process in processes:
process.join()
return result_dict

def execute_tc(config):
"""Execute test code in parallel"""
# create shared results dictionary
manager = multiprocessing.Manager()
result_dict = manager.dict({})
# create processes
processes = []
for func, platform, args in config:
function = getattr(test_functions, func)
worker = multiprocessing.Process(target=function, args=[result_dict, platform, args])
worker.daemon = True
worker.start()
processes.append(worker)
for process in processes:
process.join()
return result_dict
I think you have change the execute test case method to execute particular test cases.

Restart Bottle app (programmatically)

How do I restart my Bottle app programmatically?
def error_handler(error):
if error.message == "connection already closed":
RESTART_BOTTLE_SERVER() # This will reacquire connection

You can stop the bottle app (thread) with the approach described in this answer.

I'll recommed you'll run your bottle server as a daemon in the background on your OS. You can than start and stop your server and use simple python code to kill the thread. BottleDaemon might do the job for you.
from bottledaemon import daemon_run
from bottle import route
#route("/hello")
def hello():
return "Hello World"
if __name__ == "__main__":
daemon_run()
The above application will launch in the background. This top-level script can be used to start/stop the background process easily:
jonathans-air:bottle-daemon jhood$ python bottledaemon/bottledaemon.py
usage: bottledaemon.py [-h] {start,stop}
Now you can use bottledaemon.py to start or stop or restart your application and call it from your main python file.