Develop Reference

Constantly dying workers Flask-SocketIO + uWSGI

I'm trying to implement sockets usage in my flask project and followed by flask-socketio documentation I had made it run perfectly with gunicorn, but found myself unable to make it work using uwsgi. It seems like it should be working even with the simplest app and settings but it just keeps killing the workers whatever configuration I try to use. Here's an example of an environment and code I have:
app.py:
from flask import Flask
from flask_socketio import SocketIO
app = Flask(__name__)
app.config['SECRET_KEY'] = 'extremelysecret'
socketio = SocketIO(app)
if __name__ == '__main__':
app.run()
pip freeze:
click==7.1.2
Flask==1.1.2
Flask-SocketIO==4.3.0
gevent==20.5.0
greenlet==0.4.15
gunicorn==20.0.4
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
python-engineio==3.12.1
python-socketio==4.5.1
six==1.15.0
uWSGI==2.0.18
Werkzeug==1.0.1
Running with an example from documentation:
uwsgi --http :5000 --gevent 1000 --http-websockets --master --wsgi-file app.py --callable app
Output:
your processes number limit is 63299
your memory page size is 4096 bytes
detected max file descriptor number: 1024
- async cores set to 10 - fd table size: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :5008 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:39875 (port auto-assigned) fd 3
Python version: 3.8.0 (default, Jan 9 2020, 23:03:43) [GCC 7.4.0]
Python main interpreter initialized at 0x55899af6a860
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 501072 bytes (489 KB) for 20 cores
*** Operational MODE: preforking+async ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x55899af6a860 pid: 22479
(default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 22479)
spawned uWSGI worker 1 (pid: 22481, cores: 10)
spawned uWSGI worker 2 (pid: 22482, cores: 10)
*** running gevent loop engine [addr:0x558998a7f860] ***
spawned uWSGI http 1 (pid: 22483)
DAMN ! worker 1 (pid: 22481) died :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 22484)
DAMN ! worker 2 (pid: 22482) died :( trying respawn ...
Respawned uWSGI worker 2 (new pid: 22485)
*** running gevent loop engine [addr:0x558998a7f860] ***
DAMN ! worker 1 (pid: 22484) died :( trying respawn ...
So far I've tried:
- playing with uwsgi parameters;
- using different app configurations in app.py;
- using couple different versions of python;
- figuring out is everything installed properly;
And still, my guess is that I miss something obvious here. Thanks in advance for pointing me on that.

When I enable uwsgi thread support and start the scheduler the api stops working

I'm new to python, flask, nginx and all that stuff.
I have a flask app that acts as API for a frontend. Also when the flask app is started I would like to start a scheduled task with APScheduler.
The problem is that when I enable uwsgi thread support and start the scheduler the api stops working (504 Gateway Time-out). But the scheduler works as seen in the logfile. When i remove the scheduler / thread support the api works but I obviously don't have the scheduler anymore.
Somehow I suspect the scheduler prevents the flask app from being run properly?
As I am new to those technologies I'll post my setup below. If you need any more file infos please tell me.(The whole thing is run on a raspberry pi and the api is accessed from my pc over lan)
app.service
[Unit]
Description=uWSGI instance to serve app
After=network.target
[Service]
User=pi
Group=www-data
WorkingDirectory=/home/pi/flask
Environment="PATH=/home/pi/flask/appenv/bin"
ExecStart=/home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
[Install]
WantedBy=multi-user.target
app.ini
[uwsgi]
module = wsgi:app
master = true
processes = 5
socket = /home/pi/flask/app.sock
chmod-socket = 660
vacuum = true
die-on-term = true
app.py
#!/usr/bin/env python3
from flask import Flask, request
from apscheduler.schedulers.background import BackgroundScheduler
import logging
logging.basicConfig(filename='logfile.log',level=logging.DEBUG)
from api.Controller import Controller
from Handler.Handler import Handler
from apscheduler.executors.pool import ThreadPoolExecutor, ProcessPoolExecutor
api_controller = Controller()
handler = Handler()
def startHandlerJob():
handler.ExecuteAllSensors()
app = Flask( __name__ )
#app.route('/app')
def apiDefinition():
return 'API Definition: GetHumidityValues, TODO'
#app.route( "/app/GetHumidityValues", methods=["GET"] )
def GetHumidityValues():
logging.info("app.py: API-call GetHumidityValues")
return api_controller.GetHumidityValues()
if (__name__ == "__main__"):
app.run(host='0.0.0.0')
executors = {
'default': ThreadPoolExecutor(20),
'processpool': ProcessPoolExecutor(5)
}
job_defaults = {
'coalesce': False,
'max_instances': 1
}
scheduler = BackgroundScheduler(daemon=True, executors=executors, job_defaults=job_defaults)
scheduler.start()
scheduler.add_job(startHandlerJob,'cron', minute='*')
logfile.log
WARNING:apscheduler.scheduler:Execution of job "startHandlerJob (trigger: cron[minute=''], next run at: 2019-12-18 19:01:00 CET)" skipped: maximum number of running instances reached (1)
DEBUG:apscheduler.scheduler:Next wakeup is due at 2019-12-18 19:02:00+01:00 (in 59.980780 seconds)
DEBUG:apscheduler.scheduler:Looking for jobs to run
WARNING:apscheduler.scheduler:Execution of job "startHandlerJob (trigger: cron[minute=''], next run at: 2019-12-18 19:02:00 CET)" skipped: maximum number of running instances reached (1)
DEBUG:apscheduler.scheduler:Next wakeup is due at 2019-12-18 19:03:00+01:00 (in 59.979407 seconds)
systemctl status app
* app.service - uWSGI instance to serve app
Loaded: loaded (/etc/systemd/system/app.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-12-18 18:40:57 CET; 23min ago
Main PID: 21129 (uwsgi)
Tasks: 8 (limit: 2200)
Memory: 22.9M
CGroup: /system.slice/app.service
|-21129 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
|-21148 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
|-21149 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
|-21150 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
|-21151 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
`-21152 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
Dec 18 18:40:57 raspberrypi uwsgi[21129]: mapped 386400 bytes (377 KB) for 5 cores
Dec 18 18:40:57 raspberrypi uwsgi[21129]: *** Operational MODE: preforking ***
Dec 18 18:40:59 raspberrypi uwsgi[21129]: WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0xa6f900 pid: 21129 (default app)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: *** uWSGI is running in multiple interpreter mode ***
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI master process (pid: 21129)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 1 (pid: 21148, cores: 1)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 2 (pid: 21149, cores: 1)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 3 (pid: 21150, cores: 1)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 4 (pid: 21151, cores: 1)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 5 (pid: 21152, cores: 1)
The logfile shows that the scheduler is active. But when I try to use http:/raspberryipaddress/app the answer is a 504 Gateway Time-out response. When I remove the scheduler and disable thread support this call works as desired.
Any help is appreciated. Probably I am missing something obvious, because I am new to all those things.
Thanks!

I hesitate a bit to put this as an answer but here goes...
In uwsgi app.ini after setting processes (5 in your case, I use 4) I also set
threads = 2. I don't know if this has any direct relation to the --enable-threads option because that seems to be for your app to start it's own threads, but it might help for uwsgi to have its own threads per process. Also the UWSGI documentation points out (somewhere) that more processes are not necessarily better.
Also on that, your logfile shows warnings from the scheduler that the maximum number of running instances being reached. I take this means the job you've given the scheduler is not finishing before the next scheduled run (1 minute later)? If that's the case is it stuck somewhere, blocking everything else?
Finally, if all else fails, something else from the docs (UWSGI Security and availability)
A common problem with webapp deployment is “stuck requests”. All of your threads/workers are stuck (blocked on request) and your app cannot accept more requests. To avoid that problem you can set a harakiri timer.It is a monitor (managed by the master process) that will destroy processes stuck for more than the specified number of seconds (choose harakiri value carefully).
And finally, finally :) there's a top like monitoring tool for uwsgi which might be handy to see what's going on, just
pip install uwsgitop
uwsgitop 127.0.0.1:9191
plus telnetting into that adress:port apparently shows you a lot more info (haven't tried it myself).

Okay thanks for the answers. I did not get it to work with your suggested solutions. Therefore I simply decided to run the scheduler as separate systemd service. That way the scheduling does not stop the flask api from working.

I had exactly the same problem. The fix for me was to use ProcessPoolExecutor instead of ThreadPoolExecutor.
For some unknown reasons (at least unknown for me), when there were several threads in ThreadPoolExecutor, that damn problem showed itself.
After I changed most cpu-bound threads to processes (i.e. using ProcessPoolExecutor for those jobs), that damn problem disappeared ...

uWSGI Emperor does not reload Vassal by touching the .ini file

I have multiple uWSGI vassals, all monitored by uwsgi emperor. I update the code for my app (Django) and I want the emperor to perform a clean reload of one of the vassals. To do that I
touch vassal-foo.ini
In the logs I see [emperor] reload the uwsgi instance vassal-foo.ini. This sounds promising, but the app is not reloaded. It continues to run the old version. Checking the process (PID) startup time, indeed, it has not been restarted.
Any hints what might cause this? Few things that might be uncommon:
Neither the emperor nor the vassal run in master mode
Emperor was installed with pip and runs under initctl
kill -9-ing the vassal triggers a correct reload (obviously)
I use symlinks
I have a secondary thread inside my python app (threading.Thread(target).start()) running with daemon=True
Things I tried and did not work:
Run the process without any additional threads (remove threading.Thread(target).start())
Touching with touch --no-dereference vassal-foo.ini
Starting emperor with --emperor-nofollow
vassal-foo.ini:
master = false
processes = 1
thunder-lock = true
enable-threads = true
socket = /tmp/%n.sock
chmod-socket = 666
vacuum = true
Emperor:
exec /tmp/uwsgi --emperor /tmp/configs/uwsgi/ --die-on-term --uid me --gid me --logto /tmp/logs/uwsgi-emperor.log
uWSGI version
$ uwsgi --version
2.0.17

The problem is that your vassal not run in master mode.
All the ways uwsgi reload a requires master process.
In emperor mode, when you touch the ini, the emperor send a sighup to the vassal and record in the log that the vassal is reload. But if the vassal doesn't has a master it just ignore the sighup. So that is why you see reload in log but nothing happened.

Attempting to deploy a python pyramid application with uWSGI

I'm attempting to deploy a pyramid application using uWSGI.
The application works fine when served with the included pyramid development server.
Also, I have set this up before, and I swear it worked at one time.
However, putting in the magic phrases right now is resulting in "This webpage is not available"
I'm trying to keep all of the configuration parameters as similar as possible to what I have currently so I don't have to worry about firewall issues.
uWSGI section in development.ini looks like this (from: Setup uWSGI as webserver with pyramid (no NGINX)):
[uwsgi]
socket = localhost:8080
virtualenv = /var/www/finance/finance-env
die-on-term = 1
master = 1
#logto = /var/log/wsgi/uwsgi.log
enable-threads = true
offload-threads = N
py-autoreload = 1
wsgi-file = /var/www/finance/wsgi.py
wsgy.py looks like this:
from pyramid.paster import get_app, setup_logging
ini_path = '/var/www/finance/corefinance/development.ini'
setup_logging(ini_path)
application = get_app(ini_path, 'main')
Here's the output right now. Everything seems to be listening just fine on port 8080.
user1#finance1:~$ sudo /var/www/finance/finance-env/bin/uwsgi --ini-paste-logg /var/www/finance/corefinance/development.ini
[uWSGI] getting INI configuration from /var/www/finance/corefinance/development.ini
*** Starting uWSGI 2.0.11.2 (64bit) on [Fri Jan 15 21:13:31 2016] ***
compiled with version: 4.7.2 on 16 November 2015 20:13:35
os: Linux-4.1.5-x86_64-linode61 #7 SMP Mon Aug 24 13:46:31 EDT 2015
nodename: finance1
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/user1
detected binary path: /var/www/finance/finance-env/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your processes number limit is 3934
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address localhost:8080 fd 3
Python version: 3.2.3 (default, Feb 20 2013, 14:49:46) [GCC 4.7.2]
Set PythonHome to /var/www/finance/finance-env
Python main interpreter initialized at 0xfd0a10
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145536 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xfd0a10 pid: 6275 (default app)
mountpoint already configured. skip.
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 6275)
spawned uWSGI worker 1 (pid: 6282, cores: 1)
Python auto-reloader enabled

Unless you are behind a proxy such as nginx, you need to use the internal http routing support in uwsgi, change
socket = localhost:8080
to
http = 0.0.0.0:8080
Here are the uwsgi http support docs

UWSGI - 500s, appears overloaded

I've built an application with Flask, and I've deployed it to servers with UWSGI and NGINX web servers on AWS.
I have 2 sets of servers, Web Servers and Application servers. Requests to the Application servers are proxied through the Web Servers.
When I do any number of API calls in a row, I start getting random 500s to some requests. I know the issue isn't related to my servers getting overloaded as I can see CPU Utilization running around 10%. I'm also pretty confident that the issue isn't related to my application code or database timeouts, as no matter how hard I try, I can't replicate the behavior on the Flask development web server.
I'm using UWSGI Emperor and checking the logs, I see no application/python errors. I just see the fact that a 500 is being returned. eg:
[pid: 2938|app: 0|req: 8/8] 86.40.78.178 () {42 vars in 1038 bytes} [Thu Sep 17 23:06:33 2015] GET /api/issue/231/workflow_items/ => generated 291 bytes in 84 msecs (HTTP/1.1 500) 6 headers in 557 bytes (1 switches on core 4)
As this is my first time working with UWSGI, I'm wondering if I'm doing anything wrong in terms of my configuration. Here's what my start up log looks like:
*** Starting uWSGI 2.0.11.1 (64bit) on [Thu Sep 17 23:06:20 2015] ***
compiled with version: 4.8.2 20140120 (Red Hat 4.8.2-16) on 17 September 2015 23:03:32
os: Linux-3.14.48-33.39.amzn1.x86_64 #1 SMP Tue Jul 14 23:43:07 UTC 2015
nodename: ip-10-0-1-90
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /etc/uwsgi/vassals
detected binary path: /var/www/workspace/venv/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 7826
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /var/www/workspace/workspace_uwsgi.sock fd 3
Python version: 2.7.9 (default, Apr 1 2015, 18:18:03) [GCC 4.8.2 20140120 (Red Hat 4.8.2-16)]
Set PythonHome to /var/www/workspace/venv
Python main interpreter initialized at 0x1b3bdc0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 341832 bytes (333 KB) for 10 cores
*** Operational MODE: preforking+threaded ***
added /var/www/workspace/ to pythonpath.
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1b3bdc0 pid: 2933 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 2933)
spawned uWSGI worker 1 (pid: 2938, cores: 5)
spawned uWSGI worker 2 (pid: 2939, cores: 5)
Here is what my UWSGI .ini looks like:
[uwsgi]
#application's base folder
base = /var/www/workspace
#python module to import
app = issuesite
module = %(app)
home = %(base)/venv
pythonpath = %(base)
#socket file's location
socket = /var/www/workspace/%n.sock
#permissions for the socket file
chmod-socket = 666
#the variable that holds a flask application inside the module imported at line #6
callable = app
processes=2
threads=5
#location of log files
logto = /var/log/uwsgi/%n.log
and the upstart script for emperor:
description "uWSGI"
start on runlevel [2345]
stop on runlevel [06]
respawn
env UWSGI=/var/www/workspace/venv/bin/uwsgi
env LOGTO=/var/log/uwsgi/emperor.log
exec $UWSGI --master --emperor /etc/uwsgi/vassals --die-on-term --uid nginx --gid nginx --enable-threads --logto $LOGTO
I saw something about --thunder-lock in the logs so I tried it but it made no difference, I still experience the same issues.
I wonder if anyone could give me some advice? I considered the possibility that the UWSGI is running out of memory? Note that using t2.micros on AWS.