Develop Reference

Run Python Pyramid server in production w/o Apache

So we used to run our Pyramid server with Apache in production. But we are moving to Docker containerization for easier prod deployments etc, and we want to adhere to the philosophy of "one process per container"..so instead of running Apache in the container + 4 python procs, we just want 1 python proc.
So my question is - is there a way to run a Pyramid server in production directly? Without using WSGI+Apache?
https://www.digitalocean.com/community/tutorials/how-to-use-the-pyramid-framework-to-build-your-python-web-app-on-ubuntu
My understanding is that pserve is for development only?
Create an application.py file and fill it with the following contents:
from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
def hello_world(request):
return Response('<h1>Hello world!</h1>')
if __name__ == '__main__':
config = Configurator()
config.add_view(hello_world)
app = config.make_wsgi_app()
server = make_server('0.0.0.0', 8080, app)
server.serve_forever()
Will the above work as a production-grade server?

The latest official recommendation is one concern per container. From the Docker docs (emphasis my own):
Each container should have only one concern. Decoupling applications
into multiple containers makes it easier to scale horizontally and
reuse containers. For instance, a web application stack might consist
of three separate containers, each with its own unique image, to
manage the web application, database, and an in-memory cache in a
decoupled manner.
Limiting each container to one process is a good rule of thumb, but it
is not a hard and fast rule. For example, not only can containers be
spawned with an init process, some programs might spawn additional
processes of their own accord. For instance, Celery can spawn multiple
worker processes, and Apache can create one process per request.
In your case, your web application server is a single concern. Running Apache+WSGI is totally fine. Don't worry about the processes—That's Apache's job.
For a better understanding of the "one concern" rule, this post is a good overview of what problems its trying to solve.

You can use Waitress, which, according to their documentation, is
... meant to be a production-quality pure-Python WSGI server with very
acceptable performance. It has no dependencies except ones which live
in the Python standard library.
Waitress is a part of the Pylons Project just like Pyramid is.

It looks like Bjoern is a solid choice when it comes to running Python directly, where the Python server has WSGI bindings:
https://www.appdynamics.com/blog/engineering/a-performance-analysis-of-python-wsgi-servers-part-2/
https://github.com/jonashaag/bjoern

Manually stop processes launched by mod_wsgi, and monitor how many processes are running

I know it's not recommended to run a Bottle or Flask app on production with python myapp.py --port=80 because it's a development server only.
I think it's not recommended as well to run it with python myapp.py --port=5000 and link it to Apache with: RewriteEngine On, RewriteRule /(.*) http://localhost:5000/$1 [P,L] (or am I wrong?), because WSGI is preferred.
So I'm currently setting up Python app <-> mod_wsgi <-> Apache (without gunicorn or other tool to keep things simple).
Question: when using WSGI, I know it's Apache and mod_wsgi that will automatically start/stop enough processes running myapp.py when requests will come, but:
how can I manually stop these processes?
more generally, is there a way to monitor them / know how many processes started by mod_wsgi are currently still running? (one reason, among others, is to check if the processes terminate after a request or if they stay running)
Example:
I made some changes in myapp.py, and I want to restart all processes running it, that have been launched by mod_wsgi (Note: I know that mod_wsgi can watch changes on the source code, and relaunch, but this only works on changes made on the .wsgi file, not on the .py file. I already read that touch myapp.wsgi can be a solution for that, but more generally I'd like to be able to stop and restart manually)
I want to temporarily stop the whole application myapp.py (all instances of it)
I don't want to use service apache2 stop for that because I also run other websites with Apache, not just this one (I have a few VirtualHosts). For the same reason (I run other websites with Apache, and some client might be downloading a 1 GB file at the same time), I don't want to do service apache2 restart that would have an effect on all websites using Apache.
I'm looking for a cleaner way than kill pid or SIGTERM, etc. (because I read it's not recommended to use signals in this case).
Note: I already read How to do graceful application shutdown from mod_wsgi, it helped, but here it's complementary questions, not a duplicate.
My current Python Bottle + Apache + mod_wsgi setup:
Installation:
apt-get install libapache2-mod-wsgi
a2enmod wsgi # might be done automatically by previous line, but just to be sure
Apache config (source: Bottle doc; a more simple config can be found here):
<VirtualHost *:80>
ServerName example.com
WSGIDaemonProcess yourapp user=www-data group=www-data processes=5 threads=5
WSGIScriptAlias / /home/www/wsgi_test/app.wsgi
<Directory />
Require all granted
</Directory>
</VirtualHost>
There should be up to 5 processes, is that right? As stated before in the question, how to know how many are running, how to stop them?
/home/www/wsgi_test/app.wsgi (source: Bottle doc)
import os
from bottle import route, template, default_app
os.chdir(os.path.dirname(__file__))
#route('/hello/<name>')
def index(name):
return template('<b>Hello {{name}}</b>!', name=name)
application = default_app()

Taken partially from this question, add display-name to WSGIDaemonProcess so you can grab them using a command like:
ps aux | grep modwsgi
Add this to your configuration:
Define GROUPNAME modwsgi
WSGIDaemonProcess yourapp user=www-data group=www-data processes=5 threads=5 display-name=%{GROUPNAME}
Update
There are a couple of reasons why ps would not give you the DaemonProcess display-name.
As shown in the docs:
display-name=value Defines a different name to show for the daemon
process when using the ps command to list processes. If the value is
%{GROUP} then the name will be (wsgi:group) where group is replaced
with the name of the daemon process group.
Note that only as many characters of the supplied value can be
displayed as were originally taken up by argv0 of the executing
process. Anything in excess of this will be truncated.
This feature may not work as described on all platforms. Typically it
also requires a ps program with BSD heritage. Thus on some versions of
Solaris UNIX the /usr/bin/ps program doesn’t work, but /usr/ucb/ps
does. Other programs which can display this value include htop.
You could:
Set a display-name of smaller length:
WSGIDaemonProcess yourapp user=www-data group=www-data processes=5 threads=5 display-name=wsws
And try to find them by:
ps aux | grep wsws
Or set it to %{GROUP} and filter using the name of the daemon process group (wsgi:group).

The way which processes are managed with mod_wsgi for each mode is described in:
http://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html
For embedded mode, where your WSGI application is run inside of the Apache child worker processes, Apache manages when processes are created and destroyed based on the Apache MPM settings. Because of how Apache manages the processes, they can be shutdown at any time if there is insufficient request throughput, or more processes could be created if request throughput increases. When running, the same process will handle many requests over time until it gets shutdown. In other words, Apache dynamically manages the number of processes.
Because of this dynamic process management, it is a bad idea to use embedded mode of mod_wsgi unless you know how to tune Apache properly and many other things as well. In short, never use embedded mode unless you have a good amount of experience with Apache and running Python applications with it. You can watch a video about why you wouldn't want to run in embedded mode at:
https://www.youtube.com/watch?v=k6Erh7oHvns
There is also the blog post:
http://blog.dscpl.com.au/2012/10/why-are-you-using-embedded-mode-of.html
So use daemon mode and verify that your configuration is correct and you are in fact using daemon mode by using the check in:
http://modwsgi.readthedocs.io/en/develop/user-guides/checking-your-installation.html#embedded-or-daemon-mode
For daemon mode, the WSGI application runs in a separate set of managed processed. These are created at the start and will run until Apache is restarted, or reloading of the process is triggered for various reasons, including:
The daemon process is sent a direct signal to shutdown by a user.
The code of the application sends itself a signal.
The WSGI script file is modified, which will trigger a shutdown so the WSGI application can be reloaded.
A defined request timeout occurs due to stuck or long running request.
A defined maximum number of requests has occurred.
A defined inactivity timeout expires.
A defined timer for periodic process restart expires.
A startup timeout is defined and the WSGI application failed to load in that time.
In these cases, when the process is shutdown, it is replaced.
More details about the various timeout options and how the processes respond to signals can be found in:
http://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html
More details about source code reloading and touching of the WSGI script file can be found in:
http://modwsgi.readthedocs.io/en/develop/user-guides/reloading-source-code.html
One item which is documented is how you can incorporate code which will look for any changes to Python code files used by your application. When a change occurs to any of the files, the process will be restarted by sending itself a signal. This should only be used for development and never in production.
If you are using mod_wsgi-express in development, which is preferable to hand configuring Apache yourself, you can use the --reload-on-changes option.
If sending a SIGTERM signal to the daemon process, there is a set shutdown sequence where it will wait a few seconds to wait for current requests to finish. If the requests don't finish, the process will be shutdown anyway. That period of time is dictated by the shutdown timeout. You shouldn't play with that value.
If sending a SIGUSR1 signal to the daemon process, by default it acts just like sending a SIGTERM signal. If however you specify the graceful timeout for shutdown, you can extend how long it will wait for current requests to finish. New requests will be accepting during that period. That graceful timeout also applies in other cases as well, such as maxmimum number of requests received, or timer for periodic restart triggered. If you need the timeout when using SIGUSR1 to be different to those cases, define the eviction timeout instead.
As to how to identify the daemon processes to be sent the signal, use the display-name of option WSGIDaemonProcess. Then use ps to identify the processes, or possibly use killall if it uses the modified process name on your platform. Send the daemon processes the SIGUSR1 signal if want more graceful shutdown and SIGTERM if want them to restart straight away.
If you want to track how long a daemon process has been running, you can use:
import mod_wsgi
metrics = mod_wsgi.process_metrics()
The metrics value will include output like the following for the process the call is made in:
{'active_requests': 1,
'cpu_system_time': 0.009999999776482582,
'cpu_user_time': 0.05000000074505806,
'current_time': 1525047105.710778,
'memory_max_rss': 11767808,
'memory_rss': 11767808,
'pid': 4774,
'request_busy_time': 0.001851,
'request_count': 2,
'request_threads': 2,
'restart_time': 1525047096.31548,
'running_time': 9,
'threads': [{'request_count': 2, 'thread_id': 1},
{'request_count': 1, 'thread_id': 2}]}
If you just want to know how many processes/threads are used for the current daemon process group you can use:
mod_wsgi.process_group
mod_wsgi.application_group
mod_wsgi.maximum_processes
mod_wsgi.threads_per_process
to get details about the process group. The number of process is fixed at this time for daemon mode and the name maximum_processes is just to be consistent with what the name is in embedded mode.
If you need to run code on process shutdown, you should NOT try and define your own signal handlers. Do that and mod_wsgi will actually ignore them as they will interfere with normal operation of Apache and mod_wsgi. Instead, if you need to run code on process shutdown, use atexit.register(). Alternatively, you can subscribe to special events generated by mod_wsgi and trigger something off the process shutdown event.

Edit: a more simple WSGI config is given in my question of Python WSGI handler directly in Apache .htaccess, not in VirtualHost
Based on Evhz's answer, I made a simple test to check that the processes are still running:
Apache config:
<VirtualHost *:80>
ServerName example.com
<Directory />
AllowOverride All
Require all granted
</Directory>
WSGIScriptAlias / /home/www/wsgi_test/app.wsgi
WSGIDaemonProcess yourapp user=www-data group=www-data processes=5 threads=5 display-name=testwsgi
</VirtualHost>
app.wsgi file:
import os, time
from bottle import route, template, default_app
os.chdir(os.path.dirname(__file__))
#route('/hello/<name>')
def index(name):
global i
i += 1
return template('<b>Hello {{name}}</b>! request={{i}}, pid={{pid}}',
name=name, i=i, pid=os.getpid())
i = 0
time.sleep(3) # wait 3 seconds to make the client notice we launch a new process!
application = default_app()
Now access http://www.example.com/hello/you many times:
The initial time.sleep(3) will help, from the client browser, to see exactly when a new process is started, and the request counter i will allow to see how many requests have been served by each process.
The PIDs will correspond to those present in ps aux | grep testwsgi:
Also the time.sleep(3) will happen maximum 5 times (at the startup of each of the 5 processes), then the processes should run forever, until we restart/stop the server or modify the app.wsgi file (modifying it triggers a restart of the 5 processes, you can see new PIDs).
[I'll check that by letting my test run now, and access http://www.example.com/hello/you in 2 days to see if it's still a previously-launched process or a new one!]
Edit: the next day, the same processes were still up and running. Now, two days after, when reloading the same URL, I noticed new processes were created... (Is there a time after which a process with no request dies?)

uWSGI, cherrypy and threading

preface: I would like to separate these problems into smaller questions, but apparently, I am missing some pieces of the puzzle and it seems impossible to me.
I developed my cherrypy application using cherrypy's built in WSGI server. I naively assumed that when the time comes, I will be able to use created WSGI Application class and deploy it using any WSGI compliant server.
I used this blog post to create my own (but very similar) cherrypy Plugin and Tool to connect to database using SQLAlchemy during http requests.
I expected that any server will somehow work like cherrypy's built in server:
main process will spawn X threads to satisfy X concurrent requests
my engine Plugin will create SQLalchemy engine with connection pool = X (so any request will have its connection)
on request arrival, my Tool will supply sql alchemy connection from pool
This flow does not match with uWSGI (as long as I understand it).
I assign my application.py in uWSGI configuration. This file looks something like this:
cherrypy.tools.db = DbConnectorTool()
cherrypy.engine.dbengine = DbEnginePlugin(cherrypy.engine, settings.database)
cherrypy.config.update({
'engine.dbengine.on': True
})
from myapp.application import Application
root = Application(settings)
application = cherrypy.Application(root, script_name='', config=settings)
I was using this application.py to mount my application into cherrypy's built in server when I was developing and testing it.
The problems are that uWSGI does not create any threads itself and my SQLAlchemy plugin is not working with it, because no cherrypy.engine is created.
Does uWSGI support threading in the meaning of using threads to serve multiple concurrent requests? Can I start these threads in my application.py? Will uWSGI understand it and use these threads for concurrent requests? And how can this be done? I think cherrypy can be used somehow, or not?
And what about my SQLAlchemy Plugin, how can I start cherrypy.engine when using only WSGI cherrypy.Application?
Any help or information that could help me will be appreciated.
Edit:
My uWSGI configuration:
<uwsgi>
<socket>127.0.0.1:9001</socket>
<master/>
<daemonize>/var/log/uwsgi/app.log</daemonize>
<logdate/>
<threads/>
<pidfile>/home/web/uwsgi.pid</pidfile>
<uid>uwsgi</uid>
<gid>uwsgi</gid>
<workers>2</workers>
<harakiri>90</harakiri>
<harakiri-verbose/>
<home>/home/web/</home>
<pythonpath>/home/web/instance</pythonpath>
<module>core.application</module>
<no-orphans/>
<touch-reload>/home/web/uwsgi-reload-web</touch-reload>
</uwsgi>

uWSGI uses worker processes, not threads. It's worth noting that it means that the globals are not shared between all requests any more. You can use SharedArea for global data.
The processes are forked by default, so make sure you're ok with that or adjust settings (see Things to know).
Get Cherrypy's WSGI application with cherrypy.tree.mount(root, config=settings) call.
If your DB plugin does not have threading / shared data issues, chances are it will work. Like you say, you may need cherrypy.engine.start(), but definitely not cherrypy.engine.block(), since your main thread is now uWSGI worker.

You should post your uWSGI config, otherwise it will be hard to understand what is going on.
By the way to spawn additional threads (per worker) you simply need to add --threads N

Python: moving from dev server to production server

I am developing an application with the bottlepy framework. I am using the standard library WSGIRefServer() to run a development server. It is a single threaded server.
Now when going into production, I will want to move to a multi-threaded production server, and there are many choices. Let's say I choose CherryPy.
Now, in my code, I am initializing a single wsgi application. Other than that, I am also initializing other things...
Memcached connection
Mako templates
MongoDB connection
Since standard library wsgiref is a single threaded server, and I am creating only a single wsgi app (wsgi callable), everything works just fine.
What I want to know is that when I move to the multi-threaded server, how will my wsgi app, initialization code, connections to different server, etc. behave.
Will a multi-threaded server create a separate instance of wsgi app for every thread. And will a new thread be spawned for each new request (which then means a new wsgi app for each request)?
Will my connections to memcached, mongoDB, etc, be shared across threads or not. What else will be shared between threads
Please explain the request-response cycle for a threaded server

In general your application is using wsgi compliant framework and you shouldn't be afraid of multi-threaded / single-threaded server side. It's meant to work transparent and has to react same way despite of what kind of server is it, as long as it is wsgi compliant.
Every code block before bottle.run() will be run only once. As so, every connection (database, memcached) will be instantiated only once and shared.
When you call bottle.run() bottlepy starts wsgi server for you. Every request to that server fires some wsgi callable inside bottlepy framework. You are not really interested if it is single or multi -threaded environment, as long as you don't do something strange.
For strange i mean for instance synchronizing something through global variables. (Exception here is global request object for which bottlepy ensures that it contains proper request in proper context).
And in response to first question on the list: request may be computed in newly spawned thread or thread from the pool of threads (CherryPy is thread-pooled)

Can I have some code constantly run inside Django like a daemon

I'm using mod_wsgi to serve a django site through Apache. I also have some Python code that runs as a background process (dameon?). It keeps polling a server and inserts data into one of the Django models. This works fine but can I have this code be a part of my Django application and yet able to constantly run in the background? It doesn't need to be a process per se but a art of the Django site that is active constantly. If so, could you point me to an example or some documentation that would help me accomplish this?
Thanks.

You could either set up a cron job that runs some function you have defined, or - the more advanced and probably recommended method, integrate celery in your project (which is quite easy, actually).

You could create a background thread from the WSGI script when it is first being imported.
import threading
import time
def do_stuff():
time.sleep(60)
... do periodic job
_thread = threading.Thread(target=do_stuff)
_thread.setDaemon(True)
_thread.start()
For this to work though you would have to be using only one daemon process otherwise each process would be doing the same thing which you probably do not want.
If you are using multiple process in daemon process group, an alternative is to create a special daemon process group which the only purpose of is to run this background thread. In other words, the process doesn't actually receive any requests.
You can do this by having:
WSGIDaemonProcess django-jobs processes=1 threads=1
WSGIImportScript /usr/local/django/mysite/apache/django.wsgi \
process-group=django-jobs application-group=%{GLOBAL}
The WSGIImportScript directive says to load that script and run it on startup in the context of the process group 'django-jobs'.
To save having multiple scripts, I have pointed it at what would be your original WSGI script file you used for WSGIScriptAlias. We don't want it to run when it is loaded by that directive though, so we do:
import mod_wsgi
if mod_wsgi.process_group == 'django-jobs':
_thread = threading.Thread(target=do_stuff)
_thread.setDaemon(True)
_thread.start()
Here it looks at the name of the daemon process group and only runs when started up within the special daemon process group set up with single process just for this.
Overall you are just using Apache as a big gloried process manager, albeit one which is already known to be robust. It is a bit of overkill as this process will consume additional memory on top of those accepting and handling requests, but depending on the complexity of what you are doing it can still be useful.
One cute aspect of doing this is that since it is still a full Django application in there, you could map specific URLs to just this process and so provide a remote API to manage or monitor the background task and what it is doing.
WSGIDaemonProcess django-jobs processes=1 threads=1
WSGIImportScript /usr/local/django/mysite/apache/django.wsgi \
process-group=django-jobs application-group=%{GLOBAL}
WSGIDaemonProcess django-site processes=4 threads=5
WSGIScriptAlias / /usr/local/django/mysite/apache/django.wsgi
WSGIProcessGroup django-site
WSGIApplicationGroup %{GLOBAL}
<Location /admin>
WSGIProcessGroup django-jobs
</Location>
Here, all URLs except for stuff under /admin run in 'django-site', with /admin in 'django-jobs'.
Anyway, that addresses the specific question of doing it within the Apache mod_wsgi daemon process as requested.
As pointed out, the alternative is to have a command line script which sets up and loads Django and does the work and execute that from a cron job. A command line script means occasional transient memory usage, but startup cost for job is higher as need to load everything each time.

I previously used a cron job but I telling you, you will switch to celery after a while.
Celery is the way to go. Plus you can tasked long async process so you can speed up the request/response time.