Before posting this, I have read quite a few resources online, including the mod_wsgi wiki, but I am confused about how exactly Apache processes/threads interact with mod_wsgi.
This is my current understanding: Apache can be configured to run such that one or more child processes can handle incoming requests, and each of these child processes can be configured to in turn use one or more threads to service requests. After that, things start getting hazy for me. My doubts are:
What is a WSGIDaemonProcess, and who actually calls my Django app using the python sub interpreter?
If I have my Django app running under a mode where multiple threads are allowed in a single Apache child process - does that mean that multiple requests could be simultaneously accessing my app at the same time? If so - would doing something like setting a module level variable (say that of an user's ID) could be over-written by other parallel requests and lead to non-thread safe behavior?
For the case above, with Python's global interpreter lock, would the threads actually be executing in parallel?
Answers to each of the points.
1 - WSGIDaemonProcess/WSGIProcessGroup indicate that mod_wsgi should fork of a separate process for running the WSGI application in. This is a fork only and not a fork/exec, so mod_wsgi is still in control of it. When it is detected that a URL maps to a WSGI application running in daemon mode, then mod_wsgi code in the Apache child worker processes will proxy the request details through to the daemon mode process where the mod_wsgi code there reads it and calls up into your WSGI application.
2 - Yes, multiple requests can be operating concurrently and be wanting to modify the module global data at the same time.
3 - For the time that execution is within Python itself then no, they aren't strictly running in parallel as the global interpreter lock means that only one thread can be executing Python code at a time. The Python interpreter will periodically switch which thread is getting to run. If one of the threads calls into C code and releases the GIL then at least for the time that thread is in that state it can run in parallel to other threads, running in Python or C code. As example, when calls are made down into the Apache/mod_wsgi layer to write back response data, the GIL is released. This means that the actual writing back of response data at the lower layers is not prevent other threads from running.
Related
What exactly does passing threaded = True to app.run() do?
My application processes input from the user, and takes a bit of time to do so. During this time, the application is unable to handle other requests. I have tested my application with threaded=True and it allows me to handle multiple requests concurrently.
As of Flask 1.0, the WSGI server included with Flask is run in threaded mode by default.
Prior to 1.0, or if you disable threading, the server is run in single-threaded mode, and can only handle one request at a time. Any parallel requests will have to wait until they can be handled, which can lead to issues if you tried to contact your own server from a request.
With threaded=True requests are each handled in a new thread. How many threads your server can handle concurrently depends entirely on your OS and what limits it sets on the number of threads per process. The implementation uses the SocketServer.ThreadingMixIn class, which sets no limits to the number of threads it can spin up.
Note that the Flask server is designed for development only. It is not a production-ready server. Don't rely on it to run your site on the wider web. Use a proper WSGI server (like gunicorn or uWSGI) instead.
How many requests will my application be able to handle concurrently with this statement?
This depends drastically on your application. Each new request will have a thread launched- it depends on how many threads your machine can handle. I don't see an option to limit the number of threads (like uwsgi offers in a production deployment).
What are the downsides to using this? If i'm not expecting more than a few requests concurrently, can I just continue to use this?
Switching from a single thread to multi-threaded can lead to concurrency bugs... if you use this be careful about how you handle global objects (see the g object in the documentation!) and state.
What exactly does passing threaded = True to app.run() do?
My application processes input from the user, and takes a bit of time to do so. During this time, the application is unable to handle other requests. I have tested my application with threaded=True and it allows me to handle multiple requests concurrently.
As of Flask 1.0, the WSGI server included with Flask is run in threaded mode by default.
Prior to 1.0, or if you disable threading, the server is run in single-threaded mode, and can only handle one request at a time. Any parallel requests will have to wait until they can be handled, which can lead to issues if you tried to contact your own server from a request.
With threaded=True requests are each handled in a new thread. How many threads your server can handle concurrently depends entirely on your OS and what limits it sets on the number of threads per process. The implementation uses the SocketServer.ThreadingMixIn class, which sets no limits to the number of threads it can spin up.
Note that the Flask server is designed for development only. It is not a production-ready server. Don't rely on it to run your site on the wider web. Use a proper WSGI server (like gunicorn or uWSGI) instead.
How many requests will my application be able to handle concurrently with this statement?
This depends drastically on your application. Each new request will have a thread launched- it depends on how many threads your machine can handle. I don't see an option to limit the number of threads (like uwsgi offers in a production deployment).
What are the downsides to using this? If i'm not expecting more than a few requests concurrently, can I just continue to use this?
Switching from a single thread to multi-threaded can lead to concurrency bugs... if you use this be careful about how you handle global objects (see the g object in the documentation!) and state.
I've got a mod_wsgi server setup with 5 processes and a celery worker queue (2 of them) all on the same VM. I'm running into problems where the loggers are stepping on each other and while it appears there are some solutions if you are using python multiprocessing, I don't see how that applies to mod_wsgi processes combined also with celery processes.
What is everyone else doing with this problem? The celery tasks are using code that logs in the same files as the webserver code.
Do I somehow have to add a pid to the logfilename? That seems like it could get messy fast with lots of logfiles with unique names and no real coherent way to pull them all back together.
Do I have to write a log daemon that allows all the processes to log to it? If so, where do you start it up so that it is ready for all of the processes that might want to log.....
Surely there is some kind of sane pattern out there for this, I just don't know what it is yet.
As mentioned in the docs, you could use a separate server process which listens on a socket and logs to different destinations, and has whatever logging configuration you want (in terms of files, console and so on). The other processes just configure a SocketHandler to send their events to the server process. This is generally better than separate log files with pids in their filenames.
The logging docs contain an example socket server implementation which you can adapt to your needs.
I'm running a Django application on Apache with mod_wsgi. Will there be any downtime during an upgrade?
Mod_wsgi is running in daemon mode, so I can reload my code by touching the .wsgi script file, as described in the "ReloadingSourceCode" document: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode. Presumably, that reload requires some non-zero amount of time. What happens if a request comes in during the reload? Will Apache queue the request and then complete it once the wsgi daemon is ready?
The documentation includes the following statement:
So, if you are using Django in daemon mode and needed to change your 'settings.py' file, once you have made the required change, also touch the script file containing the WSGI application entry point. Having done that, on the next request the process will be restarted and your Django application reloaded.
To me, that suggests that Apache will gracefully handle every request, but I thought I would ask to be sure. My app isn't critical (a little downtime wouldn't be disastrous) so the question is mostly academic.
Thank you.
In daemon mode there is no concept of a graceful restart when WSGI script file is touched to force a download. That is, unlike Apache itself, which will start new Apache server child processes while waiting for old processes to finish up with current requests, for mod_wsgi daemon processes, the existing process must exit before a new one starts up.
The consequences of this are that mod_wsgi can't wait indefinitely for current requests to complete. If it did, then there is a risk that if all daemon processes are tied up waiting for current requests to finish, that clients would see a noticeable delay in being handled.
At the other end of the scale however, the daemon process can't be immediately killed as that would cause current requests to be interrupted.
A middle ground therefore exists. The daemon process will wait for requests to finish before exiting, but if they haven't completed within the shutdown period, then the daemon process will be forcibly quit and the active requests will be interrupted.
The period of this shutdown timeout defaults to 5 seconds. It can be overridden using the shutdown-timeout option to WSGIDaemonProcess directive, but due consideration should be given to the effects of changing it.
Thus, in respect of this specific issue, if you have long running requests still active when the first request comes in after you touched the WSGI script file, there is the risk that the active long requests will be interrupted.
The next notable thing you may see is that even if there are no long running requests and processes shutdown promptly, then it is still necessary to load up the WSGI application again within the new process. The time this takes will be seen as a delay in handling the request. How big that delay is will depend on the framework and your application. The worst offender as far as time taken to start up that I know of is TurboGears. Django somewhat better and the best as far as quick start up times being lightweight micro frameworks such as Flask.
Do note that any new requests which come in while these shutdown and startup delays occur should not be lost. This is because the HTTP listener socket has a certain depth and connections queue up in that waiting to be accepted. If the number of requests arriving is huge though and that queue fills up, then you will start to see connection refused errors in the browser.
No, there will be no downtime. Requests using the old code will complete, and new requests will use the new code.
There will be a small bit more load on the server as the new code loads but unless your application is colossal and your servers are already nearly overloaded this will be unnoticeable.
This is like the apachectl graceful command for Apache as a whole, which tells it to start a new configuration without downtime.
I'm using the python signals library to kill a function if it runs longer than a set period of time.
It works well in my tests but when hosted on the server I get the following error
"signal only works in main thread"
I have set the WSGI signals restriction to be off in my httpd.conf
WSGIRestrictSignal Off
as described
http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Registration_Of_Signal_Handlers
I'm using the functions from the recipe described here
http://code.activestate.com/recipes/307871/
Not sure what I'm doing wrong. Is there a way to ensure that the signals are called in the main thread?
The only time any code under Apache/mod_wsgi runs as main thread is when a WSGI script file is being imported via WSGIImportScript or equivalent methods. Although one could use that method to register the signal handler from the main thread, it will be of no use as all subsequent requests are serviced via secondary threads and never the main thread. As a consequence, I don't think your signal handler will ever run as recollect that it can only run if the main thread is actually doing something, and that will not be the case as the main thread, will be blocked in Apache/mod_wsgi simply waiting for the process to be shutdown.
What is the operation doing that you are trying to kill? Doing it within context of web application, especially if web application is multi threaded probably isn't a good idea.