So what's the trick? Nginx is facing the client. Normally the requests are forwarded to gunicorn A at port 80.
You can't run code update in-place, since something might be wrong. So you do a fresh code checkout and launch a separate gunicorn B on some port 5678.
Once you test the new code on a development/testing database, you:
Adjust gunicorn B to point to the database, but do not send any requests.
Stop gunicorn A. Nginx now, ever so briefly, responds with an error.
Set nginx to point to gunicorn B, still at port 5678.
Restart nginx.
Is this about right? Do you just write a script to run the four actions faster and minimize the duration (between steps 2 and 4) the server responds with an error?
Nginx supports configuration reloading. Using this feature, updating your application can work like this:
Start a new instance Gunicorn B.
Adjust the nginx configuration to forward traffic to Gunicorn B.
Reload the nginx configuration with nginx -s reload. After this, Gunicorn B will serve new requests, while Gunicorn A will still finish serving old requests.
Wait for the old nginx worker process to exit (which means all requests initiated before the reload are now done) and then stop Gunicorn A.
Assuming your application works correctly with two concurrent instances, this gives you a zero-downtime update.
The relevant excerpt from the nginx documentation:
Once the master process receives the signal to reload configuration, it checks the syntax validity of the new configuration file and tries to apply the configuration provided in it. If this is a success, the master process starts new worker processes and sends messages to old worker processes, requesting them to shut down. Otherwise, the master process rolls back the changes and continues to work with the old configuration. Old worker processes, receiving a command to shut down, stop accepting new connections and continue to service current requests until all such requests are serviced. After that, the old worker processes exit.
Related
I am running a django server in gunicorn. I noticed after creating a certain number of threads it cannot start new thread.
This is my gunicorn service file:
So, I tried to see the status of the gunicorn service file:
sudo systemctl status medai-app.service
output :
It's showing the limit of tasks is 1143. I want to know how to increase this number. How is it determined as I did not find any configuration setting to tune this. My user's maxproc limit is 5000.
The number of threads that's been created doesn't decrease, why? Shouldn't they be killed after finishing a task?
How can a fully stop a Heroku dyno using the Heroku API so it doesn't restart?
I have Heroku set up to run a Python script that always loops and never exits to check sensors. This main.py script is run as a Dyno worker via my Procfile. Right now, it always runs as anticipated. It even restarts upon a crash, which is also helpful for this app.
Using the Heroku API, I would like to be able to completely stop this process. This won't be reoccurring often, just a failsafe kill-switch. I can stop the dyno worker by logging in to Heroku and clicking the edit button in the Resources tab and turn off the worker. But, I would like to turn off the worker via the API.
I have tried the following API calls (via Postman), which both stop the current process, but after a few seconds, Heroku starts the worker back up again.
Stop the current Dyno
POST https://api.heroku.com/apps/<app_id>/dynos/worker.1/actions/stop
Deletes / restarts all dynos in the app
DELETE https://api.heroku.com/apps/<app_id>/dynos
Also, I can put the app in to maintenance mode, but that only stops HTTP traffic, not the Dynos
PATCH https://api.heroku.com/apps/<app_id>
with payload of {"maintenance":true}
One option is completely deleting the app, but that's a bit too much for me as it will also remove all log files, and be a pain to set it back up again.
DELETE https://api.heroku.com/apps/<app_id>
Is there any way to completely stop a Dyno or app via the API until I manually start it back up again?
You have to use Heroku Formation API to scale the processes up and down, it will not restart the App by itself.
For detailed answer check this.
Seems like you are stoping the worker rather than a dyno. You have to provide a dyno id.
Stoping dyno
After stopping correctly restart to start it again.
restarting the dyno
While testing nginx server with uwsgi and django I am having problem with uwsgi process. I am sending two posts, which are taking a lot of time. Meanwhile server processing I am sending get request from webrowser and I must wait till this two post finished. I am starting uwsgi with this command:
cd /home/pi/cukierek && uwsgi -
-max-requests=5000
--socket /tmp/cukierek.sock
--module config.wsgi
--master-fifo /tmp/cukierek.fifo
--chmod-socket=777 --processes 2
--daemonize /home/pi/cukierek/wsgi.log
--enable-threads
It is possible to get answer from browser while this two post are beeing in progress ? I am using default nginx settings.
You have a uwsgi server configured to spawn 2 processes. Then you run 2 long requests. Those 2 processes are busy with the long requests, so new requests must wait until the long requests finish.
If you want to send new reqeusts to the server while the long requests run, increase the processes to more than 2 (ie --processes 4)
I would like to run APScheduler which is a part of WSGI (via Apache's modwsgi with 3 workers) webapp. I am new in WSGI world thus I would appreciate if you could resolve my doubts:
If APScheduler is a part of webapp - it becomes alive just after first request (first after start/reset Apache) which is run at least by one worker? Starting/resetting Apache won't start it - at least one request is needed.
What about concurrent requests - would every worker run same set of APScheduler's tasks or there will be only one set shared between all workers?
Would once running process (webapp run via worker) keep alive (so APScheduler's tasks will execute) or it could terminate after some idle time (as a consequence - APScheduler's tasks won't execute)?
Thank you!
You're right -- the scheduler won't start until the first request comes in.
Therefore running a scheduler in a WSGI worker is not a good idea. A better idea would be to run the scheduler in a separate process and connect to the scheduler when necessary via some RPC mechanism like RPyC or Execnet.
I'm trying to publish a Django application on the production server using Nginx + Gunicorn. When I doing a simple stress test on the server (holding the F5 key for a minute) the server returns a 504 Gateway Time-out error. Why does this happen? This error only appears for the user when doing multiple concurrent requests, or the system will be fully unavailable to everyone?
When you hold down F5:
You've started hundreds of requests.
Those requests have filled your gunicorn request queue.
The request handlers have not been culled as soon as the connection drops.
Your latest requests are stuck in the queue behind all the previous requests.
Nginx times out.
For everyone.
Solutions:
Set up rate-limiting buckets in Nginx, keyed on IP, such that one malicious user can't spam you with requests and DOS your site.
Set up a global rate-limiting bucket in Nginx such that you don't overfill your request queue.
Make Nginx serve a nice "Reddit is under heavy load" style page, so users know that this is a purposeful event
Or:
Replace gunicorn with uwsgi. It's faster, more memory efficient, integrates smoothly with nginx, and most importantly: It will kill the request handler immediately if the connection drops, such that F5 spam can't kill your server.
https://medium.com/#paragsharma.py/504-gateway-timeout-django-gunicorn-nginx-4570deaf0922
504 can be caused by gunicorn timeout you need to start it with --timeout arg like
gunicorn --access-logfile - --workers 3 --timeout 300 --bind unix:/home/ubuntu/myproject/myproject.sock myproject.wsgi:application