Develop Reference

Uninterrupted nginx + 2 gunicorns setup

So what's the trick? Nginx is facing the client. Normally the requests are forwarded to gunicorn A at port 80.
You can't run code update in-place, since something might be wrong. So you do a fresh code checkout and launch a separate gunicorn B on some port 5678.
Once you test the new code on a development/testing database, you:
Adjust gunicorn B to point to the database, but do not send any requests.
Stop gunicorn A. Nginx now, ever so briefly, responds with an error.
Set nginx to point to gunicorn B, still at port 5678.
Restart nginx.
Is this about right? Do you just write a script to run the four actions faster and minimize the duration (between steps 2 and 4) the server responds with an error?

Nginx supports configuration reloading. Using this feature, updating your application can work like this:
Start a new instance Gunicorn B.
Adjust the nginx configuration to forward traffic to Gunicorn B.
Reload the nginx configuration with nginx -s reload. After this, Gunicorn B will serve new requests, while Gunicorn A will still finish serving old requests.
Wait for the old nginx worker process to exit (which means all requests initiated before the reload are now done) and then stop Gunicorn A.
Assuming your application works correctly with two concurrent instances, this gives you a zero-downtime update.
The relevant excerpt from the nginx documentation:
Once the master process receives the signal to reload configuration, it checks the syntax validity of the new configuration file and tries to apply the configuration provided in it. If this is a success, the master process starts new worker processes and sends messages to old worker processes, requesting them to shut down. Otherwise, the master process rolls back the changes and continues to work with the old configuration. Old worker processes, receiving a command to shut down, stop accepting new connections and continue to service current requests until all such requests are serviced. After that, the old worker processes exit.

GCP instance returns ERR_CONNECTION_REFUSED for Ajax 127.0.0.1 route

I have a flask application using bokeh that is running in a Docker container, and it works when I use it on local machines.
However, when I deploy it to a GCP instance, even though I can reach the server, I have some AjaxDataSource() objects which are failing to connect.
Some details,
All the machines, local and gcp vm are running Ubuntu 18.04
The flask app is started like this,
app.run(host="0.0.0.0", port=6600, debug=False)
The Ajax route looks like this,
http://127.0.0.1:6600/land/tmidemo/data_rate?name=ResultBaseKeysV1
The GCP firewall rules look like,
Name Type Targets Filters Protocols / ports Action Priority Network
tmiserver-egress Egress Apply to all IP ranges: 0.0.0.0/0 tcp:6600 udp:6600 Allow 1000 default
tmiserver-ingress Ingress Apply to all IP ranges: 0.0.0.0/0 tcp:6600 udp:6600 Allow 1000 default
The docker container is run like this,
docker run --net tminet --hostname=TEST -p 6600:6600 -v $(pwd):/app/public --name myserver --rm myserver
I am not using a Bokeh server. The AjaxDataSource() calls point back to the flask application, not another (bokeh) server
There is a lot that works,
able to use the GCP external ip address and reach the server
going from web page to web page works, so flask routing is working
Whats NOT working is that Ajax() call which uses 127.0.0.1, although this DOES work when I run the container on a local machine.
The error I see in the inspect window is ERR_CONNECTION_REFUSED
The GCP instance hosts.conf DOES include a line for 127.0.0.1 localhost
I tried (from here) on the GCP VM instance, same result,
iptables -A INPUT -i docker0 -j ACCEPT
I also tried (from here) changing the Docker run network to --net="host" and the result is identical.
I also tried adding --add-host localhost:127.0.0.1 to the Docker run command, same result.
I think the problem is configuring the GCP to know how to route a request to 127.0.0.1, but I don't know where to check, configure this, beyond what I have already done.

I wasn't able to specifically resolve the issue I was having, but I tried a different approach to the URL for the AjaxDataSource() and it worked and I think a better approach...
I used Flask url_for() function to create a link to the route that the AjaxDataSource() needs and this worked. The resulting link looks something like,
/land/tmidemo/data_rate/ResultBaseKeysV1
ie, no http://127.0.0.1, and this seems to work in all cases, my dev environment and GCP.
I think I tried this a long time ago and it didn't work, because I use "flask" URLs all over the place, but for some reason I thought I needed "http://127.0.0.1" for the Ajax stuff. Its works now.... moving on!

Gunicorn timeout exception for specific urls

I use Gunicorn for my python web service with NGINX.
I observe the following problem: even though I have client_max_body_size is big enough in nginx's config, I still get 502 error code when I try to upload big files. I believe that's because uploading is not finished within timeout 90 seconds set when I run my gunicorn workers.
So, my question - is it possible to specify timeout exception for some URLs in Gunicorn command line or config? For example, for file upload URL I want higher timeout and small one for other URLs. If not, what workaround could be implemented?
Thanks in advance!

Python process suspends on SSH logout after nohup/screen

I have a remote server through Blue Host that's intended to run a server based on Twisted for Python. The only access I have to it is over SSH, so to keep Python running after I log out I tried using nohup python server.py & and screen -dm python server.py, getting the same results for each. Everything works fine until I log out of SSH - even though Python is running in the background as expected, once I've logged out, my client can no longer communicate with the server. The strange part is that if I log back in over SSH and check the running processes with ps aux, I see Python running and my client can successfully communicate with the server again. Even if I don't type anything at all once I log back in, everything works as expected. But, of course, as soon as I log back out, it's as if the server is gone.
I've contacted support for the hosting service in case this is some oddity on their end, but hopefully this is something that can be resolved on my end instead.
Edit: Looks like Blue Host doesn't want me doing server-y stuff without buying the VPS upgrade so it looks like that's the big problem.
Edit 2: Okay, so in case anybody ends up having a similar problem, here's what the main issue turned out to be. I was mistaken in my original description; I was able to connect to the server but I was getting kicked off immediately for what turned out to be a MySQL error. I guess trying to connect to a localhost database with no active connection somehow causes problems, so instead I changed the MySQL connection command to connect to my site's IP address instead, even though it was the same IP as the server. That seemed to do the trick in terms of my main issue.

Don't use this method to keep the server process running. Instead try using supervisor (apt-get install supervisor). It allows you to daemonize your process, and ability to stop/restart etc.
Here's a sample config entry (/etc/supervisor/supervisord.conf):
[program:my_server]
command=python /path/to/server/server.py
directory=/path/to/server/
autostart=true
autorestart=true
stdout_logfile=/var/log/server.log
stderr_logfile=/var/log/server_error.log
user=your_linux_user_name
After you edit your config, do
sudo service supervisor stop
sudo service supervisor start #need to do this - doing a `restart` doesn't reload the config file!
your server should now be running properly. You can manage its lifecycle via sudo supervisorctl

Django Nginx Gunicorn = 504 Timeout

I'm trying to publish a Django application on the production server using Nginx + Gunicorn. When I doing a simple stress test on the server (holding the F5 key for a minute) the server returns a 504 Gateway Time-out error. Why does this happen? This error only appears for the user when doing multiple concurrent requests, or the system will be fully unavailable to everyone?

When you hold down F5:
You've started hundreds of requests.
Those requests have filled your gunicorn request queue.
The request handlers have not been culled as soon as the connection drops.
Your latest requests are stuck in the queue behind all the previous requests.
Nginx times out.
For everyone.
Solutions:
Set up rate-limiting buckets in Nginx, keyed on IP, such that one malicious user can't spam you with requests and DOS your site.
Set up a global rate-limiting bucket in Nginx such that you don't overfill your request queue.
Make Nginx serve a nice "Reddit is under heavy load" style page, so users know that this is a purposeful event
Or:
Replace gunicorn with uwsgi. It's faster, more memory efficient, integrates smoothly with nginx, and most importantly: It will kill the request handler immediately if the connection drops, such that F5 spam can't kill your server.

https://medium.com/#paragsharma.py/504-gateway-timeout-django-gunicorn-nginx-4570deaf0922
504 can be caused by gunicorn timeout you need to start it with --timeout arg like
gunicorn --access-logfile - --workers 3 --timeout 300 --bind unix:/home/ubuntu/myproject/myproject.sock myproject.wsgi:application