I created an endpoint on my flask which generates a spreadsheet from a database query (remote db) and then sends it as a download in the browser. Flask doesn't throw any errors. Uwsgi doesn't complain.
But when I check nginx's error.log I see a lot of
2014/12/10 05:06:24 [error] 14084#0: *239436 upstream prematurely
closed connection while reading response header from upstream, client:
34.34.34.34, server: me.com, request: "GET /download/export.csv HTTP/1.1", upstream: "uwsgi://0.0.0.0:5002", host: "me.com", referrer:
"https://me.com/download/export.csv"
I deploy the uwsgi like
uwsgi --socket 0.0.0.0:5002 --buffer-size=32768 --module server --callab app
my nginx config:
server {
listen 80;
merge_slashes off;
server_name me.com www.me.cpm;
location / { try_files $uri #app; }
location #app {
include uwsgi_params;
uwsgi_pass 0.0.0.0:5002;
uwsgi_buffer_size 32k;
uwsgi_buffers 8 32k;
uwsgi_busy_buffers_size 32k;
}
}
server {
listen 443;
merge_slashes off;
server_name me.com www.me.com;
location / { try_files $uri #app; }
location #app {
include uwsgi_params;
uwsgi_pass 0.0.0.0:5002;
uwsgi_buffer_size 32k;
uwsgi_buffers 8 32k;
uwsgi_busy_buffers_size 32k;
}
}
Is this an nginx or uwsgi issue, or both?
As mentioned by #mahdix, the error can be caused by Nginx sending a request with the uwsgi protocol while uwsgi is listening on that port for http packets.
When in the Nginx config you have something like:
upstream org_app {
server 10.0.9.79:9597;
}
location / {
include uwsgi_params;
uwsgi_pass org_app;
}
Nginx will use the uwsgi protocol. But if in uwsgi.ini you have something like (or its equivalent in the command line):
http-socket=:9597
uwsgi will speak http, and the error mentioned in the question appears. See native HTTP support.
A possible fix is to have instead:
socket=:9597
In which case Nginx and uwsgi will communicate with each other using the uwsgi protocol over a TCP connection.
Side note: if Nginx and uwsgi are in the same node, a Unix socket will be faster than TCP. See using Unix sockets instead of ports.
Change nginx.conf to include
sendfile on;
client_max_body_size 20M;
keepalive_timeout 0;
See self answer uwsgi upstart on amazon linux for full example
In my case, problem was nginx was sending a request with uwsgi protocol while uwsgi was listening on that port for http packets. So either I had to change the way nginx connects to uwsgi or change the uwsgi to listen using uwsgi protocol.
I had the same sporadic errors in Elastic Beanstalk single-container Docker WSGI app deployment. On EC2 instance of the environment upstream configuration looks like:
upstream docker {
server 172.17.0.3:8080;
keepalive 256;
}
With this default upstream simple load test like:
siege -b -c 16 -t 60S -T 'application/json' 'http://host/foo POST {"foo": "bar"}'
...on the EC2 led to availability of ~70%. The rest were 502 errors caused by upstream prematurely closed connection while reading response header from upstream.
The solution was to either remove keepalive setting from the upstream configuration, or which is easier and more reasonable, is to enable HTTP keep-alive at uWSGI's side as well, with --http-keepalive (available since 1.9).
Replace uwsgi_pass 0.0.0.0:5002; with uwsgi_pass 127.0.0.1:5002; or better use unix sockets.
It seems many causes can stand behind this error message. I know you are using uwsgi_pass, but for those having the problem on long requests when using proxy_pass, setting http-timeout on uWSGI may help (it is not harakiri setting).
There are many potential causes and solutions for this problem. In my case, the back-end code was taking too long to run. Modifying these variables fixed it for me.
Nginx:
proxy_connect_timeout, proxy_send_timeout, proxy_read_timeout, fastcgi_send_timeout, fastcgi_read_timeout, keepalive_timeout, uwsgi_read_timeout, uwsgi_send_timeout, uwsgi_socket_keepalive.
uWSGI: limit-post.
I fixed this issue by passing socket-timeout = 65 (uwsgi.ini file) or --socket-timeout=65 (uwsgi command line) option in uwsgi. We have to check with different value depends on the web traffic. This value socket-timeout = 65 in uwsgi.ini file worked in my case.
I fixed this by reverting to pip3 install uwsgi.
I was trying out the setup with Ubuntu and Amazon Linux side by side. I initially used a virtual environment and did pip3 install uwsgi both systems work fine. Later, I did continue the setup with virtual env turned off. On Ubuntu I install using pip3 install uwsgi and on Amazon Linux yum install uwsgi -y. That was the source of the problem for me.
Ubuntu works fine, but not the Amazon Linux
The fix,
yum remove uwsgi and pip3 install uwsgi restart and it works fine.
This issue can also be caused by a mismatch between timeout values.
I had this issue when nginx had a keepalive_timeout of 75s, while the upstream server's value was a few seconds.
This caused the upstream server to close the connection when its timeout was reached, and nginx logged Connection reset by peer errors.
When having such abrupt "connection closed" errors, please check the upstream timeout values are higher than nginx' values (see Raphael's answer for a good list to check)
Related
I want to run django with gunicorn and nginx as a proxy server on a remote Ubuntu VPS.
The site works with djangos dev server:
python manage.py runserver 0.0.0.0:8000
The site works with gunicorns server (even static files don't work):
gunicorn my_project.wsgi --bind 0.0.0.0:8000
But with nginx on top I get the following error:
This site can’t be reached ... refused to connect. ERR_CONNECTION_REFUSED
Also both nginx log files error.log & access.log are empty.
Here is how I configured nginx:
server {
listen 80;
server_name my_ip_address;
location / {
proxy_pass http://127.0.0.1:8001;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
}
}
In this case gunicorn runs with --bind 127.0.0.1:8001 of course.
Status check (service nginx status) returns:
● nginx.service - A high performance web server and a reverse proxy server
Active: active (running) since Fri 2019-09-20 07:41:00 UTC; 1min 19s ago
Starting A high performance web server and a reverse proxy server...
nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
Started A high performance web server and a reverse proxy server.
First, check your configuration with nginx -t. The configuration you posted is not valid as a standalone config file, but I assume you are using the common nginx config structure of having a main nginx.conf and sites-available and sites-enabled directories.
If it does not complain, introduce an error, e.g. by removing a closing bracket, and try again. If it still doesn't complain, your configuration is not being picked up by nginx.
In this case, check if you created a correct symlink from sites-enabled/your_config to sites-available/your_config.
If that all seems correct:
check if nginx is actually running: ps aux | grep nginx
check if nginx is listening to port 80: netstat -tulpen | grep ":80"
check firewall rules
I follow the steps in http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html but when all the steps done without any error I visit 127.0.0.1:8000, it response with a time-out, my nginx log shows that
upstream timed out (110: Connection timed out) while reading response header from upstream,
By the way, I can access 127.0.0.1:8001 where uwsgi and django works well.
And I can access image in 127.0.0.1:8000/image/1.jpg as well, but just cannot access 127.0.0.1:8000
here's my nginx.conf
upstream django {
server 127.0.0.1:8001;
}
server {
listen 8000;
server_name 127.0.0.1
charset utf-8;
client_max_body_size 75M;
location /media {
alias /home/zhaolei/virtualdjango/bin/mysite/media;
}
location /image {
alias /home/zhaolei/virtualdjango/bin/mysite/image;
}
location / {
uwsgi_pass django;
include /home/zhaolei/virtualdjango/bin/mysite/uwsgi_params;
}
}
I use uwsgi --http 127.0.0.1:8001 --chdir=mysite --module=mysite.wsgi
to run uwsgi. I use uwsgi_params hosts in https://github.com/nginx/nginx/blob/master/conf/uwsgi_params
uWSGI have 2 protocols to communicate with web server. One of them is normal HTTP protocol, that can also be used to communicate directly with clients. But there is also special uwsgi protocol, optimized for communication between HTTP Proxy server and uWSGI.
That protocol is used by nginx when using uwsgi_pass directive, and by uWSGI when you're starting your uWSGI server with --socket param.
If you're starting uWSGI with --http param, uWSGI will use HTTP protocol (that is what you're doing), but if nginx is still using uwsgi_pass it's expecting uWSGI protocol on that socket, not HTTP.
To fix it you have to either change your uwsgi start command to use --socket instead of --http (that's recommended way, but you won't be able to check if uWSGI is functioning properly by entering 127.0.0.8001 directly in your browser, but that's okay: if your command with --http worked properly, there won't be any difference using --socket) or use proxy_pass instead of uwsgi_pass in your nginx config.
And it's described on link that you're mentioned, right here
I am stuck with setting the https with django on aws with nginx and gunicorn.
my configuration is:
server {
listen 80;
listen 443 ssl;
server_name logitech.enterpriselist.com;
rewrite ^ https://logitech.enterpriselist.com$request_uri? permanent;
root /home/ubuntu/git/elist/static/;
#` ssl on;
ssl_certificate /etc/ssl/elist.crt;
ssl_certificate_key /etc/ssl/elist.key;
location / {
# proxy_pass http://logitech.enterpriselist.com/;
}
location /static/ {
alias /home/ubuntu/git/elist/static/;
}
}
It is working fine with http with port 8001:
gunicorn configs.wsgi:application --bind 172.31.14.102:8001`
and not with domain
http://logitech.enterpriselist.com:8001/.
But I also want to run the things with the default port, but when I run
gunicorn configs.wsgi:application --bind 172.31.14.102:80
it says address already in use!
Also with https when I open http://logitech.enterpriselist.com/, it goes to https://logitech.enterpriselist.com/ but it says website have redirect loop so I need help in sorting this.
You haven't got anything to tell nginx it should be proxying requests to gunicorn. In particular, you need a proxy_pass directive and an upstream section.
Also, you don't want to run gunicorn on port 80, since that is what nginx is already bound to. That's what the proxy is for.
The gunicorn deployment docs have an example nginx configuration which works fine.
I followed this post to serve my django project. The project runs well with manage.py runserver and I want to set it up for production. Here are my setting files:
nginx.conf:
upstream django {
server /tmp/vc.sock;
#server 10.9.1.137:8002;
}
server {
listen 8001;
server_name 10.9.1.137;
charset utf-8;
client_max_body_size 25M;
location /media {
alias /home/deploy/vc/media;
}
location /static {
alias /home/deploy/vc/static;
}
location / {
uwsgi_pass django;
include /etc/nginx/uwsgi_params;
}
}
uwsgi.ini:
[uwsgi]
chdir = /home/deploy/vc
wsgi-file = vc/wsgi.py
master = true
processes = 2
#socket = :8002
socket = /tmp/vc.sock
chmod-socket = 666
vacuum = true
If I use TCP port socket (server 10.9.1.137:8002 and socket = :8002), it's going to be fine. However if I comment them out and use Unix sockets(server /tmp/vc.sock and socket = /tmp/vc.sock), the server will return 502 error. How should I fix it?
EDIT
Here's the nginx error log when I run /etc/init.d/nginx restart
nginx: [emerg] invalid host in upstream "/tmp/vc.sock" in /etc/nginx/conf.d/vc.conf:2
nginx: configuration file /etc/nginx/nginx.conf test failed
And this is the warning when I run uwsgi --ini vc/uwsgi.ini:
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Can't I run uWSGI as root?
roberto's comment should be an answer!
the syntax in nginx for unix socket path is wrong, you need to prefix
it with unix:
Check your nginx error log, very probably is telling you it does not have permissions to the socket. Unix sockets honour file system permissions, so nginx must have write privilege over the socket file. Short answer: 664 is not enough, you need 666
Here's a data flow:
http <--> nginx <--> uWSGI <--> python webapp
I guess there's http2uwsgi transfer in nginx, and uwsgi2http in uWSGI.
What if I want to directly call uWSGI to test an API in a webapp?
actually i'm using pyramid. just config [uwsgi] in .ini and run uWSGI. but i want to test if uWSGI hold webapp function normally, the uWSGI socket is not directly reachable by http.
Try using uwsgi_curl
$ pip install uwsgi-tools
$ uwsgi_curl 10.0.0.1:3030 /path
or if you need to do some more requests try uwsgi_proxy from the same package
$ uwsgi_proxy 10.0.0.1:3030
Proxying remote uWSGI server 10.0.0.1:3030 "" to local HTTP server 127.0.0.1:3030...
so you can browse it locally at http://127.0.0.1:3030/.
If your application allows only certain Host header, you can specify host name as well
$ uwsgi_curl 10.0.0.1:3030 host.name/path
$ uwsgi_proxy 10.0.0.1:3030 -n host.name
If application has static files, you can redirect such requests to your front server using -s argument. You can also specify different local port if needed.
From your question I'm assuming, you want to directly run your WSGI-compliant app with uWSGI and open an HTTP-Socket. You can do so by configuring your uwsgi.ini (or whatever the filename is) with
http=127.0.0.1:8080
uwsgi will now open an HTTP-socket that listen on port 8080 for incoming connections from localhost (see documentation: http://uwsgi-docs.readthedocs.org/en/latest/HTTP.html)
Alternatively you can directly start your process from the command-line with the http-parameter:
$ uwsgi --http=127.0.0.1:8080 --module=yourapp:wsgi_entry_point
If you use unix-sockets to configure uwsgi nginx is able to communicate with that socket via the uwsgi-protocol (http://uwsgi-docs.readthedocs.org/en/latest/Protocol.html).
Keep in mind, that if you usually serve static content (css, javascript, images) through nginx you will need to set that up, too, if you run uwsgi directly. But if you only want to test a REST-API this should work out for you.
First, consider those questions:
On which port is uWSGI running?
Is uWSGI running on your or on a remote machine?
If it's running on a remote machine, is the port accessible from your computer? (iptables rules might forbid external access)
If you made sure you have access, you can just call http://hostname:port/path/to/uWSGI for direct API access.
I know this is an old question but I just needed this and found out that this docker+nginx solution works for me the best
cat > /tmp/nginx.conf << EOF
events {}
http {
server {
listen 8000;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:3031;
}
}
}
EOF
docker run -it --network=host --rm --name uswgi-proxy -v /tmp/nginx.conf:/etc/nginx/nginx.conf:ro nginx