So i am making a big social media app but i have a problem which framework to choose flask or express.js i like flask so much but it cant handle too much requests. Express.js can handle about 25k request per second (google). So is there anyway to make flask handle 25k request per second using gunicorn currently i am using this command $ gunicorn -w 4 -b 0.0.0.0:5000 your_project:app but it can only handle 4 request at a time. And uh one more question can flask handle 1Million user at a time. Should i choose express.js because it can handle 25k request
You can use multithreads or gevent to increase gunicorn's concurrency.
Option1 multithreads
eg:
gunicorn -w 4 --threads 100 -b 0.0.0.0:5000 your_project:app
--threads 100 means 100 threads per process.
-w 4 means 4 processes, so -w 4 --threads 100 means 400 requests at a time
Option2 gevent worker
eg:
pip install gevent
gunicorn -w 4 -k gevent --worker-connections 1000 -b 0.0.0.0:5000 your_project:app
-k gevent --worker-connections 1000 means 1000 coroutines per gevent worker process.
-w 4 means 4 processes, so -w 4 -k gevent --worker-connections 1000 means 4000 requests at a time.
For more information, you can refer to my blog post: https://easydevguide.com/posts/gunicorn_concurrency
Related
It is not possible to limit the response time of a request from an application. It is necessary that the service interrupt the connection after 10s.
command=gunicorn -w 3 -k uvicorn.workers.UvicornWorker --timeout 2
--timeout is intended for another as far as I understand
I am trying to deploy a machine learning model (based on tensorflow and keras) as a service using FastAPI and Gunicorn, but I am not able to get enough throughput from the API even after increasing number of Gunicorn workers and threads.
I have tried with the following configs:
1 Worker: gunicorn model:app -k uvicorn.workers.UvicornWorker -b hostname:port
This is giving me a throughput of 15 responses/sec
5 workers: gunicorn model:app -k uvicorn.workers.UvicornWorker --workers=5 -b hostname:port
This is giving me a throughput of 30 responses/sec
30 responses/sec is the maximum throughput I am able to get, while I have to scale it to around 300 responses/sec. I have tried increasing the number of threads too, but that did not result in increase in throughput either.
When I am timing the request-response with single worker: it takes around 80ms for the response to return (Done through Postman)
I am trying to run this on Linux machine with the following details:
OS - CentOS
CPU(s) - 8
Core(s) per socket - 4
Thread(s) per core - 2
Memory - ~65Gig
The system is almost idle when I am trying to run the service (less than 5% CPU usage).
I have a flask application which is running using Gunicorn.
This flask application have an API which takes two hours to complete.
If the same API is called twice after a 30 minutes gap between two , then the process handling the first API call is getting restarted after the second API call.
Example:
Initial process starts with API_1
After 30 mins API_1 is called again , then the process handling previous API_1 call is getting restarted.
Command used to start the Gunicorn server:
nohup gunicorn --bind 0.0.0.0:5000 --workers=8 run:app --timeout 7200 --preload> output.log&
No of core : 8
I am not facing any issue while running flask in development mode.
Any idea why its behaving like this ?
You can use other types of worker. Your current worker type is sync. If you want to send a long request probably you must use a thread worker or async worker.
nohup gunicorn --bind 0.0.0.0:5000 -k gevent run:app
Refer to this document
I tried adding the websockets example project to the datastore project and the websockets work but when a page queries the datastore or tries to put a new entity I get a 502 response. In the logs it shows a critical error on the service worker. If I remove the websocket code the datastore code works as intended. The only difference I can see is the entrypoints for the app samples slightly differ
the websocket sample uses
entrypoint: gunicorn -b :$PORT -k flask_sockets.worker main:app
while the datastore sample uses
entrypoint: gunicorn -b :$PORT main:app
websocket sample https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible/websockets
datastore sample
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible/datastore
The problem appears to be that GRPC (the default transport mechanism of the Cloud Datastore client) is not compatible with gevent. Aside from using a different websockets framework, you can work around the issue by activating grpc's gevent compatibility patch, using the following code:
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()
As a complement of Andrew's answer, you can extend the gunicorn worker class to run gRPC applications.
# gevent_grpc_worker.py
from gunicorn.workers.ggevent import GeventWorker
from grpc.experimental import gevent
class GeventGrpcWorker(GeventWorker):
def patch(self):
super(GeventGrpcWorker, self).patch()
gevent.init_gevent()
self.log.info('patched grpc')
# config.py for gunicorn
import multiprocessing
from gevent_grpc_worker import GeventGrpcWorker
# http://docs.gunicorn.org/en/stable/design.html#how-many-workers
workers = multiprocessing.cpu_count() * 2 + 1
worker_connections = 10000
# Use an asynchronous worker as most of the work is waiting for websites to load
worker_class = '.'.join([GeventGrpcWorker.__module__,
GeventGrpcWorker.__name__])
timeout = 30
Then start your managed application by:
gunicorn -c config.py app:app
As you said it seems there is a problem with the flask_socket.worker, I have test it and it does not work with the datastore client.
I have tried with the Flask-SocketIO framework using the eventlet worker and the datastore queries work fine.
entrypoint: gunicorn -b :$PORT --worker-class eventlet -w 1 main:app
Also you need to add the eventlet module in the requirements.txt file eventlet==0.24.1
The downside of this is that it breaks the compatibility with the websocket code so you need to rewrite this part. Keep in mind that code samples are just intended to show in a few lines how to use the Google Cloud products and copy-paste them without modifying the configuration undelied in the app.yaml is not a good idea.
I have a simple flask script that uses requests to make http requests to third party web service. The way I run the script in gunicorn is
gunicorn abc:APP -b 0.0.0.0:8080 -w 4 -k gevent --timeout 30 --preload
However, after I upgrade the code to python 3.6.2, I can still run the server, but whenever the webserver received a request, it shows
RecursionError: maximum recursion depth exceeded while calling a Python object
on every worker, and the server seems are still running. When I change the running command to
gunicorn abc:APP -b 0.0.0.0:8080 -w 4 --timeout 30 --preload
It all works again. So is there any issue with gunicorn's async worker and requests in python 3.6.2? Is there a way to fix this?
(This question is also asked at https://github.com/benoitc/gunicorn/issues/1559)
This is because of the ssl is imported before monkey patch, gunicorn import the ssl module(config.py in gevent) when loading config，however the monkey patch is called when init the worker or at the top of app.py file which is definitely after import of ssl, this is why we get the warning.
A simple solution is to use a config file for gunicorn.
we could do gevent monkey patch at the beginning of config file, and start gunicorn with the config file, in this way, the monkey patch could be completed before import ssl, and thus avoid this problem.
A config file named gunicorn_config.py could contain lines below:
import gevent.monkey
gevent.monkey.patch_all()
workers = 8
and then we could start gunicorn with
gunicorn --config config.py --worker-class gevent --preload -b 0.0.0.0:5000 app:app
More information could be found here
Please see https://github.com/benoitc/gunicorn/issues/1559. This may possibly be fixed in the next version of gunicorn, but unfortunately you may have to stay with python 3.5.2 if you don't want to break gunicorn.