Develop Reference

gunicorn threads getting killed silently

gunicorn version 19.9.0
Got the following gunicorn config:
accesslog = "access.log"
worker_class = 'sync'
workers = 1
worker_connections = 1000
timeout = 300
graceful_timeout = 300
keepalive = 300
proc_name = 'server'
bind = '0.0.0.0:8080'
name = 'server.py'
preload = True
log_level = "info"
threads = 7
max_requests = 0
backlog = 100
As you can see, the server is configured to run 7 threads.
The server is started with:
gunicorn -c gunicorn_config.py server:app
Here are the number of lines and thread IDs from our log file at the beginning (with the last line being the thread of the main server):
10502 140625414080256
10037 140624842843904
9995 140624859629312
9555 140625430865664
9526 140624851236608
9409 140625405687552
2782 140625422472960
6 140628359804736
So 7 threads are processing the requests. (Already we can see that thread 140625422472960 is processing substantially fewer requests than the other threads.)
But after the lines examined above, thread 140625422472960 just vanishes and the log file only has:
19602 140624859629312
18861 140625405687552
18766 140624851236608
18765 140624842843904
12523 140625414080256
2111 140625430865664
(excluding the main thread here)
From the server logs we could see that the thread received a request and started processing it, but never finished. The client received no response either.
There is no error/warning in the log file, nor in stderr.
And running the app for a little longer, two more threads are gone:
102 140624842843904
102 140624851236608
68 140624859629312
85 140625405687552
How to debug this?

Digging into the stderr logs further, finally found something like this exception stack trace:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
req = six.next(parser)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/parser.py", line 41, in __next__
self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 181, in __init__
super(Request, self).__init__(cfg, unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 230, in parse
self.headers = self.parse_headers(data[:idx])
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 74, in parse_headers
remote_addr = self.unreader.sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
[2018-11-04 17:57:55 +0330] [31] [ERROR] Socket error processing request.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
req = six.next(parser)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/parser.py", line 41, in __next__
self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 181, in __init__
super(Request, self).__init__(cfg, unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 230, in parse
self.headers = self.parse_headers(data[:idx])
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 74, in parse_headers
remote_addr = self.unreader.sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
This is due to this gunicorn bug.
An interim solution until this bug is fixed is to monkey patch gunicorn as done by asantoni.

Celery and RabbitMQ not starting

I installed RabbitMQ, Celery, Flask and Python but when I tried to run celery worker to test. It does not work, these are the error that was pop out in the cmd.
[2019-01-18 09:56:37,443: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\worker\consumer\consumer.py", line 317, in start
blueprint.start(self)
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\bootsteps.py", line 119, in start
step.start(parent)
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\worker\consumer\mingle.py", line 40, in start
self.sync(c)
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\worker\consumer\mingle.py", line 44, in sync
replies = self.send_hello(c)
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\worker\consumer\mingle.py", line 57, in send_hello
replies = inspect.hello(c.hostname, our_revoked._data) or {}
below is my code:
from celery import Celery
app = Celery('test_celery', broker='amqp://myuser:mypassword#localhost/myvhost',backend='rpc://')

[2019-01-18 09:56:37,443: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
As mentioned in the error, there is no broker running. You need to start Rabbitmq before making a connection to it. That is why the consumer is throwing Connection to broker lost as the broker is not running.

How to run pycrate corenet

I want to run the core simulator of https://github.com/P1sec/pycrate.
And I want to run SERVER_ENB only, so configure like below and create a server.
from pycrate_corenet import Server
Server.CorenetServer.SERVER_HNB = {}
Server.CorenetServer.SERVER_ENB['IP'] = '127.0.0.1'
Server.CorenetServer.SERVER_ENB['GTPU'] = '127.0.0.1'
epc = Server.CorenetServer()
But, I got following error.
$ sudo /usr/local/anaconda3/bin/python EPC.py
CorenetServer: loading all ASN.1 and NAS modules, be patient...
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/local/anaconda3/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/anaconda3/lib/python3.6/site-packages/pycrate-0.3-py3.6.egg/pycrate_corenet/Server.py", line 345, in start
self.GTPUd = self.__class__.GTPUd()
File "/usr/local/anaconda3/lib/python3.6/site-packages/pycrate-0.3-py3.6.egg/pycrate_corenet/ServerGTPU.py", line 466, in __init__
sk.bind((gtpip, self.GTP_PORT))
OSError: [Errno 99] Cannot assign requested address
How can I run the server?
Could you give a usage for this pycrate corenet?

this port is probably occupied by something else (including the other instance on this server).
If you're on Linux - you can check if it's already listen with netstat -anp | grep 36412. Then if something listed there you have to kill the app or change server's port in SERVER_ENB structure

I got a solution.
from pycrate.pycrate_corenet import Server, ServerGTPU
Server.CorenetServer.SERVER_HNB = {}
Server.CorenetServer.SERVER_ENB['IP'] = '127.0.0.1'
Server.CorenetServer.SERVER_ENB['GTPU'] = '127.0.0.1'
ServerGTPU.GTPUd.GTP_IF = ['127.0.0.1'] # set GTP IP
epc = Server.CorenetServer()

Deploy readthedocs on production (nginx + gunicorn)

I am trying to deploy to a production server the project Read The Docs (http://docs.readthedocs.io/en/latest/install.html) for internal use at the company I work.
I followed the install steps at the above url, it worked when I run with 'python manage.py 0.0.0.0:8000', but when I tried to deploy with Nginx + Gunicorn + Supervisord, the builds doesn't start, it keep showing 'Triggered version latest (html)'
On the serve I got the error below, but I have no idea what I did wrong.
Is the Read The Docs able to run with Nginx + Gunicorn + Supervisord? Do I have to install or configure celery?
Thanks in advance!
[09/Feb/2018 15:29:59] "GET /api/v2/project/2/ HTTP/1.1" 403 39
[09/Feb/2018 15:29:59] readthedocs.projects.tasks:159[15266]: ERROR An unhandled exception was raised during build setup
Traceback (most recent call last):
File "/webapps/readthedocs/src/readthedocs/projects/tasks.py", line 144, in run
self.project = self.get_project(pk)
File "/webapps/readthedocs/src/readthedocs/projects/tasks.py", line 299, in get_project
project_data = api_v2.project(project_pk).get()
File "/webapps/readthedocs/rtd_env/local/lib/python2.7/site-packages/slumber/__init__.py", line 155, in get
resp = self._request("GET", params=kwargs)
File "/webapps/readthedocs/rtd_env/local/lib/python2.7/site-packages/slumber/__init__.py", line 101, in _request
raise exception_class("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
HttpClientError: Client Error 403: http://localhost:8000/api/v2/project/2/
[09/Feb/2018 15:29:59] celery.app.trace:248[15266]: ERROR Task readthedocs.projects.tasks.update_docs[1cf185cd-57dd-478b-8689-bb795f26543c] raised unexpected: AttributeError("'UpdateDocsTask' object has no attribute 'setup_env'",)
Traceback (most recent call last):
File "/webapps/readthedocs/rtd_env/local/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
R = retval = fun(*args, **kwargs)
File "/webapps/readthedocs/src/readthedocs/projects/tasks.py", line 163, in run
build_id=build_pk,
AttributeError: 'UpdateDocsTask' object has no attribute 'setup_env'

I also ran into same problem everytime I used a port other then 8000. Finally I used port 8000. You don't need to configure celery. I would suggest to check your local settings(readthedocs/settings/local_settings.py) once again. Specifically the PRODUCTION_DOMAIN setting.
Mine looks like this -
PRODUCTION_DOMAIN = "mydomain.com"
SITE_ID = 2 # i have overided it from 1 to 2.
ALLOW_PRIVATE_REPOS = True
SECRET_KEY = "some random secret key"
PUBLIC_API_URL = 'http://{0}'.format(PRODUCTION_DOMAIN)

Deploying aiohttp.web Application using gunicorn and nginx

I am trying to deploy an aiohttp web app, but can't figure out how to get the app to serve over a unix socket, which I think I need in order to get nginx and gunicorn to talk to each other.
Simple example app from aiohttp documentation saved as app.py:
import asyncio
from aiohttp import web
#asyncio.coroutine
def hello(request):
return web.Response(body=b'Hello')
app = web.Application()
app.router.add_route('GET', '/', hello)
if __name__ == "__main__":
loop = asyncio.get_event_loop()
handler = app.make_handler()
f = loop.create_server(handler, '0.0.0.0', 8080)
srv = loop.run_until_complete(f)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
loop.run_until_complete(handler.finish_connections(1.0))
srv.close()
loop.run_until_complete(srv.wait_closed())
loop.run_until_complete(app.finish())
loop.close()
Running this with gunicorn directly works:
gunicorn -k aiohttp.worker.GunicornWebWorker -b 0.0.0.0:8000 app:app
But when I try to bind it instead to a unix socket, I get the following errors.
gunicorn -k aiohttp.worker.GunicornWebWorker -b unix:my_sock.sock app:app
Traceback:
[2015-08-09 12:26:05 -0700] [26898] [INFO] Booting worker with pid: 26898
[2015-08-09 12:26:06 -0700] [26898] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/home/claire/absapp/venv/lib/python3.4/site- packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/home/claire/absapp/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 28, in init_process
super().init_process()
File "/home/claire/absapp/venv/lib/python3.4/site-packages/gunicorn/workers/base.py", line 124, in init_process
self.run()
File "/home/claire/absapp/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 34, in run
self.loop.run_until_complete(self._runner)
File "/usr/lib/python3.4/asyncio/base_events.py", line 268, in run_until_complete
return future.result()
File "/usr/lib/python3.4/asyncio/futures.py", line 277, in result
raise self._exception
File "/usr/lib/python3.4/asyncio/tasks.py", line 236, in _step
result = next(coro)
File "/home/claire/absapp/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 81, in _run
handler = self.make_handler(self.wsgi, *sock.cfg_addr)
TypeError: make_handler() takes 4 positional arguments but 11 were given
[2015-08-09 12:26:06 -0700] [26898] [INFO] Worker exiting (pid: 26898)
I came across something in an aiohttp issue (https://github.com/KeepSafe/aiohttp/issues/136)
that uses socket to create a socket to put as a parameter in the loop.create_server() function, but I just couldn't get anything to work. (I also don't know if the app in his code is the same web.Application object)
Does anybody know how I can make this work? Thanks!

The problem is that GunicornWebWorker doesn't support unix domain sockets. It comes from GunicornWebWorker.make_handler(self, app, host, port), which wants parameters: host and port. Obviously you don't have them if you're using unix socket, but have path to socket instead.
Let's take a look at the beginning of GunicornWebWorker._run():
def _run(self):
for sock in self.sockets:
handler = self.make_handler(self.wsgi, *sock.cfg_addr)
...
In case of -b localhost:8000 sock.cfg_addr is ['localhost', 8000], but for -b unix:my_sock.sock it's just 'my_sock.sock'. This is where error TypeError: make_handler() takes 4 positional arguments but 11 were given comes from. It unpacks string, instead of list.
The quick way to fix it is to subclass GunicornWebWorker and redefine GunicornWebWorker.make_handler() to ignore host and port. They are not used anyway. You can do it like this:
class FixedGunicornWebWorker(worker.GunicornWebWorker):
def make_handler(self, app, *args):
if hasattr(self.cfg, 'debug'):
is_debug = self.cfg.debug
else:
is_debug = self.log.loglevel == logging.DEBUG
return app.make_handler(
logger=self.log,
debug=is_debug,
timeout=self.cfg.timeout,
keep_alive=self.cfg.keepalive,
access_log=self.log.access_log,
access_log_format=self.cfg.access_log_format)
NOTE You'll need to have package with fixed worker in your PYTHONPATH. Otherwise Gunicorn won't be able to locate it. For example if you put fixed worker inside fixed_worker.py file inside the same directory you run gunicorn from, you can use it like:
$ PYTHONPATH="`pwd`:$PYTHONPATH" gunicorn -k fixed_worker.FixedGunicornWebWorker -b unix:my_sock.sock app:app
UPD Also opened issue in aiohttp repository.