Develop Reference

SynchronousOnlyOperation error when upgrading from python 3.6 to python 3.7 using django channels

I've been attempting to upgrade python from 3.6 to 3.7 for our Django & Django channels application. With that change, Django throws a SynchronousOnlyOperation anytime any HTTP request is made (even if it has nothing to do with WebSockets). My guess is that somehow the python upgrade has made the Django check more strict.
I believe that Django channels are serving both the HTTP requests and WebSocket requests so it expects all code to be async compliant.
How do I get Django channels runserver to run the wsgi app synchronously, while the channel consumers asynchronously?
# project/asgi.py
application = ProtocolTypeRouter({"http": get_asgi_application(), "websocket": routing})
# project/wsgi.py
application = get_wsgi_application()
Stacktrace:
It's clear to me that the auth middleware that is running for a normal wsgi view is not async compliant, how can I get it to run in a sync environment?
ERROR Internal Server Error: /a/api
Traceback (most recent call last):
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
return self._session_cache
AttributeError: 'SessionStore' object has no attribute '_session_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/___/___/___/___/apps/core/middleware.py", line 60, in __call__
request.is_user_verified = request.user.is_verified()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/utils/functional.py", line 240, in inner
self._setup()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/utils/functional.py", line 376, in _setup
self._wrapped = self._setupfunc()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django_otp/middleware.py", line 38, in _verify_user
user.otp_device = None
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/utils/functional.py", line 270, in __setattr__
self._setup()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/utils/functional.py", line 376, in _setup
self._wrapped = self._setupfunc()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/auth/middleware.py", line 24, in <lambda>
request.user = SimpleLazyObject(lambda: get_user(request))
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/auth/middleware.py", line 12, in get_user
request._cached_user = auth.get_user(request)
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/auth/__init__.py", line 174, in get_user
user_id = _get_user_session_key(request)
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/auth/__init__.py", line 58, in _get_user_session_key
return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/sessions/backends/base.py", line 65, in __getitem__
return self._session[key]
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
self._session_cache = self.load()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
s = self._get_session_from_db()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 34, in _get_session_from_db
expire_date__gt=timezone.now()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/db/models/query.py", line 425, in get
num = len(clone)
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/db/models/query.py", line 269, in __len__
self._fetch_all()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/db/models/query.py", line 1303, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/db/models/query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql
cursor = self.connection.cursor()
File "/___/___/.pyenv/versions/pd37/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.
Versions
Django==3.1.1
channels==3.0.2
channels-redis==3.2.0

So a summary of what I've tried:
First, I have a channels app that had no database at all, no contrib apps except staticfiles.
I added database, DRF and one app with a single model and list/detail views
Enabled the sync http consumer:
# dwtools.routing
from channels.routing import ProtocolTypeRouter, URLRouter
from django.core.asgi import get_asgi_application
import romanize.routing
application = ProtocolTypeRouter(
{
"websocket": URLRouter(romanize.routing.websocket_urlpatterns),
"http": get_asgi_application(),
},
)
settings:
INSTALLED_APPS = [
"channels",
"romanize.apps.RomanizeConfig",
"django.contrib.staticfiles",
"rest_framework",
"agencies.apps.AgenciesConfig",
]
WSGI_APPLICATION = "dwtools.wsgi.application"
ASGI_APPLICATION = "dwtools.routing.application"
I can pull up DRF's standard views, create entries etc.
Then added a simple middleware that accesses the database (gets a count of the number of Agencies and puts it in request). Not even using MiddlewareMixin, just barebones init can call protocol.
System check identified no issues (0 silenced).
December 07, 2020 - 14:38:56
Django version 3.1.4, using settings 'dwtools.settings'
Starting ASGI/Channels version 3.0.2 development server at http://127.0.0.1:3401/
# ^^^^^^^^ ^^^^^^^^^^^ ( Channels runserver)
Quit the server with CONTROL-C.
HTTP GET /agencies/ 200 [0.05, 127.0.0.1:60460]
The stack uses local nginx to route to port 3401 for /ws (websocket) and /agencies (http). Perhaps you spot something...

Finally got to the bottom of this. For us, we were using gevent and had this line in our manage.py
# manage.py
import gevent
gevent.patch_all()
For whatever reason, that does not play nicely with python 3.7+, Django channels, and Django (where python 3.6, Django channels, and Django worked fine).
We were lucky and were able to remove this from our codebase; we did not figure out how to fix the gevent patch.

Deploy readthedocs on production (nginx + gunicorn)

I am trying to deploy to a production server the project Read The Docs (http://docs.readthedocs.io/en/latest/install.html) for internal use at the company I work.
I followed the install steps at the above url, it worked when I run with 'python manage.py 0.0.0.0:8000', but when I tried to deploy with Nginx + Gunicorn + Supervisord, the builds doesn't start, it keep showing 'Triggered version latest (html)'
On the serve I got the error below, but I have no idea what I did wrong.
Is the Read The Docs able to run with Nginx + Gunicorn + Supervisord? Do I have to install or configure celery?
Thanks in advance!
[09/Feb/2018 15:29:59] "GET /api/v2/project/2/ HTTP/1.1" 403 39
[09/Feb/2018 15:29:59] readthedocs.projects.tasks:159[15266]: ERROR An unhandled exception was raised during build setup
Traceback (most recent call last):
File "/webapps/readthedocs/src/readthedocs/projects/tasks.py", line 144, in run
self.project = self.get_project(pk)
File "/webapps/readthedocs/src/readthedocs/projects/tasks.py", line 299, in get_project
project_data = api_v2.project(project_pk).get()
File "/webapps/readthedocs/rtd_env/local/lib/python2.7/site-packages/slumber/__init__.py", line 155, in get
resp = self._request("GET", params=kwargs)
File "/webapps/readthedocs/rtd_env/local/lib/python2.7/site-packages/slumber/__init__.py", line 101, in _request
raise exception_class("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
HttpClientError: Client Error 403: http://localhost:8000/api/v2/project/2/
[09/Feb/2018 15:29:59] celery.app.trace:248[15266]: ERROR Task readthedocs.projects.tasks.update_docs[1cf185cd-57dd-478b-8689-bb795f26543c] raised unexpected: AttributeError("'UpdateDocsTask' object has no attribute 'setup_env'",)
Traceback (most recent call last):
File "/webapps/readthedocs/rtd_env/local/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
R = retval = fun(*args, **kwargs)
File "/webapps/readthedocs/src/readthedocs/projects/tasks.py", line 163, in run
build_id=build_pk,
AttributeError: 'UpdateDocsTask' object has no attribute 'setup_env'

I also ran into same problem everytime I used a port other then 8000. Finally I used port 8000. You don't need to configure celery. I would suggest to check your local settings(readthedocs/settings/local_settings.py) once again. Specifically the PRODUCTION_DOMAIN setting.
Mine looks like this -
PRODUCTION_DOMAIN = "mydomain.com"
SITE_ID = 2 # i have overided it from 1 to 2.
ALLOW_PRIVATE_REPOS = True
SECRET_KEY = "some random secret key"
PUBLIC_API_URL = 'http://{0}'.format(PRODUCTION_DOMAIN)

Appengine Python DevServer Module Background Thread 500 Error

I'm on version 1.9.9 of the SDK and I'm having issues with the devserver. I have a manually scaled module with 1 instance. I created a webapp2.RequestHandler for /_ah/start. In that handler I start a background thread. When I run my app in the devserver, the _ah/start handler returns a 200, but /_ah/background will randomly return 500 errors for a while. After sometime (usually a minute or two, but sometimes more), the 500 errors stop, but will randomly occur again every few hours. It also seems that everytime I open a new browser tab (Chrome), I get the same error. Anyone know what could be causing this?
Here is the RequestHandler for /_ah/start:
class StartupHandler(webapp2.RequestHandler):
def get(self):
runtime.set_shutdown_hook(shutdown_hook)
global foo
if foo is None:
foo = Foo()
background_thread.start_new_background_thread(do_foo, [])
self.response.http_status_message(200)
Here is the 500 error:
ERROR 2014-08-18 07:39:36,256 module.py:717] Request to '/_ah/background' failed
Traceback (most recent call last):
File "\appengine\tools\devappserver2\module.py", line 694, in _handle_request
environ, wrapped_start_response)
File "\appengine\tools\devappserver2\request_rewriter.py", line 311, in _rewriter_middleware
response_body = iter(application(environ, wrapped_start_response))
File "\appengine\tools\devappserver2\module.py", line 1672, in _handle_script_request
request_type)
File "\appengine\tools\devappserver2\module.py", line 1624, in _handle_instance_request
request_id, request_type)
File "\appengine\tools\devappserver2\instance.py", line 382, in handle
request_type))
File "\appengine\tools\devappserver2\http_proxy.py", line 190, in handle
response = connection.getresponse()
File "E:\Programing\Python27\lib\httplib.py", line 1030, in getresponse
response.begin()
File "E:\Programing\Python27\lib\httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "E:\Programing\Python27\lib\httplib.py", line 365, in _read_status
line = self.fp.readline()
File "E:\Programing\Python27\lib\socket.py", line 430, in readline
data = recv(1)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
INFO 2014-08-18 07:39:36,257 module.py:1890] Waiting for instances to restart
INFO 2014-08-18 07:39:36,262 module.py:642] lease: "GET /_ah/background HTTP/1.1" 500 -

Well this might be not the answer , but how long will it take to complete a specific task to assign to a backend? Seems like an issue with concurrency

Looks like the issue (as far as I can currently tell) is that I'm using PyCharm, which synchronizes the project's files when its window is entered or exited. This rewrites the project files even if there are no changes, which causes the devserver to restart all instances, leading to the 500 errors.
More info on PyCharm Synchronization
Link to issue at PyCharm

Django 1.5 with psycopg2 throw random errors

I faced with some strange problem. When I run single worker for my django application? like this:
python manage.py run_gunicorn -w 1
python manage.py runserver
All requests served correctly, but if start multiple workers like:
python manage.py run_gunicorn -w 10
Django respond with error to (about) half of my requests:
2013-06-13 18:02:39 [10205] [ERROR] Error handling request
Traceback (most recent call last):
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle_request
for item in respiter:
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/raven/middleware.py", line 27, in __call__
iterable = self.application(environ, start_response)
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 255, in __call__
response = self.get_response(request)
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 177, in get_response
signals.got_request_exception.send(sender=self.__class__, request=request)
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 170, in send
response = receiver(signal=self, sender=sender, **named)
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/django/db/__init__.py", line 68, in _rollback_on_exception
transaction.rollback_unless_managed(using=conn)
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/django/db/transaction.py", line 143, in rollback_unless_managed
connection.rollback_unless_managed()
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 232, in rollback_unless_managed
self._rollback()
File "/home/mezhenin/venv/karma/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 59, in _rollback
return self.connection.rollback()
OperationalError: no connection to the server
I tried sqlite3 as DB backend for Django and it works fine. Do someone have the same problems with psycopg2?
Django==1.5
gunicorn==0.17.2
psycopg2==2.5

Sorry for disturbing. Problem was not in psycopg/Django/gunicorn. I use Nose for testing purposes:
==== settings.py ====
INSTALLED_APPS = INSTALLED_APPS + ('django_nose',)
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
NOSE_ARGS = ['--with-doctest']
when removed this lines, random errors disappear.

dev-server HTTP Error 403: Forbidden

After updating from 1.7.5 (where everything worked fine) I'm getting a HTTP Error 403: Forbidden when trying to open any sites via localhost. Strange thing is I have pretty much the same setup at home as here at work and everything works there... Might be an issue with proxy server we're using at work, since that's the only difference I can think of? Here's the error log I'm getting, so if anyone knows what's going on please help (;
Traceback (most recent call last):
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "U:\Dev\GAE\google\appengine\tools\devappserver2\wsgi_server.py", line 246, in __call__
return app(environ, start_response)
File "U:\Dev\GAE\google\appengine\tools\devappserver2\request_rewriter.py", line 311, in _rewriter_middleware
response_body = iter(application(environ, wrapped_start_response))
File "U:\Dev\GAE\google\appengine\tools\devappserver2\python\request_handler.py", line 89, in __call__
self._flush_logs(response.get('logs', []))
File "U:\Dev\GAE\google\appengine\tools\devappserver2\python\request_handler.py", line 220, in _flush_logs
apiproxy_stub_map.MakeSyncCall('logservice', 'Flush', request, response)
File "U:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 94, in MakeSyncCall
return stubmap.MakeSyncCall(service, call, request, response)
File "U:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 320, in MakeSyncCall
rpc.CheckSuccess()
File "U:\Dev\GAE\google\appengine\api\apiproxy_rpc.py", line 156, in _WaitImpl
self.request, self.response)
File "U:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 200, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "U:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 226, in _MakeRealSyncCall
encoded_response = self._server.Send(self._path, encoded_request)
File "U:\Dev\GAE\google\appengine\tools\appengine_rpc.py", line 393, in Send
f = self.opener.open(req)
File "U:\Dev\Python\lib\urllib2.py", line 410, in open
response = meth(req, response)
File "U:\Dev\Python\lib\urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "U:\Dev\Python\lib\urllib2.py", line 448, in error
return self._call_chain(*args)
File "U:\Dev\Python\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "U:\Dev\Python\lib\urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
INFO 2013-04-19 12:28:52,576 server.py:561] default: "GET / HTTP/1.1" 500 -
INFO 2013-04-19 12:28:52,619 server.py:561] default: "GET /favicon.ico HTTP/1.1" 304 -
Also, the launcher throws an error when closing:
Traceback (most recent call last):
File "launcher\mainframe.pyc", line 327, in OnStop
File "launcher\taskcontroller.pyc", line 167, in Stop
File "launcher\dev_appserver_task_thread.pyc", line 82, in stop
File "launcher\taskthread.pyc", line 107, in stop
File "launcher\platform.pyc", line 397, in KillProcess
pywintypes.error: (5, 'TerminateProcess', 'Access is denied.')

I had this very same issue with my MacOSX when using a proxy server using Google App Engine Launcher 1.8.6. Apparently there's an issue with "proxy_bypass" on "urllib2.py".
There are two possible solutions:
Downgrade to 1.7.5, but, who wants to downgrade?
Edit "[GAE Instalattion path]/google/appengine/tools/appengine_rpc.py" and look for the line that says
opener.add_handler(fancy_urllib.FancyProxyHandler())
In my computer it was line 578, and then put a hash (#) at the beginning of the line, like this:
`#opener.add_handler(fancy_urllib.FancyProxyHandler())`
Save the file, stop and then restart your application. Now dev_appserver.py shouldn't try to use any proxy server at all.
If your application uses any external resources like a SOAP Webservice or something like that and you can't reach the server without the proxy server, then you'll have to downgrade. Please keep in mind that external javascript files (like facebook SDK or similar) are loaded from your browser, not from your application.
Since I'm not using any external REST or SOAP services it worked for me!
Hopefully it will work for you as well.

Try either:
-Accessing it through a different proxy. I.E a . proxy within a proxy
-Accessing it through your local IP i.e 192.168.1.1

I faced the same issue with version 1.9.5. Seems that the API proxy is sending some RPCs to the proxy server, which are then being rejected with HTTP 403 (since proxy servers are generally configured to reject connection attempts to arbitrary ports). In my case I was using the urlfetch module in my app to access external web pages, so disabling the proxy server was not a choice for me.
This is how I worked around the issue some time back (most probably it was based on comments found under this issue, but I cannot remember the exact sources).
NOTE:
For this approach to work, you'll have to know the hostname/IP address and default port of your proxy server, and change them appropriately in the code if you happen to connect to a different proxy server.
When you are not behind the proxy server, you will have to revert the applied changes in order to return to a working state (if you want internet access inside your app).
Here it goes:
Disable proxy settings for the Python (Google App Engine Launcher) environment in some way. (In my case it was easy since I was launching the dev_appserver.py from a Terminal shell (on Linux), and the unset http_proxy and unset https_proxy commands did the trick.)
Edit {App Engine SDK root}/google/appengine/api/urlfetch_stub.py. Find the code block
if _CONNECTION_SUPPORTS_TIMEOUT:
connection = connection_class(host, timeout=deadline)
else:
connection = connection_class(host)
(lines 376-379 in my case) and replace it with:
if _CONNECTION_SUPPORTS_TIMEOUT:
if host[:9] == 'localhost' or host[:9] == '127.0.0.1':
connection = connection_class(host, timeout=deadline)
else:
connection = connection_class('your_proxy_host_goes_here', your_proxy_port_number_goes_here, timeout=deadline)
else:
if host[:9] == 'localhost' or host[:9] == '127.0.0.1':
connection = connection_class(host)
else:
connection = connection_class('your_proxy_host_goes_here', your_proxy_port_number_goes_here)
replacing the placeholders your_proxy_host_goes_here and your_proxy_port_number_goes_here with appropriate values.
(I believe this code can be written more elegantly, though... any Python geeks out there? :) )
In my case, I also had to delete the existing compiled file urlfetch_stub.pyc (located in the same directory as urlfetch_stub.py) because the SDK didn't seem to pick up the changes until I did so.
Now you can use dev_appserver to launch your app, and use urlfetch-backed services within the app, free from HTTP 403 errors.