Develop Reference

TypeError from pkg_resources when using static assets in Pyramid

Trying to set up static assets on a Pyramid app. I used the following call:
config.add_static_view(name='static', path='toolsofknowledge:static')
The file main.css is stored under toolsofknowledge/static/main.css, as expected. The toolsofknowledge package is installed locally with an editable link, using pip3 install -e ..
Requesting http://localhost:8080/static/main.css with curl causes this exception on the server:
amoe#cslp019129 $ PYRAMID_RELOAD_TEMPLATES=1 pserve3 --reload development.ini
Starting monitor for PID 3796.
Starting server in PID 3796.
Serving on http://localhost:8080
Serving on http://localhost:8080
ERROR:waitress:Exception when serving /static/main.css
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pyramid/tweens.py", line 13, in _error_handler
response = request.invoke_exception_view(exc_info)
File "/usr/lib/python3/dist-packages/pyramid/view.py", line 769, in invoke_exception_view
raise HTTPNotFound
pyramid.httpexceptions.HTTPNotFound: The resource could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/waitress/channel.py", line 336, in service
task.service()
File "/usr/lib/python3/dist-packages/waitress/task.py", line 175, in service
self.execute()
File "/usr/lib/python3/dist-packages/waitress/task.py", line 452, in execute
app_iter = self.channel.server.application(env, start_response)
File "/usr/lib/python3/dist-packages/pyramid/router.py", line 270, in __call__
response = self.execution_policy(environ, self)
File "/usr/lib/python3/dist-packages/pyramid/router.py", line 279, in default_execution_policy
return request.invoke_exception_view(reraise=True)
File "/usr/lib/python3/dist-packages/pyramid/view.py", line 768, in invoke_exception_view
reraise_(*exc_info)
File "/usr/lib/python3/dist-packages/pyramid/compat.py", line 179, in reraise
raise value
File "/usr/lib/python3/dist-packages/pyramid/router.py", line 277, in default_execution_policy
return router.invoke_request(request)
File "/usr/lib/python3/dist-packages/pyramid/router.py", line 249, in invoke_request
response = handle_request(request)
File "/usr/lib/python3/dist-packages/pyramid/tweens.py", line 43, in excview_tween
response = _error_handler(request, exc)
File "/usr/lib/python3/dist-packages/pyramid/tweens.py", line 17, in _error_handler
reraise(*exc_info)
File "/usr/lib/python3/dist-packages/pyramid/compat.py", line 179, in reraise
raise value
File "/usr/lib/python3/dist-packages/pyramid/tweens.py", line 41, in excview_tween
response = handler(request)
File "/usr/lib/python3/dist-packages/pyramid/router.py", line 148, in handle_request
registry, request, context, context_iface, view_name
File "/usr/lib/python3/dist-packages/pyramid/view.py", line 657, in _call_view
response = view_callable(context, request)
File "/usr/lib/python3/dist-packages/pyramid/viewderivers.py", line 401, in viewresult_to_response
result = view(context, request)
File "/usr/lib/python3/dist-packages/pyramid/static.py", line 102, in __call__
if resource_isdir(self.package_name, resource_path):
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1138, in resource_isdir
return get_provider(package_or_requirement).resource_isdir(
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 364, in get_provider
return _find_adapter(_provider_factories, loader)(module)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1392, in __init__
self.module_path = os.path.dirname(getattr(module, '__file__', ''))
File "/usr/lib/python3.7/posixpath.py", line 156, in dirname
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Python 3.7, Pyramid 1.10.2.

Solution: add __init__.py to toolsofknowledge directory.
The problem is quite simple -- the toolsofknowledge package was not a real package, despite being installed, as it was missing an __init__.py. This causes toolsofknowledge.__loader__ to be an instance of NamespaceLoader, when it should be SourceFileLoader. Obviously, pkg_resources cannot resolve resources relative to something that's not a real package. Arguably it should handle this case more smoothly though.

"cannot allocate memory in static TLS block" in a Python Flask application

I've searched online but haven't found anything that fixed my issue. I'm writing a web app in Flask using Python, but after transferring my data to a new computer and recreating the virtual environment, I'm no longer able to use Flask's built-in development web server. I'm getting the traceback below. I can't tell if the issue lies in the SQL or Flask modules, but I've tried reinstalling and compiling them from source to no avail. The application throws the same error even when not in the virtual environment, as well.
127.0.0.1 - - [07/Aug/2020 14:29:49] "GET / HTTP/1.1" 500 -
Traceback (most recent call last):
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/sqlalchemy/util/_collections.py", line 1020, in __call__
return self.registry[key]
KeyError: 139700690765568
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
return self.finalize_request(rv)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask/app.py", line 1969, in finalize_request
response = self.process_response(response)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask/app.py", line 2268, in process_response
self.session_interface.save_session(self, ctx.session, response)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask_session/sessions.py", line 534, in save_session
saved_session = self.sql_session_model.query.filter_by(
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask_sqlalchemy/__init__.py", line 519, in __get__
return type.query_class(mapper, session=self.sa.session())
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/sqlalchemy/orm/scoping.py", line 78, in __call__
return self.registry()
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/sqlalchemy/util/_collections.py", line 1022, in __call__
return self.registry.setdefault(key, self.createfunc())
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3286, in __call__
return self.class_(**local_kw)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask_sqlalchemy/__init__.py", line 138, in __init__
bind = options.pop('bind', None) or db.engine
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask_sqlalchemy/__init__.py", line 937, in engine
return self.get_engine()
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask_sqlalchemy/__init__.py", line 956, in get_engine
return connector.get_engine()
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask_sqlalchemy/__init__.py", line 561, in get_engine
self._engine = rv = self._sa.create_engine(sa_url, options)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/flask_sqlalchemy/__init__.py", line 966, in create_engine
return sqlalchemy.create_engine(sa_url, **engine_opts)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/sqlalchemy/engine/__init__.py", line 500, in create_engine
return strategy.create(*args, **kwargs)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 118, in dbapi
return __import__("MySQLdb")
File "/home/jhonig/ttx/venv/lib/python3.8/site-packages/MySQLdb/__init__.py", line 18, in <module>
from . import _mysql
ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: cannot allocate memory in static TLS block

For anyone who might come across this issue, there was an issue with the libmysqlclient21 package, which was fixed in the 8.0.21 version:
Bug: https://bugs.launchpad.net/ubuntu/+source/opencv/+bug/1890170
Fix: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1889851

Upgrading my system using sudo apt upgrade seemed to fix the issue, though I don't know how. It appears that some MySQL packages were updated though, so maybe it was a bug?
For posterity, here is a list of packages that were upgraded:
The following packages will be upgraded:
libmysqlclient-dev libmysqlclient21 libmysqlclient21:i386 linux-generic
linux-headers-5.4.0-7634 linux-headers-5.4.0-7634-generic linux-headers-generic
linux-image-5.4.0-7634-generic linux-image-generic linux-libc-dev
linux-modules-5.4.0-7634-generic linux-modules-extra-5.4.0-7634-generic
linux-system76 openjdk-8-jre-headless pop-shell
15 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Graphite or Django seems to be missing some templates and I don't know how to supply them

I'm following this tutorial: https://www.infoq.com/articles/graphite-intro
Everything works until the end when I execute this line:
tail -f /opt/graphite/storage/log/webapp/process.log
I see on the terminal:
django-admin.py runserver --pythonpath /opt/graphite/webapp --settings graphite.settings 0.0.0.0:8085
but then I navigate with my browser to my_ip:8085 where my_ip clearly is my otherwise working ip address.
I then get this error in the browser:
A server error occurred. Please contact the administrator.
And in my terminal I get this error:
[19/Apr/2017 03:48:03] "GET / HTTP/1.1" 500 59
Traceback (most recent call last):
File "/usr/lib64/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/usr/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 241, in __call__
response = self.get_response(request)
File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 153, in get_response
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 228, in handle_uncaught_exception
return callback(request, **param_dict)
File "/opt/graphite/webapp/graphite/views.py", line 7, in server_error
template = loader.get_template(template_name)
File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 145, in get_template
template, origin = find_template(template_name)
File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 138, in find_template
raise TemplateDoesNotExist(name)
TemplateDoesNotExist: 500.html
How can I resolve these errors?

Flask + gevent + uwsgi - greenlet spawn cause 502 Bad Gateway

I have an issue since I added gevent to my flask project: the page spawning greenlets always timeout / bad gateway.
The part of code causing this issue is this: https://github.com/Kyria/LazyBlacksmith/blob/master/lazyblacksmith/views/ajax.py#L145-L221
Here what i've tested:
Running my flask app with python app.py : works fine with or without gevent/greenlets.
Running behind uwsgi without gevent/greenlet : works fine.
Running behind uwsgi with gevent/greenlet : fails.
So, after searching for a while, reading uwsgi doc, i tried to add some stuff to my uwsgi conf :
loop = async
async = 10
greenlet = true
ugreen = true
After restarting my uwsgi, i didn't have to wait for error, the Bad Gateway was really quick, but with an error in the log this time:
Traceback (most recent call last):
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "./lazyblacksmith/views/ajax.py", line 152, in get_price_and_tax
crest = get_crest()
File "./lazyblacksmith/utils/crestutils.py", line 17, in get_crest
crest()
File "./lazyblacksmith/utils/pycrest/eve.py", line 212, in __call__
self._data = APIObject(self.get(self._endpoint), self)
File "./lazyblacksmith/utils/pycrest/eve.py", line 156, in get
res = self._session.get(resource, params=prms, headers=headers)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
self._validate_conn(conn)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 782, in _validate_conn
conn.connect()
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 215, in connect
conn = self._new_conn()
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 135, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/connection.py", line 66, in create_connection
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/gevent/_socketcommon.py", line 268, in getaddrinfo
return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/gevent/resolver_thread.py", line 64, in getaddrinfo
return self.pool.apply(_socket.getaddrinfo, args, kwargs)
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/gevent/pool.py", line 300, in apply
return self.spawn(func, *args, **kwds).get()
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/gevent/event.py", line 284, in get
result = self.hub.switch()
File "/home/lazyblacksmith/LazyBlacksmith/env/local/lib/python2.7/site-packages/gevent/hub.py", line 592, in switch
return greenlet.switch(self)
MemoryError
I also tried to add the following in my uwsgi app conf :
gevent = 10
gevent-monkey-patch = true
limit-as = 512
And i tried to increase the timeout in nginx uwsgi_read_timeout 300, but it didn't change anything...
My uwsgi conf is the following:
# lazyblacksmith.ini file
[uwsgi]
chdir = /home/lazyblacksmith/LazyBlacksmith
home = /home/lazyblacksmith/LazyBlacksmith/env/
callable = app
wsgi-file = app.py
master = true
processes = 10
socket = /home/lazyblacksmith/LazyBlacksmith/env/lazyblacksmith.socket
chmod-socket = 664
vacuum = true
uid = lazyblacksmith
gid = www-data
My nginx conf :
upstream lbflask {
server unix:/home/lazyblacksmith/LazyBlacksmith/env/lazyblacksmith.socket;
}
# configuration of the server
server {
# the port your site will be served on
listen 80;
# the domain name it will serve for
server_name site.com; #
charset utf-8;
location /static {
alias /home/lazyblacksmith/LazyBlacksmith/lazyblacksmith/static;
include /etc/nginx/mime.types;
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass lbflask;
include /etc/nginx/uwsgi_params; # the uwsgi_params file you installed
}
}
I don't find anything relevant anymore :/ I think i'll end up moving my gevent stuff in tasks (with celery), but i still want to understand what is my problem...
For information:
Debian 8
uwsgi 2.0.7-debian
nginx/1.6.2
python 2.7.9
gevent 1.1rc2
greenlet 0.4.9
Thank you !

Error 111 after following Werkzeug tutorial "shortly"

I followed the tutorial for Werkzeug "Shortly" here
And I get this error message after submitting a valid url.
Traceback (most recent call last)
File "/home/sadik/NLM/shortly/shortly.py", line 87, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/Werkzeug-0.9.4-py2.7.egg/werkzeug/wsgi.py", line 579, in __call__
return self.app(environ, start_response)
File "/home/sadik/NLM/shortly/shortly.py", line 83, in wsgi_app
response = self.dispatch_request(request)
File "/home/sadik/NLM/shortly/shortly.py", line 33, in dispatch_request
return getattr(self, 'on_' + endpoint)(request, **values)
File "/home/sadik/NLM/shortly/shortly.py", line 45, in on_new_url
short_id = self.insert_url(url)
File "/home/sadik/NLM/shortly/shortly.py", line 72, in insert_url
short_id = self.redis.get('reverse-url:' + url)
File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/client.py", line 705, in get
return self.execute_command('GET', name)
File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/client.py", line 464, in execute_command
connection.send_command(*args)
File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/connection.py", line 334, in send_command
self.send_packed_command(self.pack_command(*args))
File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/connection.py", line 316, in send_packed_command
self.connect()
File "/usr/local/lib/python2.7/dist-packages/redis-2.9.1-py2.7.egg/redis/connection.py", line 253, in connect
raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting localhost:6379. Connection refused.
The error message indicates that there is something wrong with localhost:6379
The relevant part of code is here:
def create_app(redis_host='localhost', redis_port=6379, with_static=True):
app = Shortly({
'redis_host': redis_host,
'redis_port': redis_port
})
if with_static:
app.wsgi_app = SharedDataMiddleware(app.wsgi_app, {
'/static': os.path.join(os.path.dirname(__file__), 'static')
})
return app
if __name__ == '__main__':
from werkzeug.serving import run_simple
app = create_app()
run_simple('127.0.0.1', 5000, app, use_debugger=True, use_reloader=True)
That means of course that the server is running on localhost:5000. So why is there another port number in the create_app function? That confuses me a bit.

I'm not familiar with shortly or werkzeug but it looks like you're missing the redis server, install one using your favourite package manager and try again.