I am using sanic/sanic-ext 22.9.0 , I need to load a static open api yaml or json file rather than autogenerate so that when I access it as <url>/docs, the static yaml file is loaded and UI display either Swagger or redoc version of UI with the API definition.
Any suggestion on how to achieve this.
You have a few options.
OPTION 1
Load your custom OAS into Sanic, and server that.
#app.before_server_start
async def load_oas(app: Sanic):
custom_oas_dict = load_spec_from_yaml()
app.ext.openapi.raw(custom_oas_dict)
OPTION 2
Turn off OAS and roll your own solution, including swagger/redoc, etc
app.config.OAS = False
OPTION 3
Serve your custom OAS, and overwrite the HTML to point to your custom file.
app.config.OAS_PATH_TO_REDOC_HTML = "/path/to/custom/redoc.html"
app.config.OAS_PATH_TO_SWAGGER_HTML = "/path/to/custom/swagger.html"
app.static("/custom/oas.json", "/path/to/custom/oas.json")
One of the libraries my project requires that a folder with the CSS files that were in the application root called "themes". web.py by default, uses the folder "static" to return the static file and just rename her... not One of the solutions I found online was the following
in urls it is necessary to add the line
'/(?:img|js|css)/.*', 'app.controllers.public.public',
in app.controllers.public
require nex code
class public:
def GET(self):
public_dir = 'themes'
try:
file_name = web.ctx.path.split('/')[-1]
web.header('Content-type', mime_type(file_name))
return open(public_dir + web.ctx.path, 'rb').read()
except IOError:
raise web.notfound()
def mime_type(filename):
return mimetypes.guess_type(filename)[0] or 'application/octet-stream'
but this solution does not work and files are still picked up from static...
is there a simple and clear solution to the problem? maybe we should change the name of the folder inside the web.py?
There's no simple way to change web.py's use of /static/, but there is a really easy way to add one of your own, with no need to add anything to your list of urls.
Look at web.py's code and you'll find web.httpserver.StaticMiddleware is where this is defined. Your job, create another WSGI middleware, with the new prefix. Then, because this is WSGI middleware, add your new class to the run chain.
from web.httpserver import StaticMiddleware
if __name__ == '__main__':
app = web.application(urls, globals())
app.run(lambda app: StaticMiddleware(app, '/themes/')
If that was too terse for you, consider it's the same as explicitly creating a new subclass and passing that subclass to app.run():
from web.httpserver import StaticMiddleware
class MyStaticMiddleware(StaticMiddleware):
def __init__(self, app, prefix='/themes/'):
StaticMiddleware.__init__(self, app, prefix)
if __name__ == '__main__':
app = web.application(urls, globals())
app.run(MyStaticMiddleware)
Note that '/static/' will still work, loading files from the /static/ subdirectory: All you've done is added another processor, which does the same thing, but from the '/themes/' subdirectory.
I want to add set of css files to head. For instance,
python file:
modules.py
class SomeModule(tornado.web.UIModule):
def css_files(self):
return [self.handler.static_url('css/modules/some-module.css'),]
def render(self, some_data=None):
result = ''
if some_data is not None:
"""to do something with data"""
return result
server.py
app = Application(
...
ui_module=modules
...
)
template file:
...
{% module SomeModule(some_data=put_data_here) %}
As the result, I see only data that were returned from render. But css files weren't set between head tags.
What's the result of the template file depends on how you generate the template file. As far as I can see, you didn't include the code for that in your question.
If you want to use css_files() or similar, you need to use self.render_string() in order to give Tornado a chance to insert the CSS tags in the proper places.
For an example of how to use tornado.web.UIModule, see this Slideshare presentation
I'm working on some universal solution for problem with static files and updates in it.
Example: let's say there was site with /static/styles.css file - and site was used for a long time - so a lot of visitors cached this file in browser
Now we doing changes in this css file, and update on server, but some users still have old version (despite modification date returned by server)
The obvious solution is to add some version to file /static/styles.css?v=1.1 but in this case developer must track changes in this file and manually increase version
A second solution is to count the md5 hash of the file and add it to the url /static/styels.css/?v={mdp5hashvalue} which looks much better, but md5 should be calculated automatically somehow.
they possible way I see it - create some template tag like this
{% static_file "style.css" %}
which will render
<link src="/static/style.css?v=md5hash">
BUT, I do not want this tag to calculate md5 on every page load, and I do not want to store hash in django-cache, because then we will have to clear after updating file...
any thoughts ?
Django 1.4 now includes CachedStaticFilesStorage which does exactly what you need (well... almost).
Since Django 2.2 ManifestStaticFilesStorage should be used instead of CachedStaticFilesStorage.
You use it with the manage.py collectstatic task. All static files are collected from your applications, as usual, but this storage manager also creates a copy of each file with the MD5 hash appended to the name. So for example, say you have a css/styles.css file, it will also create something like css/styles.55e7cbb9ba48.css.
Of course, as you mentioned, the problem is that you don't want your views and templates calculating the MD5 hash all the time to find out the appropriate URLs to generate. The solution is caching. Ok, you asked for a solution without caching, I'm sorry, that's why I said almost. But there's no reason to reject caching, really. CachedStaticFilesStorage uses a specific cache named staticfiles. By default, it will use your existing cache system, and voilà! But if you don't want it to use your regular cache, perhaps because it's a distributed memcache and you want to avoid the overhead of network queries just to get static file names, then you can setup a specific RAM cache just for staticfiles. It's easier than it sounds: check out this excellent blog post. Here's what it would look like:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
'LOCATION': '127.0.0.1:11211',
},
'staticfiles': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'staticfiles-filehashes'
}
}
I would suggest using something like django-compressor. In addition to automatically handling this type of stuff for you, it will also automatically combine and minify your files for fast page load.
Even if you don't end up using it in entirety, you can inspect their code for guidance in setting up something similar. It's been better vetted than anything you'll ever get from a simple StackOverflow answer.
I use my own templatetag which add file modification date to url: https://bitbucket.org/ad3w/django-sstatic
Is reinventing the wheel and creating own implementation that bad? Furthermore I would like low level code (nginx for example) to serve my staticfiles in production instead of python application, even with backend. And one more thing: I'd like links stay the same after recalculation, so browser fetches only new files. So here's mine point of view:
template.html:
{% load md5url %}
<script src="{% md5url "example.js" %}"/>
out html:
static/example.js?v=5e52bfd3
settings.py:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(PROJECT_DIR, 'static')
appname/templatetags/md5url.py:
import hashlib
import threading
from os import path
from django import template
from django.conf import settings
register = template.Library()
class UrlCache(object):
_md5_sum = {}
_lock = threading.Lock()
#classmethod
def get_md5(cls, file):
try:
return cls._md5_sum[file]
except KeyError:
with cls._lock:
try:
md5 = cls.calc_md5(path.join(settings.STATIC_ROOT, file))[:8]
value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
except IsADirectoryError:
value = settings.STATIC_URL + file
cls._md5_sum[file] = value
return value
#classmethod
def calc_md5(cls, file_path):
with open(file_path, 'rb') as fh:
m = hashlib.md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
#register.simple_tag
def md5url(model_object):
return UrlCache.get_md5(model_object)
Note, to apply changes an uwsgi application (to be specific a process) should be restarted.
Django 1.7 added ManifestStaticFilesStorage, a better alternative to CachedStaticFilesStorage that doesn't use the cache system and solves the problem of the hash being computed at runtime.
Here is an excerpt from the documentation:
CachedStaticFilesStorage isn’t recommended – in almost all cases ManifestStaticFilesStorage is a better choice. There are several performance penalties when using CachedStaticFilesStorage since a cache miss requires hashing files at runtime. Remote file storage require several round-trips to hash a file on a cache miss, as several file accesses are required to ensure that the file hash is correct in the case of nested file paths.
To use it, simply add the following line to settings.py:
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
And then, run python manage.py collectstatic; it will append the MD5 to the name of each static file.
How about you always have a URL Parameter in your URL with a version and whenever you have a major release you change the version in your URL Parameter. Even in the DNS. So if www.yourwebsite.com loads up www.yourwebsite.com/index.html?version=1.0 then after the major release the browser should load www.yourwebsite.com/index.html?version=2.0
I guess this is similar to your solution 1. Instead of tracking files can you track whole directories? For example ratehr than /static/style/css?v=2.0 can you do /static-2/style/css or to make it even granular /static/style/cssv2/.
There is an update for #deathangel908 code. Now it works well with S3 storage also (and with any other storage I think). The difference is using of static file storage for getting file content. Original doesn't work on S3.
appname/templatetags/md5url.py:
import hashlib
import threading
from django import template
from django.conf import settings
from django.contrib.staticfiles.storage import staticfiles_storage
register = template.Library()
class UrlCache(object):
_md5_sum = {}
_lock = threading.Lock()
#classmethod
def get_md5(cls, file):
try:
return cls._md5_sum[file]
except KeyError:
with cls._lock:
try:
md5 = cls.calc_md5(file)[:8]
value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
except OSError:
value = settings.STATIC_URL + file
cls._md5_sum[file] = value
return value
#classmethod
def calc_md5(cls, file_path):
with staticfiles_storage.open(file_path, 'rb') as fh:
m = hashlib.md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
#register.simple_tag
def md5url(model_object):
return UrlCache.get_md5(model_object)
The major advantage of this solution: you dont have to modify anything in the templates.
This will add the build version into the STATIC_URL, and then the webserver will remove it with a Rewrite rule.
settings.py
# build version, it's increased with each build
VERSION_STAMP = __versionstr__.replace(".", "")
# rewrite static url to contain the number
STATIC_URL = '%sversion%s/' % (STATIC_URL, VERSION_STAMP)
So the final url would be for example this:
/static/version010/style.css
And then Nginx has a rule to rewrite it back to /static/style.css
location /static {
alias /var/www/website/static/;
rewrite ^(.*)/version([\.0-9]+)/(.*)$ $1/$3;
}
Simple templatetag vstatic that creates versioned static files urls that extends Django's behaviour:
from django.conf import settings
from django.contrib.staticfiles.templatetags.staticfiles import static
#register.simple_tag
def vstatic(path):
url = static(path)
static_version = getattr(settings, 'STATIC_VERSION', '')
if static_version:
url += '?v=' + static_version
return url
If you want to automatically set STATIC_VERSION to the current git commit hash, you can use the following snippet (Python3 code adjust if necessary):
import subprocess
def get_current_commit_hash():
try:
return subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).strip().decode('utf-8')
except:
return ''
At settings.py call get_current_commit_hash(), so this will be calculated only once:
STATIC_VERSION = get_current_commit_hash()
I use a global base context in all my views, where I set the static version to be the millisecond time (that way, it will be a new version every time I restart my application):
# global base context
base_context = {
"title": settings.SITE_TITLE,
"static_version": int(round(time.time() * 1000)),
}
# function to merge context with base context
def context(items: Dict) -> Dict:
return {**base_context, **items}
# view
def view(request):
cxt = context({<...>})
return render(request, "page.html", cxt)
my page.html extends my base.html template, where I use it like this:
<link rel="stylesheet" type="text/css" href="{% static 'style.css' %}?v={{ static_version }}">
fairly simple and does the job
I use xmlrpclib, wsapi4plone to connect to plone:
client = xmlrpclib.ServerProxy('http://user:password#blah.com/plone')
is there a method to check if a folder on plone exists by its url? something like: client.exists('/sites/ng/path/to/folder')
What I did is a bit of cheating:
try:
client.get_types('/sites/ng/path/to/folder')
except:
#if there's an exception, that means there's no folder -> create it here
client.post_object(folder)
I dont have the admin rights so i can't look at the methods list (which I was told that it's somewhere on the plone site but I need to be the admin). I don't want to keep having to ask question on here about what method is available, is there a plone's methods list anywhere on the web?
A fast solution is to query the catalog, like this:
client = xmlrpclib.ServerProxy('http://user:password#blah.com/plone')
completePath = '/'.join(client.getPhysicalPath()) + '/sites/ng/path/to/folder'
if len(client.portal_catalog.searchResults(path=completePath)):
return True
Another solution could be to traverse the folders structure like this:
client = xmlrpclib.ServerProxy('http://user:password#blah.com/plone')
path = '/sites/ng/path/to/folder'
subdirs = path.split('/')[1:]
dir = client
for subdir in subdirs:
if subdir in dir.objectIds():
dir = dir[subdir]
else:
return False
return True
edit:
I have to ammend my answer. I tried to interact with the portal_catalog via xmlrpc and actually it's not so easy. My two options are good, but not for use via xmlrpc. So, taking as example transmogrify.ploneremote, a simple option (not very different from your implementation) for checking if a remote folder exists is this:
try:
path = 'http://user:password#blah.com/plone/sites/ng/path/to/folder'
xmlrpclib.ServerProxy(path).getPhysicalPath()
return True
except xmlrpclib.Fault, e:
return False