I am developing on a python webserver (Tornado). I plan to place this in a production instance with nginx in front. This will be my first time placing something into a production environment on my own. My question is how to setup files/directories for static serving. For instance my application, allows users to upload photos to the web. I recieve the requests in Tornado, and save to disk. However when a user visits their items page, I would rather the images be pulled from a static server. My question is what is the best practice for getting the images from my dynamic server to the static server? Do I rsync the image directory to the static server, then run a cron that delete the images from the dynamic server?
Best practice is use shared storage, but if can't use it, than you can use "proxy_store" option from nginx. Example from nginx doc:
location /images/ {
root /data/www;
error_page 404 = #fetch;
}
location #fetch {
internal;
proxy_pass http://backend;
proxy_store on;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /data/temp;
root /data/www;
}
Related
I cannot get a bokeh plot to work on a deployed server because of cross-domain issues. I have asked this question in a few forms and am not really getting anywhere.
I always get the error
XMLHttpRequest cannot load http://127.0.0.1:5006/bokeh/objinfo/0257493b-cce5-450d-8036-2bc57233b1dc/bd1791f4-4d28-4faa-8c9d-a6fe5a1721c1. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://my_ip_address' is therefore not allowed access. The response had HTTP status code 500.
Whether I am running an applet or trying to embed a single plot.
Here I am trying to fetch a plot script from within a Flask view
#perf.route('/_fetch_heatmap', methods=['POST'])
#login_required
def fetch_sd_heatmap():
document = Document()
session = Session(root_url='http://127.0.0.1:5006', configdir=current_app.config['BASE_DIRECTORY'])
session.use_doc('sd_viz')
session.load_document(document)
...
plots = VBox(hm_duration, hm_frequency)
document.add(plots)
push(session, document)
script = autoload_server(plots, session)
return jsonify({'script': script})
This script is returned to an ajax call within my javascript. This script is then appended to the corresponding <div>
This runs fine on my development machine.
Below is my nginx configuration for production
server {
listen my_ip default_server;
charset utf-8;
client_max_body_size 30M;
location ~ ^/(app_config.py|.git) {
deny all;
return 404;
}
location / {
index index.html index.htm;
root /home/myuser/app_directory;
try_files $uri #app;
}
location /static {
alias /home/myuser/app_directory/webapp/static;
}
location #app {
include uwsgi_params;
uwsgi_pass unix:/home/myuser/app_directory/uwsgi.sock;
uwsgi_connect_timeout 18000;
...
}
Has anyone successfully made a flask application with embedded bokeh plots from the bokeh server that runs in a production environment?
Hi just to update this discussion, as of the new Bokeh server in 0.11 there is much more extensive documentation about deployments:
http://docs.bokeh.org/en/0.11.1/docs/user_guide/server.html
Including information about running behind reverse proxies, using load-balancers and process manager and automating with tools like Salt. The never server is much more robust, scalable, and easy to use. You can see a gallery of live Bokeh server examples that have been "production" deployed continuously since January 2016 here:
http://demo.bokeh.org
As a reference, the full automated deployment is available for study on GitHub:
https://github.com/bokeh/demo.bokeh.org
Additionally, a fairly sophisticated example of embedding a session-specific Bokeh server app is demonstrated in the "Happiness" example here:
https://github.com/bokeh/bokeh/tree/master/examples/embed/server_session
But lastly I should say that the upcoiming 0.12 release will have the ability to set a custom Jinja template for Bokeh apps, meaning that things like Single Page Apps that build heavily around Bokeh documents can be served directly from the Bokeh server, without needing to embed in another web-server (if that is desired).
Yep, I want it to work like in Flask framework - there I could set parameters like this:
static_folder=os.getcwd()+"/static/", static_url_path=""
and all the files that lies in ./static/files/blabla.bla would be accessible by mysite.com/files/blabla.bla address. I really don't want to add static after mysite.com.
But if I set STATIC_URL = '/' in Django then I could get my static files by this address, but suddenly I could not fetch my pages that described in urls.py.
I believe this is not supported out of the box. Off the top of my head, one way to do it would be with a special 404 handler that, having failed to match against any of the defined URLs, treats the request as a request for a static resource. This would be reasonably easy to do in the development environment but significantly more difficult when nginx, Apache, and/or gunicorn get involved.
In other words, don't do this. Nest your statics (or put them on a different sub domain) but don't mix the URL hierarchy in this way.
This is really easy to accomplish with Nginx using try_files. Using the pseudo-settings below will make Nginx try to find a static file first, and if it fails then execution is passed to your django app.
server {
...
root /some/path/to/assets/;
try_files $uri #django;
location #django {
...
proxy_pass http://unix:/some/path/to/server.sock;
}
}
Example: The file /some/path/to/assets/myfile.ext will be available as http://mydomain/myfile.ext
I'm trying to set a reasonable cache expiry for my JS files while in development. I have the standard setup, where HTML, CSS and JS are living under the static directory.
The docs do mention this, but for the life of me I cannot get this to work. I've tried both methods implied, first
class MyFlask(flask.Flask):
def get_send_file_max_age(self, name):
if name.lower().endswith('.js'):
return 60
return flask.Flask.get_send_file_max_age(self, name)
app = MyFlask(__name__)
and
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 60
Both have had no effect, my JS files under /static are still coming back with the default cache timeout,
Cache-Control: public, max-age=43200
Any pointers appreciated.
You may want to look at webassets to manage the cache expiry. It works in both development and production environment.
I had this problem and couldn't find an answer online that worked for me.
Then I realised that my static files are not being served from Flask at all! Flask only generates my HTML. The static files are served directly by my web server (Apache in my case, yours might be Nginx or something else).
Here are the instructions for Apache.
First install the mod_expires module:
sudo a2enmod expires
Then add something like this to your .htaccess file:
ExpiresByType text/css "access plus 1 year"
ExpiresByType application/javascript "access plus 1 year"
ExpiresByType image/* "access plus 1 year"
More details on how to configure it in the Apache manual.
I am writing a web-app in web.py (a rewrite/extension of mongs) which I want to function both as a standalone application, and as a sub-app that requests can be forwarded to. The issue that I am having is that when it is used as a sub-app, static files cannot easily be served from its own static directory. Since I intend to distribute this (and not require users to combine the files into their project's static directory) I want the directory structure to be:
app_that_is_using_mongs (not mine)
static (which holds the app's static files - also not mine)
mongs (my subapp)
main.py (the code for mongs)
view (holds templates)
static (the static folder for mongs)
main.py (the code for the app that is using mongs)
...so that the entire mongs directory is separated from whatever app is using it.
I have considered a few possibilities for getting this to work:
Using a request handler that reads and outputs the files from the static directory, like:
cwd = os.path.dirname(__file__) + '/' # get current working directory
class Static:
def GET(self, filename):
"""searches for and returns a requested static file or 404s out"""
try:
return open(cwd + 'static/' + filename, 'r').read()
except:
web.application.notfound(app) # file not found
I am not sure about the performance of this solution for large files, and it seems like this should be something web.py can do on its own.
Adding another static directory by accessing the cherry.py staticdir tool through web.py... I'm not sure how to do something like this (interacting directly with the server that web.py is running on), and I don't think it would still work if I switched to a Gunicorn server (or any server but cherry.py).
Fixing the way that web.py handles static files to make it more extendable... If there is no other way, then rewriting this portion of web.py and maybe getting it pushed into the main repo is probably the best way.
So, what is the best way to do this?
In web.py the static assets aren't served through the application router. Instead the http server has a check weather the request url starts with /static. This means that weather you have a sub-application or not, the /static/... maps directly to the static directory in the root application.
Your first idea of building a static class will definitely work, but you are right that there is a definite performance implication - though, you'd have to benchmark it to really know how bad it is.
Another option, which is operationally worse, but is a temporary fix is to create a soft-link from the static directory of the parent app, into the sub-application's static directory. i.e.
parent_app/
static/
sub_app/ -> parent_app/sub_app/static/sub_app
...
sub_app/
static/
sub_app/
...
Then, when you want to access a static asset from sub_app, you would hit a url like: /static/sub_app/asset. Since this url starts with /static, it will get caught by the http server and redirect to the static directory, following the soft link, and resolves to the actual asset. Because of the sub_app directory, this solution will work when running the sub_app directly, or running the parent_app. You will have to setup this soft link on every server you deploy to, and for every development environment, which makes this less than ideal.
I'm using the Django manage.py runserver for developing my application (obviously), but it takes 10 seconds to completely load a page because the development server is very, very slow at serving static media.
Is there any way to speed it up or some kind of workaround? I'm using Windows 7.
Consider using mod_wsgi instead, and having httpd handle the static media.
Development server is simple unsafe single-threaded application, so you cannot do much.
One trick you could try is to redirect /site_media to second development server, but this is ugly and probably wouldn't help that much. So you could try bundling/compressing multiple assets into one css/js (e.g. using YUI Compressor).
And in any case, you should have separate static media server, that can serve multiple assets at once.
Install Firefox (if you haven't already), and install the Firebug Add-On. Restart your browser. In the lower-right corner click the "bug" icon and make sure that in the "Network" tab (it's a dropdown) of the Firebug panel that opens at the bottom of the browser, the network monitor is active.
Now with the network tab of Firebug open, open your Django-generated page that you observed to load slowly. Take a look at the timeline bars. You'll notice that the colored fragment(s) of each bar indicate(s) the reason for each request's total "load" time. Violet, for instance, means that actually the browser is waiting for the server to generate the response. Gray means it's receiving content. And so on. Hovering over the bars will display a color legend.
With Firebug's network monitor you should be able to pinpoint how exactly your browser and/or server are spending their 10 seconds.
Run lighttpd to serve the static content and use the MEDIA_URL to point the pages at the lighttpd server that servers the static stuff.
You can try using django-extensions runserver_plus command with the --threaded option as a replacement for Django's runserver command. Under the hood, it uses Werkzeug as the threaded WSGI server. You may notice a huge improvement in loading times for static files.
Lightning fast ans easy on resources when using NGINX for serving static and media files. Here's how it goes. However, you'll need to adapt some paths according to your use case and system. But I think this will get you started:
1) Download NGINX for your system, in your case Windows:
http://nginx.org/
2) Unpack the zip file. Here's how your NGINX config file may look like. This file lives inside nginx/conf/:
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
# root path to your project
# use "..." if spaces are in the path
# you may add a drive letter if required, e.g. root c:/foo
# use / instead of \. It's simpler and works
root /example/path/django/project/;
server {
# any free port number will do; Django dev server usually runs on port 80
listen 8000 default;
server_name localhost; # or 127.0.0.1
}
}
3) Start NGINX by calling the nginx.exe - no options needed.
4) Tweak your Djnago project's settings.py file:
if DEBUG:
STATIC_URL = 'http://localhost:8000/static/'
# against Django recommendation, I often still use the static
# directory for user uploads; old-style Django ;-)
MEDIA_URL = 'http://localhost:8000/static/uploads/'
Now, the static URLs in Django should look something like this:
http://localhost:8000/static/js/base.js
... 5) Stop the server by calling:
taskkill /f /IM nginx.exe
Well, that's it. I typed this quickly, so just let me know if anything's unclear or not working for you. I may improve this answer accordingly.