I am coming from a Java/Tomcat background and was wondering if there is anything out there which could be similar to the Tomcat manager application?
I'm imagining a webapp that I can use to easily deploy and un-deploy Flask based webapps. I guess an analogy to Tomcat would be a WSGI server with a web based manager.
Unfortunately, the deployment story for Python / WSGI is not quite as neat as Java's WAR file based deployment. (And, while Python is not Java that doesn't mean that WAR file deployments aren't nice). So you don't have anything that will quite match your expectations there - but you may be able to cobble together something similar.
First, you'll want a web server that can easily load and unload WSGI applications without requiring a server restart - the one that immediately jumps to mind is uwsgi in emperor mode (and here's an example setup).
Second, you need a consistent way lay out your applications so the WSGI file can be picked up / generated. Something as simple as always having a root-level app.wsgi file that can be copied to the directory being watched by uwsgi will do.
Third, you'll need a script that can take a web application folder / virtualenv and move / symlink it to the "available applications" folder. You'll need another one that can add / symlink, touch (to restart) and remove (to shutdown) the app.wsgi files from the directory(ies) that uwsgi is watching for new vassel applications. If you need to run it across multiple machines (or even just one remote machine) you could use Fabric.
Fourth and finally, you'll need a little web application to enable you to manage the WSGI files for these available applications without using the command line. Since you just spent all this time building some infrastructure for it, why not use Flask and deploy it on itself to make sure everything works?
It's not a pre-built solution, but hopefully this at least points you in the right direction.
Related
I am dockerizing a Python webapp using the https://hub.docker.com/r/tiangolo/uwsgi-nginx image, which uses supervisor to control the uWSGI instance.
My app actually requires an additional supervisor-mediated process to run (LibreOffice headless, with which I generate documents through the appy module), and I'm wondering what is the proper pattern to implement it.
The way I see it, I could extend the above image with the extra supervisor config for my needs (along with all the necessary OS-level install steps), but this would be in contradiction with the general principle of running the least amount of distinct processes in a given container. However, since my Python app is designed to talk with LibreOffice only locally, I'm not sure how I could achieve it with a more containerized approach. Thanks for any help or suggestion.
The recommendation for one-process-per-container is sound - Docker only monitors the process it starts when the container runs, so if you have multiple processes they're not watched by Docker. It's also a better design - you have lightweight, focused containers with single responsibilities, and you can manage them independently.
user2105103 is right though, the image you're using already loses that benefit because it runs Python and Nginx, and you could extend it with LibreOffice headless and package your whole app without changing code.
If you move to a more "best practice" approach, you'd have a distributed app running across three containers in a Docker network:
nginx - web proxy, this is the public entry point to the app. Nginx can do routing, caching, SSL termination, rate limiting etc.
app - your Python app, only visible inside the Docker network. Receives requests from nginx and uses libreoffice for document manipulation;
libreoffice - running in headless mode with the API exposed, but only available within the Docker network.
You'd need code changes for this, bringing in something like PyOO to use the LibreOffice API remotely from the app container.
You've already blown the "one process per container" -- just add another process. It's not a hard rule, or even one that everybody agrees with.
Extend away, or better yet author your own custom container. That way you own it, you understand it, and it's optimized for your purpose.
I am writing an API that reads from MySQL and Solr (which can give latencies of 150ms) to provide formatted output. I will be hosting this on a VPS, and I need to choose a web server for this application. It will be used only within localhost (and local LAN in future).
I have these concerns:
Launches multiple worker threads to minimize bottlenecks with consurrent requests (Solr can take 150ms to return a request)
Can easily respawn when a component crashes and restarting is just a matter of servd -restart
deploying a new application is as simple as copying a folder to the www directory (or equivalent) so that new requests to this app will be served from then on.
I am not optimizing for performance for now, so I need something easy to setup. And is #3 not possible for a non-load balanced Django app?
Gunicorn is very simple to deploy and manage. It has no built-in reloading capability but you could easily use an external utility such as watchdog to monitor a directory and reload gunicorn using kill -HUP <pid>.
For very simple, internal web-apps using ASP I was able to just switch IIS 'on' and then write some ASP scripts in the www directory that would start working immediately.
Is there an equivalent webserver app for Python scripts that I can run that will automatically start serving dynamic pages (python scripts) in a certain folder (with virtually no configuration)?
Solutions I've already found are either too limited (e.g. SimpleHTTPRequestHandler doesn't serve dynamic content) or require configuring the script that does the serving.
There's always CGI. Add a script mapping of .py to "C:\Python27\python.exe" -u "%s" then drop .py files in a folder and IIS will execute them.
I'd not generally recommend it for real work—in the longer term you would definitely want to write apps to WSGI, and then deploy them through any number of interfaces including CGI—but it can be handy for quick prototyping.
For development or just to play around, here's an example using the standard Python library that I have used to help friend who wanted to get a basic CGI server up and running. It will serve python scripts from cgi-bin and files from the root folder. I'm not near a Windows computer at the moment to make sure that this still works. This also assumes Python2.x. Python 3.x has this, it's just not named the same.
Make a directory on your harddrive with a cgi-bin folder in it (Ex. "C:\server\cgi-bin")
In a command window, navigate to "C:\server" directory
Type the following assuming you've installed python 2.7 in C:\Python27:
"c:\python27\python.exe -m CGIHTTPServer"
You should get a message like "Serving HTTP on 0.0.0.0 port 8000"
Linux is the same - "python -m CGIHTTPServer" in a directory with a cgi-bin/ in it.
WSGI setups are fairly easy to get started, but in no anyway turn key. django MVC has a simple built in development server if you plan on using a more comprehensive framework.
My limited experience with Python web frameworks has taught me that most go to one extreme or the other: Django on one end is a full-stack MVC framework, that will do pretty much everything for you. On the other end, there are Flask, web.py, CherryPy, etc., which do much less, but stay out of your way.
CherryPy, for example, not only comes with no ORM, and doesn't require MVC, but it doesn't even have a templating engine. So unless you use it with something like Cheetah, you can't write what would look like .asp at all.
We're writing a web-based tool to configure our services provided by multiple servers. This includes interfaces configuration, dhcp configs etc. etc.
Having configs in database and views that generate proper output, how to send it/make it available for servers?
I'm thinking about sending it through scp and invoking reload command to services through ssh. I'm also thinking about using Func to do all the job, as this is Python tool and will seemingly integrate with python-based (django) config tool.
Any other proposals?
I tried using Puppet for config management, mostly because of all the buzz around it. Unfortunately, I discovered (too late) that the puppetmaster scales horribly, and does not handle heterogeneous environments well. It works for tens of servers, but its inherent architecture prevents scaling.
So I switched to Cfengine 3, which you barely notice any performance impact of, and scales much better because of its distributed architecture. Also, I later discovered that Puppet is just an attempt to reimplement Cfengine 2 inefficiently in Ruby. See http://verticalsysadmin.com/blog/uncategorized/relative-origins-of-cfengine-chef-and-puppet
If your setup is going to be used for something useful, not just play around with, go with Cfengine 3!
You can take a look at Fabric.
As an example, this is an adapted excerpt from one of my backup scripts that starts Mercurial server on remote host and pushes local changesets there:
from fabric.api import *
env.hosts = ['login#my.host.com']
def mybckp():
run('cd ~/somedir; hg serve -a 111.222.111.222 -d') # start mercurial server in daemon mode
local('hg push') # push local changesets
To execute it, I simply type:
fab mybckp
Basically, what Fabric offers is easy&convenient SSH access to shell of one more (remote) hosts, from inside of Python script.
I think you are looking for Puppet and Foreman to manage puppet (create groups of servers).
There are many ways to do this, including Chef, Bcfg2, Capistrano etc. Puppet has biggest "lead" now. There is definitely a learning curve, but the results are worth it.
You could keep your servers config files on the puppet master (in version control). And when you deploy the latest config files on the master, puppet clients can automatically pull them and restart services. Puppet "templates" can dynamically generate config files for each server.
Puppet has "Providers" for things like Packages(apt, yum), Files and OS awareness.
It really depends what you're intending to do, as the question is a little vague. The other answers cover the tools available; choosing one over the other comes down to purpose.
Are you intending to manage servers, and services on those servers? If so, try Puppet, CFEngine, or some other tool for managing server configurations.
Or, more specifically, are you looking for a deployment/buildout tool that talks to servers? So that you can type in something along the lines of "mytool deploy myproject", and have your project propagate to all the servers? In which case, fabric would be the tool to use.
Generally a good configuration will consist of both anyway... but for what it's worth, from the sound of it (managing DHCP/network/etc.), Puppet's the way to go.
We currently run a small shared hosting service for a couple of hundred small PHP sites on our servers. We'd like to offer Python support too, but from our initial research at least, a server restart seems to be required after each source code change.
Is this really the case? If so, we're just not going to be able to offer Python hosting support. Giving our clients the ability to upload files is easy, but we can't have them restart the (shared) server process!
PHP is easy -- you upload a new version of a file, the new version is run.
I've a lot of respect for the Python language and community, so find it hard to believe that it really requires such a crazy process to update a site's code. Please tell me I'm wrong! :-)
Python is a compiled language; the compiled byte code is cached by the Python process for later use, to improve performance. PHP, by default, is interpreted. It's a tradeoff between usability and speed.
If you're using a standard WSGI module, such as Apache's mod_wsgi, then you don't have to restart the server -- just touch the .wsgi file and the code will be reloaded. If you're using some weird server which doesn't support WSGI, you're sort of on your own usability-wise.
Depends on how you deploy the Python application. If it is as a pure Python CGI script, no restarts are necessary (not advised at all though, because it will be super slow). If you are using modwsgi in Apache, there are valid ways of reloading the source. modpython apparently has some support and accompanying issues for module reloading.
There are ways other than Apache to host Python application, including the CherryPy server, Paste Server, Zope, Twisted, and Tornado.
However, unless you have a specific reason not to use it (an since you are coming from presumably an Apache/PHP shop), I would highly recommed mod_wsgi on Apache. I know that Django recommends modwsgi on Apache and most of the other major Python frameworks will work on modwsgi.
Is this really the case?
It Depends. Code reloading is highly specific to the hosting solution. Most servers provide some way to automatically reload the WSGI script itself, but there's no standardisation; indeed, the question of how a WSGI Application object is connected to a web server at all differs widely across varying hosting environments. (You can just about make a single script file that works as deployment glue for CGI, mod_wsgi, passenger and ISAPI_WSGI, but it's not wholly trivial.)
What Python really struggles with, though, is module reloading. Which is problematic for WSGI applications because any non-trivial webapp will be encapsulating its functionality into modules and packages rather than simple standalone scripts. It turns out reloading modules is quite tricky, because if you reload() them one by one they can easily end up with bad references to old versions. Ideally the way forward would be to reload the whole Python interpreter when any file is updated, but in practice it seems some C extensions seem not to like this so it isn't generally done.
There are workarounds to reload a group of modules at once which can reliably update an application when one of its modules is touched. I use a deployment module that does this (which I haven't got around to publishing, but can chuck you a copy if you're interested) and it works great for my own webapps. But you do need a little discipline to make sure you don't accidentally start leaving references to your old modules' objects in other modules you aren't reloading; if you're talking loads of sites written by third parties whose code may be leaky, this might not be ideal.
In that case you might want to look at something like running mod_wsgi in daemon mode with an application group for each party and process-level reloading, and touch the WSGI script file when you've updated any of the modules.
You're right to complain; this (and many other WSGI deployment issues) could do with some standardisation help.