I maintain several web applications and I'd like to add some "nice" reporting/analytics pages. Building that once is simple enough (e.g. using flot or similar plotting libraries) but somehow it seems like there should be a report generation library out there which "just" generates the necessary graphs without much coding + offer some filtering ability.
There are some tools out there but for some reason there was never a good fit:
must work on Linux
open source preferred though closed source works as well as long the pricing model is also suitable for small installs
Python API required (or external services using standard web protocols)
I realize that this is not exactly a unique question but I couldn't find other stackoverflow questions with the same scope. Any pointers appreciated.
Update (2012-08-09, 15:10 UTC): I realized I did not state some more requirements/wishes:
web interface to access reports
access control: Each user can only get reports on his own data (simple to do with a library, might be hard with an external server)
filtering: I need interactive filtering of values based on some parameters (e.g. "only events in this time frame", "only in place X").
Windward* is one software company that offers a solution that seems to meet most of your needs. They offer a Python API through either Jython or a RESTful API (their Java Engine and Javelin, respectively), and their main strength is that template design is done in Microsoft Office, so reports can be very flexible and are easy to put together (most people already know how to use Word, so there's also much less learning curve than other solutions out there). You can add dynamic filters that take parameters at runtime or change on-the-fly, you can output to a variety of formats including HTML and PDF, and it works with pretty much every major datasource. For a web interface, you can either build your own and easily integrate reporting into it (Engine) or buy one pre-built and modify it to your specifications (Javelin).
On the downside, they are closed-source and without knowing more about your setup, it would be difficult for me to say whether their pricing would work. Might be worth a look, though--the links above and their documentation wiki are probably good places to start looking to see if you're a fit.
*Disclaimer: I work for Windward. I do believe they are one of the better reporting packages out there, but there are others that may fit your needs too.
Related
I have a system of stochastic differential equations implemented in MATLAB. Just 4 variables integrated with Euler-Maruyama, so nothing too fancy... the technical details aren't important for this question though.
How would you recommend I go about building a web interface (i.e., let people change the parameters and initial conditions in their web browser and then display the results)?
The first step should be translating the code to numpy/matplotlib, right?
Should I be saving the output as an image or doing some fancy HTML5 plotting stuff?
Are there any publicly available tools/frameworks that will make it easy to build a nice web UI for this kind of thing?
Any tips on where to host this kind of thing, or am I basically limited to setting up my own server?
If there are no constraints on the target language, I'd simply translate to R and use RApache. There are plenty of libraries to support this. In fact, you may not need to reimplement much code, given what's available in R libraries.
The reason I suggest R is that I've ported a lot of Matlab code to make it reusable or open, and R has usually been the easiest target for me, due to the libraries already developed.
EDIT/UPDATE: I overlooked using RStudio as a server. That might be the easiest way to go. See this page: http://www.rstudio.org/docs/server/getting_started.
Regarding where to set this up, you could look at using Amazon's "micro instances", if the amount of computation is very limited. There are also some new startups doing cloud stuff. One choice might be http://cloudnumbers.com/.
So, I'd recommend:
Post a question about the SDE stuff (to get pointers to the right package(s) on CRAN).
Install RStudio and play with the package
Try out RStudio server
Look for hosting :)
...
Profit! :) :)
I would not recommend saving stuff to disk and re-loading it. It's best to have things as smoothly coupled as possible, so that you don't have to code stuff to maintain a state on the server or browser side.
RStudio is quite cool if you want to take the R route. If you want to stick with Python, I recommend you to take a look at Femhub. It's perhaps the most mature and well developed web interface to do numerical computations in Python. Jut take a look to the "Published worksheets" examples to see what is capable of.
I would like to know what various folks in the community think about tool/language choices for a small team (3-5) developers working build simple data driven applications. We want to do data munging/Analysis/Datavis.
We will likely ultimately have Hadoop on the data crunching end and will have javascript on the front end. Ideally we want some level of R integration too.
My best thought at the moment is Django, Python, using R with Rpy (http://rpy.sourceforge.net/) and Boto (http://code.google.com/p/boto/).
Are there other good alternatives? Would there be any significant down/up sides to trying to go a JVM route instead? What tools would you use and why?
Instead of JVM you could check rApache and rserve also. Well, I have no idea though what extra would you get instead of using Rpy.
I am thinking of creating a video library software which keep track of all my videos and keep track of videos that I already haven't watched and stats like this. The stats will be specific to each user using the software.
My question is, is python appropriate to create this software or do I need something like c++.
Python is perfectly appropriate for such tasks - indeed the most popular video site, YouTube, is essentially programmed in Python (using, of course, lower-level components called from Python for such tasks as web serving, relational db, video transcoding -- there are plenty of such reusable opensource components for all these kinds of tasks, but your application's logic flow and all application-level logic can perfectly well be in Python).
Just yesterday evening, at the local Python interest group meeting in Mountain View, we had new members who just moved to Silicon Valley exactly to take Python-based jobs in the video industry, and they were saying that professional level video handing in the industry is also veering more and more towards Python -- stalwarts like Pixar and ILM had been using Python forever, but in the last year or two it's been a flood of Python adoption in the industry.
If you want your code to be REAL FAST, use C++ (or parallel fortran).
However in your application, 99% of the runtime isn't going to be in YOUR code, it's going to be in GUI libraries, OS calls, waiting for user interaction, calling libraries (written in C) to open video files and make thumbnails, that kind of stuff.
So using C++ will make your code 100 times faster, and your application will, as a result, be 1% faster, which is utterly useless. And if you write it in C++ you'll need months, whereas using Python you'll be finished much faster and have lots more fun.
Using C++ could even make it a lot slower, because in Python you can very easily build more scalable algorithms by using super powerful primitives like hashes, sets, generators, etc, try several algorithms in 5 minutes to see which one is the best, import a library which already does 90% of the work, etc.
Write it in Python.
Yes. Python is much easier to use than c++ for something like this. You may want to use it as a front-end to a DB such as sqlite3
Maybe you should take a look at this project:
Moovida
It's a complete media center, open source, written in python that is easy to extend. I don't know if it will do exactly what you want out of the box but you can probably easily add the features you want.
Of course you can use almost any programming language for almost any task. But after noting that, it's also obvious that different languages are also differently well adapted for different tasks.
C/C++ are languages that are very "hardware friendly". Basically, the languages are just one abstraction level above assembler, with C's use of pointers etc. C++ is almost like a (semi-)portable object oriented assembler, if one wants to be funny. :) This makes C/C++ fast and good at talking to hardware.
But those same features become mis-features in other cases. The pointers make it possible to walk all over the memory and unless you are careful you will leak memory all over the place. So I would say (and now C people will get angry) that C/C++ in fact are directly inappropriate for what you want to do.
You want a language that are higher, does more things like memory management automatically and invisibly. There are many to choose from there, but without a doubt Python is eminently suited for this. Python has the last couple of years emerged as The New Cool Language to write these kind of softwares in, and much multimedia software such as Freevo and the already mentioned Moovida are written in Python.
On a Java portal you can have portlets that include data provided by other applications. We want to replace our existing Java portal with a Django application, which means duplicating the Java portal's ability to display portlets. The two Sun specifications in question that we want to duplicate are JSR168 and JSR286.
I need a cPython solution. Not Jython or Java. Nothing against those tools, we just don't use them. For the record, the Jython based Portletpy does the opposite of what we are aiming to do.
Also, I suspect this question has been caused by a misunderstanding on our part of how the JSR168/JSR286 specification works. I think that JSR168/JSR286 is an arcane protocol for communicating some sort of content between separate applications, but in the Java world that tends to be done by other methods such as SOAP. Instead, the issue might be that these protocols are simply definitions of how to display content objects in views. If all we have to do is handle SOAP calls and display data, then this whole question is moot.
Simple architecture image below of what we think we want to do:
I'm not sure you can do this. From JSR 168:
If I understand correctly, you want the Django application to take the place of the existing "Java Portal/Portlet Container" in the diagram. Unfortunately, the interface between the portlet container and the individual portlets is using in-memory API calls, not as a Web service. There's no easy URL-like interface where you can call into the Java piece to get a chunk of HTML which you then incorporate into a Django-served page.
JSR 286 is an update and while it refines the mechanisms for communicating between portlets, as well as serving resources from portlets, it doesn't really change the above model radically.
I'm not saying it couldn't be done - just that there's no easy, standard way to do it.
One way to get around this could be using a WSRP (Web Services for Remote Portlets, see Wikipedia) producer, that converts a JSR 168/286 into web services and consume them from django. But it seems that WSRP has not been very popular and I couldn't find any Python platform implementations (although partial works could exist). Beside this, I'm also interested in this topic.
What makes Python stand out for use in web development? What are some examples of highly successful uses of Python on the web?
Django is, IMHO, one of the major benefits of using Python. Model your domain, code your classes, and voila, your ORM is done, and you can focus on the UI. Add in the ease of templating with the built-in templating language (or one of many others you can use as well), and it becomes very easy to whip up effective web applications in no time. Throw in the built-in admin interface, and it's a no-brainer.
Certainly one successful use of Python on the web is Google App Engine. Site authors write code in (a slightly restricted subset of) Python, which is then executed by the App Engine servers in a distributed and scalable manner.
Quotes about Python:
"Python is fast enough for our site
and allows us to produce maintainable
features in record times, with a
minimum of developers," said Cuong Do,
Software Architect, YouTube.com.
YouTube uses a lot of Python and is probably the best example of a Python success story.
A great example of a Django success story is the Washington Post, who recently shared a big list of applications they have developed:
http://push.cx/2009/washington-post-update
www.lawrence.com and www.ljworld.com are two of the first sites to use Django (before it was even open source).
djangositeoftheweek.com has a bunch of good case studies.
www.everyblock.com is another great example.
Finally, http://www.djangosites.org/ links to nearly 2,000 other Django powered sites.
Short anwser: the diversity of tools readily available and freedom of choice.
This sounds like a simple question but which it really isn't. While Python is very good for web development and this has been shown by the, oh so famous, Google App Engine, Plone and Django. One has to point out that the development way in Python requires a lot more from the developer than PHP but it gives a lot more to the mix as well.
The entry level on actually producing something is higher. This is because there are bunch of different tools for doing web development with Python. Choosing the web development framework can be a hard decision for an inexperienced developer.
Having a lot of different tools is a two edged sword. To some extent it brings you the freedom of choice to pick the one you might want but then again how do you really know which one is good for what you're doing. This brings me to my point. Python stands out from the mass by not having a standard or de facto web development library. While this is pretty much against the principle of having only one simple way of doing on thing it also brings us a wide variety of different tools with different kind of design choices. At first this might feel very frustrating because it would be so much easier if somebody had made the choice for you but now that you're left to make the choice you actually might have to think about what you're doing and what would fit. ...or you might just end up picking one and blowing your head off after you've realized that you made the wrong choice. Anyway you end up, you've made the choice and no one else.
Furthermore,
Python is both strong in web and in data analytics and machine learning. For example scikit, sci-py and numpy are very strong. In some cases, it can be very interesting to have the both elements on the same server.
For example http://rankmytweet.com uses this a lot.
trac(bug tracker) and moinmoin(wiki) are too web based python tools that I find invaluable.
GNU Mailman is another project written in python that is widely successful.
As many have pointed out, Django is a great reason to use Python...so in order to figure out why Python is great for web development, the best bet is to look at why it is a good language to build a framework like Django.
IMHO Python combines the cleanest, or at least one of the cleanest, metaprogramming models of any language with a very pure object orientation. This not only makes it possible to write extremely general abstractions that are easy to use, but also allows the abstractions to combine relatively cleanly with others. This is harder to do in languages that take a code-generation based approach to metaprogramming (e.g. Ruby).
Dynamic languages are in general good for web apps because the speed of development. Python in particular has two advantages over most of them:
"batteries included" means lots of available libraries
Django. For me this is the only reason why i use Python instead of Lua (which i like a lot more).
Besides the frameworks...
Python's pervasive support for Unicode should make i18n much smoother.
A sane namespace system makes debugging much nicer, because it's typically easier to find where things are defined.
Python's inability to function as a standalone templating language should discourage the mixture of HTML with model code
Great standard library
Other examples of Python sites are Reddit and YouTube.