General system architecture for Linux configuration tool - python

I am going to write configuration tool for my Ubuntu based system. Next I would like to write frontends (text, GUI and web). But it is the most complicated project I wanted to write and I am not sure about general architecture I should use.
At the current I have functions and classes for changing system config. But these functions will probably grow & change. #Abki gave me advice how to write interface for frontends. I am going to make base classes for this interface but I don't know how to connect it with backend and next with frontends. Probably I should use design patterns like fasade, wrapper or something else.
It looks like (without interface_to_backend layer):
I don't care about UI and functions to change system config now. But I don't know how to write middle layer so It would be easy to connect it with the rest and extend functionality i the future.
I need general ideas, design patterns, advices how to implement this in Python.

I'm not sure this is entirely appropriate for SO but I'm intrigued and so I'll bite. As a rubyist I can't help much with the Python but here is some opinion on pattens from my experience.
My initial suggestion is you should review a few of the contenders out there. Specifically I'd be looking at cfengine, chef and bcfg2. They each tell a different story but if I'd summarise I'd say:
Chef has a lovely dsl syntax but is let down by a complicated architecture
bcfg2 is written in python but seems to have an annoying tendency to use XML :(
cfengine has the strongest theoretical underpinnings in promise theory (which is v.interesting BTW) but is C based.
Wikipedia also provides a pretty impressive list of configuration management tools that you will find useful.
In regard to designing your own tool I'd suggest there are three principles you want to pursue:
Simplicity, the simpler you make this the better. Simple in terms of scope, configuration and use are all important.
You'll need a single way to store data, you need to be able to trace the choices as they are made and not trample other people's changes (especially in a team environment).
Security, most configuration management tools need root privileges at some point. So you need to make sure that users can trust the code they're running.

You could use Fabric with Python as described in the article Ubuntu Server Setup with Python Fabric
The Wikipedia article at Comparison of open source configuration management software has several other tools that use Python to do this.
I like the approach taken by SALT.

If you write the GUI, text/CLI, and Web interfaces using Python, they can all use the same Python module. That way a change in one interface transparently affects the others. Plus all of those are in Python's area of strength.

Related

How to provide common web interface to various R and python programs?

We have a large, interactive R program that we would like to interface with Shiny. There is a small Python program we would also like to create an interface for alongside it. There are no dependencies between the two sets of code, but as a research institute we'd like to provide a common interface for the two programs might be accessed by the same users. What is a good way to go about it? Is it better to consolidate under python/Django and use rpy2, or make system calls to the python program through R's Shiny interface? Are there better alternatives, or recommended practices?
Django would be an overkill.
rpy2 is a good option for small modules containing simpler methods
flask is another good option for python's side. Programmers can transmit files or even build simple web-interfaces. I prefer this method. Tell your students/collegues to define fixed APIs and response format [JSON/XML] and even a new scholar wouldn't have to spend times thinking about how to make it work. Just tell him the APIs and work with it just like Alchemy etc interfaces.
Shiny is a good option for building web-interfaces on R side. A quick tutorial that works. http://shiny.rstudio.com/tutorial/lesson2/

Web interface to a system of stochastic differential equations

I have a system of stochastic differential equations implemented in MATLAB. Just 4 variables integrated with Euler-Maruyama, so nothing too fancy... the technical details aren't important for this question though.
How would you recommend I go about building a web interface (i.e., let people change the parameters and initial conditions in their web browser and then display the results)?
The first step should be translating the code to numpy/matplotlib, right?
Should I be saving the output as an image or doing some fancy HTML5 plotting stuff?
Are there any publicly available tools/frameworks that will make it easy to build a nice web UI for this kind of thing?
Any tips on where to host this kind of thing, or am I basically limited to setting up my own server?
If there are no constraints on the target language, I'd simply translate to R and use RApache. There are plenty of libraries to support this. In fact, you may not need to reimplement much code, given what's available in R libraries.
The reason I suggest R is that I've ported a lot of Matlab code to make it reusable or open, and R has usually been the easiest target for me, due to the libraries already developed.
EDIT/UPDATE: I overlooked using RStudio as a server. That might be the easiest way to go. See this page: http://www.rstudio.org/docs/server/getting_started.
Regarding where to set this up, you could look at using Amazon's "micro instances", if the amount of computation is very limited. There are also some new startups doing cloud stuff. One choice might be http://cloudnumbers.com/.
So, I'd recommend:
Post a question about the SDE stuff (to get pointers to the right package(s) on CRAN).
Install RStudio and play with the package
Try out RStudio server
Look for hosting :)
...
Profit! :) :)
I would not recommend saving stuff to disk and re-loading it. It's best to have things as smoothly coupled as possible, so that you don't have to code stuff to maintain a state on the server or browser side.
RStudio is quite cool if you want to take the R route. If you want to stick with Python, I recommend you to take a look at Femhub. It's perhaps the most mature and well developed web interface to do numerical computations in Python. Jut take a look to the "Published worksheets" examples to see what is capable of.

Looking for advice on how to develop applets for Gnome / Ubuntu

I am a linux (mostly ubuntu) user with a reasonable understanding of how the system works (although I am certainly not a linux guru!). In the past I have developed small cross-platform desktop applications in python/GTK and I delivered them to clients as self-contained filetrees, so that the only dependencies were Python itself and GTK.
Now I would like to develop a small applet for ubuntu, that I would like to release under GPL 2 or 3.
In particular these are the new steps I know I must learn in order to achieve my goal (it is very possible there are a few more that I am unaware of, though!):
Integrating with gnome: I want my application to be available as an applet in the taskbar.
Using D-bus: In particular I want my applet to use the new osd-notification framework for ubuntu, but communication with other applets is also a possible feature for a second iteration.
Packaging: I would like to setup a public PPA as soon as the application will reach alpha stage, but I also would like to use dependencies from existing packages in the official repos, rather than include the libraries again in my own package.
Of course official documentation will be my first source of knowledge, but - basing my judgment on the very useful answers that I received on another topic here on SO - I decided to turn to the SO community to collect additional advice like for example:
Are there additional steps to those I outlined before, that I have to learn in order to be able to implement my project?
Based on your own experience, would you advise me to learn those steps in advance (as the knowledge of those will influence my way of coding the core functionality) or would you consider integration with gnome / d-bus and packaging as "higher encapsulating levels" that can be added on top of core functionality afterwards (note: D-bus will be used at first just for pushing data. Input data will be retrieved with a webservice)?
Would you advise me to separate my application in two packages (back-end and front-end) or to keep it together in a single package?,
Do you know of any useful resource that you would advise me to look at, for learning any of the things that I have to?
Are you aware of any common "beginner's mistakes" that I should be aware of?
These questions are not meant to be exhaustive, though: if you feel that I am missing something from the general picture, you are more than welcomed to point me in the right direction!
PS: Should I have failed in explaining my final goal, take a look at project hamster: what I want to achieve is similar in terms of user interface (meaning: the applet should display the status and clicking on it should open the application itself, from which you could both configure the applet and perform various operations).
Well, you list python, so you'll want to have pynotify in your arsenal. It wraps DBus, and gives you a direct api for manipulating the osd-notification system.
>>> import pynotify
>>> pynotify.init("Lil' Applet")
True
>>> note = pynotify.Notification(
... pynotify.get_app_name(),
... "Lil' Applet wants you to know something's up.",
... "/usr/share/icons/Human/48x48/status/dialog-information.png")
>>> note.show()
True
This displays a notification that looks like this:
[ ] **Lil' Applet**
[ICON]
[ ] Lil' Applet wants you to know something's up.
As you already know, your first and best friend will be the code written by others - copy, paste, dissect, understand.
Luckily there are a few projects that do what you intend to achieve.
I can recommend conduit's code as a prime reference how to do things in a clean fashion. I think they also have stuff on dbus. Others to keep an eye on, would be deskbar-applet, hamster (heh), and any other app you remember having feature X. Sometimes it might require some C code deciphering though (like the applet button bit - i suggest you better take it from hamster as i was having some major time getting the thing straight)
Then the "devhelp" app will be of great assistance - it allows you to read and search in man pages fast and easy. Make sure that you also have the -doc packages for all the modules you intend to use.
For user interface i strongly suggest using glade, as that will allow you to change interface later much easier. Where you can't use glade - add and alignment box and add the widget in the box in the code.
There certainly will be quirks and things that you will learn the hard way. Should not be too hard though!
The packaging, especially the autotools will be bit of a struggle, but you will get it right. For how to do debians (and from there to PPA), you can dig in the hamster's repository history. There was once a "debian" folder.
I would suggest to start small - see if you can get a window. Then put a button on it.
You don't have to do it "right" the first time. For first time it will be ok, if something works at all.
As for the separation - i would not bother about it until you get there. Splitting up into two parts and have a core, should not be too hard later. But that all depends on your priorities.
Last thing - getting friends who know the field helps too. And one way to get new friends, is by taking part in other projects, heh.
There are some very good recommendations here already, but let me suggest that you develop your applet not so much "for Ubuntu" as "for Gnome". It doesn't take much extra effort to also make RPM packages for distributions such as Fedora, and Arch Linux packages, to name two examples. There is one major disadvantage though -- to stay compatible with Debian stable you have to stick to ancient versions of GTK and GLib, or at least make any functionality depending on newer versions optional. It's painful, but apparently Debian stable users appreciate it.
I'd also suggest setting up a source code management system somewhere as early as possible. You may not be worried about your disk crashing, but sometimes it saves you a lot of trouble just to be able to revert everything you did since the last commit.
Here's the link to the documentation on the official Gnome Panel Applet library. I don't know if it has Python bindings or not.
When I asked this question two years ago Ubuntu and Gnome were much closer to each other than they are today. At present (end of 2011) gnome adopted the gnome-shell, while Canonical decided to develop their very own UI (unity)...
Part of the tension that brought to the split specifically involved libappindicator, which makes the way I formulated this question (and probably part of the answers) obsolete.
Besides, there is now AskUbuntu on stack exchange, that would probably be a much better forum to ask about ubuntu-specific question.

Will python provide enough performance for a proxy?

I want to start writing a http proxy that will modify responses according to some rules/filters I will configure. However, before I start coding it, I want to make sure I'm making the right choice in going with Python. Later, this tool would have to be able to process a lot of requests, so, I would like to know I can count on it later on to be able to perform when "push comes to shove".
As long as the bulk of the processing uses Python's built-in modules it should be fine as far as performance. The biggest strength of Python is its clear syntax and ease of testing/maintainability. If you find that one section of your code is slowing down the process, you can rewrite that section and use it as a C module, while keeping the bulk of your control code in Python.
However if you're looking to make the most optimized Python Code you may want to check out this SO post.
Yes, I think you will find Python to be perfectly adequate for your needs. There's a huge number of web frameworks, WSGI libraries, etc. to choose from, or learn from when building your own.
There's an interesting post on the Python History blog about how Python was supporting high performance websites in 1996.
This will depend on the library you use more than the language itself. The twisted framework is known to scale well.
Here's a proxy server example in python/twisted to get you started.
Bottomline: choose your third party tools wisely and I'm sure you'll be fine.
Python performs pretty well for most tasks, but you'll need to change the way you program if you're used to other languages. See Python is not Java for more info.
If plain old CPython doesn't give the performance you need, you have other options as well.
As has been mentioned, you can extend it in C (using a tool like swig or Pyrex). I also hear good things about PyPy as well, but bear in mind that it uses a restricted subset of Python. Lastly, a lot of people use psyco to speed up performance.

What are the benefits of using Python for web programming?

What makes Python stand out for use in web development? What are some examples of highly successful uses of Python on the web?
Django is, IMHO, one of the major benefits of using Python. Model your domain, code your classes, and voila, your ORM is done, and you can focus on the UI. Add in the ease of templating with the built-in templating language (or one of many others you can use as well), and it becomes very easy to whip up effective web applications in no time. Throw in the built-in admin interface, and it's a no-brainer.
Certainly one successful use of Python on the web is Google App Engine. Site authors write code in (a slightly restricted subset of) Python, which is then executed by the App Engine servers in a distributed and scalable manner.
Quotes about Python:
"Python is fast enough for our site
and allows us to produce maintainable
features in record times, with a
minimum of developers," said Cuong Do,
Software Architect, YouTube.com.
YouTube uses a lot of Python and is probably the best example of a Python success story.
A great example of a Django success story is the Washington Post, who recently shared a big list of applications they have developed:
http://push.cx/2009/washington-post-update
www.lawrence.com and www.ljworld.com are two of the first sites to use Django (before it was even open source).
djangositeoftheweek.com has a bunch of good case studies.
www.everyblock.com is another great example.
Finally, http://www.djangosites.org/ links to nearly 2,000 other Django powered sites.
Short anwser: the diversity of tools readily available and freedom of choice.
This sounds like a simple question but which it really isn't. While Python is very good for web development and this has been shown by the, oh so famous, Google App Engine, Plone and Django. One has to point out that the development way in Python requires a lot more from the developer than PHP but it gives a lot more to the mix as well.
The entry level on actually producing something is higher. This is because there are bunch of different tools for doing web development with Python. Choosing the web development framework can be a hard decision for an inexperienced developer.
Having a lot of different tools is a two edged sword. To some extent it brings you the freedom of choice to pick the one you might want but then again how do you really know which one is good for what you're doing. This brings me to my point. Python stands out from the mass by not having a standard or de facto web development library. While this is pretty much against the principle of having only one simple way of doing on thing it also brings us a wide variety of different tools with different kind of design choices. At first this might feel very frustrating because it would be so much easier if somebody had made the choice for you but now that you're left to make the choice you actually might have to think about what you're doing and what would fit. ...or you might just end up picking one and blowing your head off after you've realized that you made the wrong choice. Anyway you end up, you've made the choice and no one else.
Furthermore,
Python is both strong in web and in data analytics and machine learning. For example scikit, sci-py and numpy are very strong. In some cases, it can be very interesting to have the both elements on the same server.
For example http://rankmytweet.com uses this a lot.
trac(bug tracker) and moinmoin(wiki) are too web based python tools that I find invaluable.
GNU Mailman is another project written in python that is widely successful.
As many have pointed out, Django is a great reason to use Python...so in order to figure out why Python is great for web development, the best bet is to look at why it is a good language to build a framework like Django.
IMHO Python combines the cleanest, or at least one of the cleanest, metaprogramming models of any language with a very pure object orientation. This not only makes it possible to write extremely general abstractions that are easy to use, but also allows the abstractions to combine relatively cleanly with others. This is harder to do in languages that take a code-generation based approach to metaprogramming (e.g. Ruby).
Dynamic languages are in general good for web apps because the speed of development. Python in particular has two advantages over most of them:
"batteries included" means lots of available libraries
Django. For me this is the only reason why i use Python instead of Lua (which i like a lot more).
Besides the frameworks...
Python's pervasive support for Unicode should make i18n much smoother.
A sane namespace system makes debugging much nicer, because it's typically easier to find where things are defined.
Python's inability to function as a standalone templating language should discourage the mixture of HTML with model code
Great standard library
Other examples of Python sites are Reddit and YouTube.

Categories

Resources