We are currently looking for ways to help the non-programming members of the sysadmin group familiarize themselves with Python scripts used for day-to-day sysadmin tasks.
Does anyone have any suggested documentation tools or best practices that we might find useful for this purpose?
Edit to address S.Lott's comment:
First, my apologies for being too brief on my initial question. My primary goal is to make sure that someone, even a non-programmer, is easily able to troubleshoot my scripts if I'm not in that day or if I leave the organization.
What I'm looking for is practices used by other people who have the "script coder" role in a technical group such as a sysadmin team. For example, before I begin the process of scripting a task, I've gotten into the habit of first writing an article in our shared wiki explaining each step in detail. I then base my Python scripts on the article--using it as pseudo code.
Other examples of the sorts of things I'm looking for:
Using tools such as Sphinx to provide easily available doc
Having group discussions to go over code before putting in production
Allowing group members to first go over the process manually (we usually go this route but perhaps we should make it a more common practice)
Or, just as valuable if not more so, negatives such as:
Found that heavy commenting is a waste of time because the logic flow is still foreign to non-programmers
Lean toward using pexpect because of the verbosity lost when using high level modules
The above are just examples of things I thought of. Hope this clarifies the question! As always, thanks SO'ers.
There is a book on this subject - "Python for Unix and Linux System Administration".
http://oreilly.com/catalog/9780596515829
And an article on developer works which might provide you the flavor that you may want to follow.
http://www.ibm.com/developerworks/aix/library/au-python/
And almost any one, irrespective of how, he is going to apply it, would want to work on the basics of the language itself. There is a good starter on web apart from tutorial that is distributed along with standard python distribution.
http://diveintopython3.ep.io/
I find that using argparse as the basis for script invocation parsing/routing tends to produce a decent first line of documentation. If used as intended, your sysadmins can run some_script --help to get a description of the script's purpose and a summary of its options.
It can be fairly trivial to link the documentation sources used by the parser building code to the actual docstrings of functions and classes in your code and that of the script itself. It depends on the complexity of the script, but this can often be a low-effort way to get sufficient documentation.
Related
I am going to write configuration tool for my Ubuntu based system. Next I would like to write frontends (text, GUI and web). But it is the most complicated project I wanted to write and I am not sure about general architecture I should use.
At the current I have functions and classes for changing system config. But these functions will probably grow & change. #Abki gave me advice how to write interface for frontends. I am going to make base classes for this interface but I don't know how to connect it with backend and next with frontends. Probably I should use design patterns like fasade, wrapper or something else.
It looks like (without interface_to_backend layer):
I don't care about UI and functions to change system config now. But I don't know how to write middle layer so It would be easy to connect it with the rest and extend functionality i the future.
I need general ideas, design patterns, advices how to implement this in Python.
I'm not sure this is entirely appropriate for SO but I'm intrigued and so I'll bite. As a rubyist I can't help much with the Python but here is some opinion on pattens from my experience.
My initial suggestion is you should review a few of the contenders out there. Specifically I'd be looking at cfengine, chef and bcfg2. They each tell a different story but if I'd summarise I'd say:
Chef has a lovely dsl syntax but is let down by a complicated architecture
bcfg2 is written in python but seems to have an annoying tendency to use XML :(
cfengine has the strongest theoretical underpinnings in promise theory (which is v.interesting BTW) but is C based.
Wikipedia also provides a pretty impressive list of configuration management tools that you will find useful.
In regard to designing your own tool I'd suggest there are three principles you want to pursue:
Simplicity, the simpler you make this the better. Simple in terms of scope, configuration and use are all important.
You'll need a single way to store data, you need to be able to trace the choices as they are made and not trample other people's changes (especially in a team environment).
Security, most configuration management tools need root privileges at some point. So you need to make sure that users can trust the code they're running.
You could use Fabric with Python as described in the article Ubuntu Server Setup with Python Fabric
The Wikipedia article at Comparison of open source configuration management software has several other tools that use Python to do this.
I like the approach taken by SALT.
If you write the GUI, text/CLI, and Web interfaces using Python, they can all use the same Python module. That way a change in one interface transparently affects the others. Plus all of those are in Python's area of strength.
there is a year old, similar question - but in case there have been changes afoot:
i'm an intermediate c++ programmer just starting out on python, post some online tuts etc i can do some basic pythoneering, but was wondering if there are good places i can look online for simple(ish) --pref console based-- code that i can learn from, ideally with some sort of commentary.
anything come to mind?
thanks
The standard library is an excellent place to the start. It's maintained by the core python team and is of high quality with a lot of interesting idioms. I'd recommend the newer modules since they don't have much backward compatibility cruft and are more representative of the language as it is now. The older ones were written for earlier versions of Python and have some restrictions when it comes to API changes etc.
The list of modules in the standard library is described at http://docs.python.org/library/. You can go through it and decide which one you want to look at (area of interest etc.).
Their sources are viewable at the mercurial repo here http://hg.python.org/cpython/file/d7e85ddb1336/Lib (as of today). These are for the mainline 2.6 release. You can also checkout the repo and browse it on your local machine.
You can also start up your interpreter, import a module (say os) and do a print os.__file__ to see where the source file is if you want to look at the code in your local editor.
ActiveState Recipes is a good source for all kinds of Python scripts. But if you want to learn the basics of Python, you might just want to look at the standard library that ships with Python ("lib" directory").
i came across This The other day, Probably you can learn some python basics and have a laugh too!
Anyways, look at the libs as they said above, they are very useful
If you enjoy riddles:
www.pythonchallenge.com
If you're an intermediate C++ programmer, you're already equipped to handle to programming concepts. I like it because it gives me a reason to learn each part of the language, without being mundane 'Hello World' tasks.
However, some of the riddles are pretty tough and/or unrelated to programming. Either way, doing the first few will probably be enough to get your confidence up with Python syntax.
I am a linux (mostly ubuntu) user with a reasonable understanding of how the system works (although I am certainly not a linux guru!). In the past I have developed small cross-platform desktop applications in python/GTK and I delivered them to clients as self-contained filetrees, so that the only dependencies were Python itself and GTK.
Now I would like to develop a small applet for ubuntu, that I would like to release under GPL 2 or 3.
In particular these are the new steps I know I must learn in order to achieve my goal (it is very possible there are a few more that I am unaware of, though!):
Integrating with gnome: I want my application to be available as an applet in the taskbar.
Using D-bus: In particular I want my applet to use the new osd-notification framework for ubuntu, but communication with other applets is also a possible feature for a second iteration.
Packaging: I would like to setup a public PPA as soon as the application will reach alpha stage, but I also would like to use dependencies from existing packages in the official repos, rather than include the libraries again in my own package.
Of course official documentation will be my first source of knowledge, but - basing my judgment on the very useful answers that I received on another topic here on SO - I decided to turn to the SO community to collect additional advice like for example:
Are there additional steps to those I outlined before, that I have to learn in order to be able to implement my project?
Based on your own experience, would you advise me to learn those steps in advance (as the knowledge of those will influence my way of coding the core functionality) or would you consider integration with gnome / d-bus and packaging as "higher encapsulating levels" that can be added on top of core functionality afterwards (note: D-bus will be used at first just for pushing data. Input data will be retrieved with a webservice)?
Would you advise me to separate my application in two packages (back-end and front-end) or to keep it together in a single package?,
Do you know of any useful resource that you would advise me to look at, for learning any of the things that I have to?
Are you aware of any common "beginner's mistakes" that I should be aware of?
These questions are not meant to be exhaustive, though: if you feel that I am missing something from the general picture, you are more than welcomed to point me in the right direction!
PS: Should I have failed in explaining my final goal, take a look at project hamster: what I want to achieve is similar in terms of user interface (meaning: the applet should display the status and clicking on it should open the application itself, from which you could both configure the applet and perform various operations).
Well, you list python, so you'll want to have pynotify in your arsenal. It wraps DBus, and gives you a direct api for manipulating the osd-notification system.
>>> import pynotify
>>> pynotify.init("Lil' Applet")
True
>>> note = pynotify.Notification(
... pynotify.get_app_name(),
... "Lil' Applet wants you to know something's up.",
... "/usr/share/icons/Human/48x48/status/dialog-information.png")
>>> note.show()
True
This displays a notification that looks like this:
[ ] **Lil' Applet**
[ICON]
[ ] Lil' Applet wants you to know something's up.
As you already know, your first and best friend will be the code written by others - copy, paste, dissect, understand.
Luckily there are a few projects that do what you intend to achieve.
I can recommend conduit's code as a prime reference how to do things in a clean fashion. I think they also have stuff on dbus. Others to keep an eye on, would be deskbar-applet, hamster (heh), and any other app you remember having feature X. Sometimes it might require some C code deciphering though (like the applet button bit - i suggest you better take it from hamster as i was having some major time getting the thing straight)
Then the "devhelp" app will be of great assistance - it allows you to read and search in man pages fast and easy. Make sure that you also have the -doc packages for all the modules you intend to use.
For user interface i strongly suggest using glade, as that will allow you to change interface later much easier. Where you can't use glade - add and alignment box and add the widget in the box in the code.
There certainly will be quirks and things that you will learn the hard way. Should not be too hard though!
The packaging, especially the autotools will be bit of a struggle, but you will get it right. For how to do debians (and from there to PPA), you can dig in the hamster's repository history. There was once a "debian" folder.
I would suggest to start small - see if you can get a window. Then put a button on it.
You don't have to do it "right" the first time. For first time it will be ok, if something works at all.
As for the separation - i would not bother about it until you get there. Splitting up into two parts and have a core, should not be too hard later. But that all depends on your priorities.
Last thing - getting friends who know the field helps too. And one way to get new friends, is by taking part in other projects, heh.
There are some very good recommendations here already, but let me suggest that you develop your applet not so much "for Ubuntu" as "for Gnome". It doesn't take much extra effort to also make RPM packages for distributions such as Fedora, and Arch Linux packages, to name two examples. There is one major disadvantage though -- to stay compatible with Debian stable you have to stick to ancient versions of GTK and GLib, or at least make any functionality depending on newer versions optional. It's painful, but apparently Debian stable users appreciate it.
I'd also suggest setting up a source code management system somewhere as early as possible. You may not be worried about your disk crashing, but sometimes it saves you a lot of trouble just to be able to revert everything you did since the last commit.
Here's the link to the documentation on the official Gnome Panel Applet library. I don't know if it has Python bindings or not.
When I asked this question two years ago Ubuntu and Gnome were much closer to each other than they are today. At present (end of 2011) gnome adopted the gnome-shell, while Canonical decided to develop their very own UI (unity)...
Part of the tension that brought to the split specifically involved libappindicator, which makes the way I formulated this question (and probably part of the answers) obsolete.
Besides, there is now AskUbuntu on stack exchange, that would probably be a much better forum to ask about ubuntu-specific question.
When digging into legacy Python code and writing Python code myself, I often use pylint. I'm also using Clone Digger. I've recently started to use rope, which is a library for automated refactoring.
But I'm looking for something else than rope. I would prefer a tool that just makes suggestions about possible refactorings: names the refactoring, optionally provides a short description of it (great for learning purposes), highlights the code section and lets me do the refactoring myself. Is there such a tool?
Check out bicycle repair man http://bicyclerepair.sourceforge.net/
What is Bicycle Repair Man?
The Bicycle Repair Man project is an attempt to create refactoring browser functionality for python. It is packaged as a library that can be added to IDEs and editors to provide refactoring capabilities. Bindings for Emacs and Vi are included with the package.
Never used it myself, but have read about it. Sounds like what you are looking for.
You might like Pythoscope, an automatic Python unit test generator, which is supposed to help you bootstrap a unit test suite by dynamically executing code.
Also, have you checked out the rope.contrib.codeassist module? It is supposed to automatically propose and perform refactorings of your source code for you.
I don't if that type of tool exists in any specific language, although the concept was mentioned in Martin Fowler's refactoring book (web reference).
The best tool I know of that currently exists is cyclomatic complexity. This article implements a cyclomatic complexity counter for python.
The other easy metric to target is method/function length, number of attributes of objects/classes and number of parameters to functions, if I recall, pylint already counted those.
Oh Forget about your tool, instead use TDD and a good book like refactoring to design patterns by Kerievsky. The problem is that refactoring is a way to improve your code and design, but only You can know what you want to achieve, no refactoring tool can do it for you.
My point is that best way to learn refactoring is to study examples, not to follow some stupid/simple tools, because they wont teach you any sophisticated refactoring nor they will tell you if you have refactoring that compose well with you code.
PS Read Fowler "Refactoring" and Kerievsky "Refactoring to design Patterns" those books are must read when learning refactoring. And they mention simple way to checking if refactoring is needed (smells).
Also consider TDD as good way to ensure that your refs are safe and do not break your code.
Beck "Test-Driven Development by example" is a good book to start with.
And Python have PyUnit for TDD.
NetBeans has an early access version that supports Python, and it is rather nice. It has some basic refactoring tools that I found the be useful. As an added bonus it works on Windows, Linux, Mac OS X and Solaris.
Check it out at:
http://www.netbeans.org/features/python/
I'm a pretty inexperienced programmer (can make tk apps, text processing, sort of understand oop), but Python is so awesome that I would like to help the community. What's the best way for a beginner to contribute?
Add to the docs. it is downright crappy
Help out other users on the dev and user mailing lists.
TEST PYTHON. bugs in programming languages are real bad. And I have seen someone discover atleast 1 bug in python
Frequent the #python channel on irc.freenode.net
Build something cool in Python and share it with others. Small values of cool are still cool. Not everyone gets to write epic, world-changing software.
Every problem solved well using Python is a way of showing how cool Python is.
I guess one way would be to help with documentation (translation, updating), until you are aware enough about the language. Also following the devs and users mail groups would give you a pretty good idea of what is being done and needs to be done by the community.
I see two ways of going about it: working on Python directly or working on something that utilizes Python
Since you're a beginner, you're probably hesitant to work on the core Python language or feel that you can't contribute in a meaningful way, which is understandable. However, as a beginner, you're in a good position to help improve documentation and other items that are essential to learning Python.
For example, the Python tutorial is less of a tutorial (in the standard sense) and more of a feature listing, at least in my opinion. When I tried to learn from it, I never got the feeling that I was building up my knowledge, like creating an application. It felt more like I was being shown all the parts that make up Python but not how to put them together into a cohesive structure.
Once I became more comfortable with the language (mostly through books and lots of practice), I eventually wrote my own tutorial, trying to provide not only the technical information but also lessons learned and "newbie gotchas".
Alternatively, you can contribute to the Python world by using Python in programs. You can contribute to projects already established, e.g. Django, PyGame, etc., or you can make your own program to "scratch an itch". Either way, you not only build your knowledge of Python but you are giving back to the community.
Finally, you can become an advocate of Python, encouraging others to learn the language. I kept suggesting to my supervisor at my last job to use Python rather than Java when a considering what to use for a new project. I tell everyone I know about the joys of Python and encourage them to give it a try. I convinced the administrator of a computer forum I frequent to create a section for Python. And, as I already said, I wrote a tutorial for Python and I'm working on a new one for wxPython.
There are many ways you can contribute to Python that aren't necessarily programming related. As your programming skills grow, you may want to move further into code contributions. But you may gain more satisfaction by helping others find the same joy you found in Python.
If you aren't up to actually working on the Python core, there are still many ways to contribute.. 2 that immediately come to mind is:
work on documentation.. it can ALWAYS be improved. Take your favorite modules and check out the documentation and add where you can.
Reporting descriptive bugs is very helpful to the development process.
Get involved with the community: http://www.python.org/dev/
Start by contributing to a Python project that you use and enjoy. This can be as simple as answering questions on the mailing list or IRC channel, offering to help with documentation and test writing or fixing bugs.