Develop Reference

Is writing a daemon in Python a good idea?

I have to write a daemon program that constantly runs in the background and performs some simple tasks. The logic is not complicated at all, however it has to run for extended periods of time and be stable.
I think C++ would be a good choice for writing this kind of application, however I'm also considering Python since it's easier to write and test something quickly in it.
The problem that I have with Python is that I'm not sure how its runtime environment is going to behave over extended periods of time. Can it eat up more and more memory because of some GC quirks? Can it crash unexpectedly? I've never written daemons in Python before, so if anyone here did, please share your experience. Thanks!

I've written a number of daemons in Python for my last company. The short answer is, it works just fine. As long as the code itself doesn't have some huge memory bomb, I've never seen any gradual degradation or memory hogging. Be mindful of anything in the global or class scopes, because they'll live on, so use del more liberally than you might normally. Otherwise, like I said, no issues I can personally report.
And in case you're wondering, they ran for months and months (let's say 6 months usually) between routine reboots with zero problems.

Yes it can leak. Yes it can crash unexpectedly. Anything can.
I'd say you're far more likely to end up accidentally leaking in an environment with manual memory management (e.g. C++) than you are with something like Python.
As for crashing unexpectedly, well, chances are an arbitrary lump of Python might be more likely to crash unexpectedly than an arbitrary lump of Java, because the latter benefits from static typing where you can catch a whole load of errors at compile time, that Python with its duck typing and other forms of flexibility.
Realistically, Python sounds a perfectly reasonable choice for what you want to do. Take a look at something like Twisted for a decent engine to build things around, or at least for an idea of structure (your question sounds like some sort of school assignment, so I'm not sure how much freedom of implementation you get)

I've written many things in C/C++ and Perl that are initiated when a LINUX box O.S. boots, launching them using the rc.d.
Also I've written a couple of java and python scripts that are started the same way I've mentioned above, but I needed a little shell-script (.sh file) to launch them and I used rc.5.
Let me tell you that your concerns about their runtime environments are completely valid, you will have to be careful about wich runlevel you'll use... (only from rc.2 to rc.5, because rc.1 and rc.6 are for the System).
If the runlevel is too low, the python runtime might not be up at the time you are launching your program and it could flop. e.g.: In a LAMP Server MySQL and Apache are started in rc.3 where the Network is already available.
I think your best shot is to make your script in python and launch it using a .sh file from rc.5.
Good luck!

Why are scripting languages (e.g. Perl, Python, and Ruby) not suitable as shell languages? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What are the differences between shell languages like Bash (bash), Z shell (zsh), Fish (fish) and the scripting languages above that makes them more suitable for the shell?
When using the command line, the shell languages seem to be much easier. It feels for me much smoother to use bash for example than to use the shell profile in IPython, despite reports to the contrary. I think most will agree with me that a large portion of medium to large scale programming is easier in Python than in Bash. I use Python as the language I am most familiar with. The same goes for Perl and Ruby.
I have tried to articulate the reason, but I am unable to, aside from assuming that the treatment of strings differently in both has something to do with it.
The reason of this question is that I am hoping to develop a language usable in both. If you know of such a language, please post it as well.
As S.Lott explains, the question needs some clarification. I am asking about the features of the shell language versus that of scripting languages. So the comparison is not about the characteristics of various interactive (REPL) environments such as history and command line substitution. An alternative expression for the question would be:
Can a programming language that is suitable for design of complex systems be at the same time able to express useful one-liners that can access the file system or control jobs? Can a programming language usefully scale up as well as scale down?

There are a couple of differences that I can think of; just thoughtstreaming here, in no particular order:
Python & Co. are designed to be good at scripting. Bash & Co. are designed to be only good at scripting, with absolutely no compromise. IOW: Python is designed to be good both at scripting and non-scripting, Bash cares only about scripting.
Bash & Co. are untyped, Python & Co. are strongly typed, which means that the number 123, the string 123 and the file 123 are quite different. They are, however, not statically typed, which means they need to have different literals for those, in order to keep them apart.
Example:
| Ruby | Bash
-----------------------------------------
number | 123 | 123
string | '123' | 123
regexp | /123/ | 123
file | File.open('123') | 123
file descriptor | IO.open('123') | 123
URI | URI.parse('123') | 123
command | `123` | 123
Python & Co. are designed to scale up to 10000, 100000, maybe even 1000000 line programs, Bash & Co. are designed to scale down to 10 character programs.
In Bash & Co., files, directories, file descriptors, processes are all first-class objects, in Python, only Python objects are first-class, if you want to manipulate files, directories etc., you have to wrap them in a Python object first.
Shell programming is basically dataflow programming. Nobody realizes that, not even the people who write shells, but it turns out that shells are quite good at that, and general-purpose languages not so much. In the general-purpose programming world, dataflow seems to be mostly viewed as a concurrency model, not so much as a programming paradigm.
I have the feeling that trying to address these points by bolting features or DSLs onto a general-purpose programming language doesn't work. At least, I have yet to see a convincing implementation of it. There is RuSH (Ruby shell), which tries to implement a shell in Ruby, there is rush, which is an internal DSL for shell programming in Ruby, there is Hotwire, which is a Python shell, but IMO none of those come even close to competing with Bash, Zsh, fish and friends.
Actually, IMHO, the best current shell is Microsoft PowerShell, which is very surprising considering that for several decades now, Microsoft has continually had the worst shells evar. I mean, COMMAND.COM? Really? (Unfortunately, they still have a crappy terminal. It's still the "command prompt" that has been around since, what? Windows 3.0?)
PowerShell was basically created by ignoring everything Microsoft has ever done (COMMAND.COM, CMD.EXE, VBScript, JScript) and instead starting from the Unix shell, then removing all backwards-compatibility cruft (like backticks for command substitution) and massaging it a bit to make it more Windows-friendly (like using the now unused backtick as an escape character instead of the backslash which is the path component separator character in Windows). After that, is when the magic happens.
They address problem 1 and 3 from above, by basically making the opposite choice compared to Python. Python cares about large programs first, scripting second. Bash cares only about scripting. PowerShell cares about scripting first, large programs second. A defining moment for me was watching a video of an interview with Jeffrey Snover (PowerShell's lead designer), when the interviewer asked him how big of a program one could write with PowerShell and Snover answered without missing a beat: "80 characters." At that moment I realized that this is finally a guy at Microsoft who "gets" shell programming (probably related to the fact that PowerShell was neither developed by Microsoft's programming language group (i.e. lambda-calculus math nerds) nor the OS group (kernel nerds) but rather the server group (i.e. sysadmins who actually use shells)), and that I should probably take a serious look at PowerShell.
Number 2 is solved by having arguments be statically typed. So, you can write just 123 and PowerShell knows whether it is a string or a number or a file, because the cmdlet (which is what shell commands are called in PowerShell) declares the types of its arguments to the shell. This has pretty deep ramifications: unlike Unix, where each command is responsible for parsing its own arguments (the shell basically passes the arguments as an array of strings), argument parsing in PowerShell is done by the shell. The cmdlets specify all their options and flags and arguments, as well as their types and names and documentation(!) to the shell, which then can perform argument parsing, tab completion, IntelliSense, inline documentation popups etc. in one centralized place. (This is not revolutionary, and the PowerShell designers acknowledge shells like the DIGITAL Command Language (DCL) and the IBM OS/400 Command Language (CL) as prior art. For anyone who has ever used an AS/400, this should sound familiar. In OS/400, you can write a shell command and if you don't know the syntax of certain arguments, you can simply leave them out and hit F4, which will bring a menu (similar to an HTML form) with labelled fields, dropdown, help texts etc. This is only possible because the OS knows about all the possible arguments and their types.) In the Unix shell, this information is often duplicated three times: in the argument parsing code in the command itself, in the bash-completion script for tab-completion and in the manpage.
Number 4 is solved by the fact that PowerShell operates on strongly typed objects, which includes stuff like files, processes, folders and so on.
Number 5 is particularly interesting, because PowerShell is the only shell I know of, where the people who wrote it were actually aware of the fact that shells are essentially dataflow engines and deliberately implemented it as a dataflow engine.
Another nice thing about PowerShell are the naming conventions: all cmdlets are named Action-Object and moreover, there are also standardized names for specific actions and specific objects. (Again, this should sound familar to OS/400 users.) For example, everything which is related to receiving some information is called Get-Foo. And everything operating on (sub-)objects is called Bar-ChildItem. So, the equivalent to ls is Get-ChildItem (although PowerShell also provides builtin aliases ls and dir – in fact, whenever it makes sense, they provide both Unix and CMD.EXE aliases as well as abbreviations (gci in this case)).
But the killer feature IMO is the strongly typed object pipelines. While PowerShell is derived from the Unix shell, there is one very important distinction: in Unix, all communication (both via pipes and redirections as well as via command arguments) is done with untyped, unstructured strings. In PowerShell, it's all strongly typed, structured objects. This is so incredibly powerful that I seriously wonder why noone else has thought of it. (Well, they have, but they never became popular.) In my shell scripts, I estimate that up to one third of the commands is only there to act as an adapter between two other commands that don't agree on a common textual format. Many of those adapters go away in PowerShell, because the cmdlets exchange structured objects instead of unstructured text. And if you look inside the commands, then they pretty much consist of three stages: parse the textual input into an internal object representation, manipulate the objects, convert them back into text. Again, the first and third stage basically go away, because the data already comes in as objects.
However, the designers have taken great care to preserve the dynamicity and flexibility of shell scripting through what they call an Adaptive Type System.
Anyway, I don't want to turn this into a PowerShell commercial. There are plenty of things that are not so great about PowerShell, although most of those have to do either with Windows or with the specific implementation, and not so much with the concepts. (E.g. the fact that it is implemented in .NET means that the very first time you start up the shell can take up to several seconds if the .NET framework is not already in the filesystem cache due to some other application that needs it. Considering that you often use the shell for well under a second, that is completely unacceptable.)
The most important point I want to make is that if you want to look at existing work in scripting languages and shells, you shouldn't stop at Unix and the Ruby/Python/Perl/PHP family. For example, Tcl was already mentioned. Rexx would be another scripting language. Emacs Lisp would be yet another. And in the shell realm there are some of the already mentioned mainframe/midrange shells such as the OS/400 command line and DCL. Also, Plan9's rc.

It's cultural. The Bourne shell is almost 25 years old; it was one of the first scripting languages, and it was the first good solution to the central need of Unix admins. (I.e., a 'glue' to tie all the other utilities together and to do typical Unix tasks without having to compile a damn C program every time.)
By modern standards, its syntax is atrocious and its weird rules and punctuation-as-statement style (useful in the 1970s when every byte counted) make it hard for non-admins to penetrate it. But it did the job. The flaws and shortcomings were addressed by evolutionary improvements in its descendants (ksh, bash, zsh) without having to reconceive the ideas behind it. Admins stuck to the core syntax because, weird as it was, nothing else handled the simple stuff better without getting in the way.
For complex stuff, Perl came along and morphed into a sort of half-admin, half-application language. But the more complex something gets, the more it's seen as an application rather than admin work, so the business people tend to look for "programmers" rather than "admins" to do it, despite the fact that the right kind of geek tends to be both. So that's where the focus went, and the evolutionary improvements to the application capabilities of Perl resulted in...well, Python and Ruby. (That's an oversimplification, but Perl was one of several inspirations for both languages.)
Result? Specialization. Admins tend to think modern interpreted languages are too heavyweight for the things they're paid to do every day. And overall, they're right. They don't need objects. They don't care about data structures. They need commands. They need glue. Nothing else tries to do commands better than the Bourne shell concept (except maybe Tcl, which was already mentioned here); and Bourne is good enough.
Programmers -- who nowadays are having to learn about devops more and more -- look at the limitations of the Bourne shell and wonder how the hell anyone could put up with it. But the tools they know, while they certainly lean towards the Unixish style of I/O and file operations, aren't better for the purpose. I've written things like backup scripts and file renaming one-offs in Ruby, because I know it better than I know bash, but any dedicated admin could do the same thing in bash -- probably in fewer lines and with less overhead, but either way, it'd work just as well.
It's a common thing to ask "Why does everyone use Y when Z is better?" -- but evolution in technology, like evolution in everything else, tends to stop at good enough. The 'better' solution doesn't win unless the difference is viewed as a deal-breaking frustration. Bourne-type scripting might be frustrating to you, but for the people who use it all the time and for the jobs it was meant for, it's always done the job.

A shell language has to be easy to use. You want to type one-time throw away commands, not small programs. I.e., you want to type
ls -laR /usr
not
shell.ls("/usr", long=True, all=True, recursive=True)
This (also) means shell languages don't really care if an argument is an option, a string, a number or something else.
Also, programming constructs in shells are an add-on, and not even always build-in. I.e. consider the combination of if and [ in Bash or Bourne shell (sh), seq for generating sequences, and so on.
Finally, shells have specific needs that you need less, or differently in programming. I.e., pipes, file redirection, process/job control, and so on.

If you know of such a language, please post it as well.
Tcl is one such language. Mainly because it is designed to primarily be a shell interpreter for CAD programs. Here's one hardcore Python programmer's* experience of realising why Tcl was designed the way it was: I can't believe I'm praising Tcl
For me, I've written and have been using and improved Tcl shell (written in Tcl, of course) as my main Linux login shell on my homebrewed router: Pure Tcl readline
Some of the reasons I like Tcl in general has everything to do with the similarity of its syntax to traditional shells:
At its most basic, Tcl syntax is command argument argument.... There's nothing else. This is the same as Bash, C shell or even DOS shell.
A bareword is considered a string. This is again similar to traditional shells allowing you to write: open myfile.txt w+ instead of open "myfile.txt" "w+".
Because of the foundations of 1 and 2, Tcl ends up with very little extraneous syntax. You write code with less punctuation: puts Hello instead of printf("Hello");. When writing programs you don't feel the hurt so much, because you spend a lot of time thinking about what to write. When you use a shell to copy a file you don't think you just type and having to type ( and " and , and ) and ; again and again gets annoying very quickly.
*Note: not me; I'm a hardcore Tcl programmer

Who says they aren't? Take a look at Zoidberg. REPLs (Read Eval Print Loops) make crappy shells because every command must be syntactically correct, and running a program goes from being:
foo arg1 arg2 arg3
to
system "foo", "arg1", "arg2", "arg3"
And don't even get me started on trying to do redirection.
So, you need a custom shell (rather than a REPL) that understands commands and redirection and the language you want to use to bind commands together. I think zoid (the Zoidberg shell) does a pretty good job of it.

These answers inspired me to take over maintenance of the Perl-based shell Zoidberg. After some fixes, it is usable again!
Check out the user's guide or install Bundle::Zoidberg using your favorite CPAN client.

No.
No, scripting languages are probably not suitable for shells.
The problem is the dichotomy between macro languages and, well, everything else.
The shell is in a category with other legacy macro languages such as nroff and m4. In these processors, everything is a string and the processor defines a mapping from input strings to output strings.
Certain boundaries are crossed in both directions in all languages, but it's usually quite clear whether a system's category is macro or, hmm, I'm not aware of an official term ... I will say "a real language".
So sure, you could type in all your commands in a language like Ruby, and it might even be a second-best choice to a real shell, but it will never be a macro language. There is too much syntax to respect. It takes too many quotes.
But the macro language has its own issues when you start programming in it, because too many compromises had to be made to get rid of all that syntax. Strings are typed in with no quotes. Various amounts of magic need to be re-introduced to inject the missing syntax. I did a code-golf in nroff once, just to be different. It was pretty strange. The source code to big implementations in macro languages is scary.

Since both are formally programming languages, what you can do in one, you can do in the other. Actually it is a design emphasis issue. Shell languages are designed for interactive use, while scripting languages aren't.
The basic difference in the design is the storage of data between commands and the scope of variables. In Bash, etc. you have to jump through hoops to store a value (for example, commands like set a='something'), while in languages like Python you simply use an assignment statement (a = 'something'). When using the values in a shell language you have to tell the language that your want the value of the variable, while in scripting languages you have to tell the language when you want the immediate value of the string. This has effects when used interactively.
In a scripting language where ls was defined as a command
a = some_value
ls a*b
(What does a mean? Does this mean some_value * (whatever b is) or do you mean
'a'anystring'b'?. In a scripting language the default is what is stored in memory for a.)
ls 'a*b' Now means what the Unix ls a*b means.
In a Bash-like language
set a=some_value
ls a*b means what the Unix ls a*b means.
ls $a*b uses an explicit recall of the value of a.
Scripting languages make it easy to store and recall values and hard to have a transient scope on a value. Shell languages make it possible to store and recall values, but have a trivially transient scope per command.

I think it's a question of parsing. Shell languages assume by default the normal $ xxx command means you mean a command to run. In Python and Ruby need you to do system("command") or what not.
It's not that they're unsuitable, just that nobody has really done it yet; at least I think so. Rush is an example attempt in Ruby, and Python has IPython or something like that.

You beg the question. Not everyone agrees that shell languages are superior. For one, _Why doesn't
Not long ago a friend asked me how to recursively search his PHP scripts for a string. He had a lot of big binary files and templates in those directories that could have really bogged down a plain grep. I couldn't think of a way to use grep to make this happen, so I figured using find and grep together would be my best bet.
find . -name "*.php" -exec grep 'search_string' {} \; -print
Here's the above file search reworked in Ruby:
Dir['**/*.php'].each do |path|
File.open( path ) do |f|
f.grep( /search_string/ ) do |line|
puts path, ':', line
end
end
end
Your first reaction may be, "Well, that's quite a bit wordier than the original." And I just have to shrug and let it be. "It's a lot easier to extend," I say. And it works across platforms.

Scalability and extensibility? Common Lisp (you can even run CLISP, and possibly other implementations, as a login shell in Unix environments).

For the Windows users, I haven't yet felt the need for PowerShell, because I still use 4NT (now Take Command Console) from JP Software. It is a very good shell with lots of programming abilities. So it combines the best of both worlds.
When you take a look at, for example, IRB (the Ruby interpreter), it must be well possible to extend it with more one-liners to do daily scripted or mass file management and on the minute tasks.

Python vs Bash - In which kind of tasks each one outruns the other performance-wise? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
Obviously Python is more user friendly, a quick search on google shows many results that say that, as Python is byte-compiled is usually faster. I even found this that claims that you can see an improvement of over 2000% on dictionary-based operations.
What is your experience on this matter? In which kind of task each one is a clear winner?

Typical mainframe flow...
Input Disk/Tape/User (runtime) --> Job Control Language (JCL) --> Output Disk/Tape/Screen/Printer
| ^
v |
`--> COBOL Program --------'
Typical Linux flow...
Input Disk/SSD/User (runtime) --> sh/bash/ksh/zsh/... ----------> Output Disk/SSD/Screen/Printer
| ^
v |
`--> Python script --------'
| ^
v |
`--> awk script -----------'
| ^
v |
`--> sed script -----------'
| ^
v |
`--> C/C++ program --------'
| ^
v |
`--- Java program ---------'
| ^
v |
: :
Shells are the glue of Linux
Linux shells like sh/ksh/bash/... provide input/output/flow-control designation facilities much like the old mainframe Job Control Language... but on steroids! They are Turing complete languages in their own right while being optimized to efficiently pass data and control to and from other executing processes written in any language the O/S supports.
Most Linux applications, regardless what language the bulk of the program is written in, depend on shell scripts and Bash has become the most common. Clicking an icon on the desktop usually runs a short Bash script. That script, either directly or indirectly, knows where all the files needed are and sets variables and command line parameters, finally calling the program. That's a shell's simplest use.
Linux as we know it however would hardly be Linux without the thousands of shell scripts that startup the system, respond to events, control execution priorities and compile, configure and run programs. Many of these are quite large and complex.
Shells provide an infrastructure that lets us use pre-built components that are linked together at run time rather than compile time. Those components are free-standing programs in their own right that can be used alone or in other combinations without recompiling. The syntax for calling them is indistinguishable from that of a Bash builtin command, and there are in fact numerous builtin commands for which there is also a stand-alone executable on the system, often having additional options.
There is no language-wide difference between Python and Bash in performance. It entirely depends on how each is coded and which external tools are called.
Any of the well known tools like awk, sed, grep, bc, dc, tr, etc. will leave doing those operations in either language in the dust. Bash then is preferred for anything without a graphical user interface since it is easier and more efficient to call and pass data back from a tool like those with Bash than Python.
Performance
It depends on which programs the Bash shell script calls and their suitability for the subtask they are given whether the overall throughput and/or responsiveness will be better or worse than the equivalent Python. To complicate matters Python, like most languages, can also call other executables, though it is more cumbersome and thus not as often used.
User Interface
One area where Python is the clear winner is user interface. That makes it an excellent language for building local or client-server applications as it natively supports GTK graphics and is far more intuitive than Bash.
Bash only understands text. Other tools must be called for a GUI and data passed back from them. A Python script is one option. Faster but less flexible options are the binaries like YAD, Zenity, and GTKDialog.
While shells like Bash work well with GUIs like Yad, GtkDialog (embedded XML-like interface to GTK+ functions), dialog, and xmessage, Python is much more capable and so better for complex GUI windows.
Summary
Building with shell scripts is like assembling a computer with off-the-shelf components the way desktop PCs are.
Building with Python, C++ or most any other language is more like building a computer by soldering the chips (libraries) and other electronic parts together the way smartphones are.
The best results are usually obtained by using a combination of languages where each can do what they do best. One developer calls this "polyglot programming".

Generally, bash works better than python only in those environments where python is not available. :)
Seriously, I have to deal with both languages daily, and will take python instantly over bash if given the choice. Alas, I am forced to use bash on certain "small" platforms because someone has (mistakenly, IMHO) decided that python is "too large" to fit.
While it is true that bash might be faster than python for some select tasks, it can never be as quick to develop with, or as easy to maintain (at least after you get past 10 lines of code or so). Bash's sole strong point wrt python or ruby or lua, etc., is its ubiquity.

Developer efficiency matters much more to me in scenarios where both bash and Python are sensible choices.
Some tasks lend themselves well to bash, and others to Python. It also isn't unusual for me to start something as a bash script and change it to Python as it evolves over several weeks.
A big advantage Python has is in corner cases around filename handling, while it has glob, shutil, subprocess, and others for common scripting needs.

When you writing scripts performance does not matter (in most cases).
If you care about performance 'Python vs Bash' is a false question.
Python:
+ easier to write
+ easier to maintain
+ easier code reuse (try to find universal error-proof way to include files with common code in sh, I dare you)
+ you can do OOP with it too!
+ easier arguments parsing. well, not easier, exactly. it still will be too wordy to my taste, but python have argparse facility built in.
- ugly ugly 'subprocess'. try to chain commands and not to cry a river how ugly your code will become. especially if you care about exit codes.
Bash:
+ ubiquity, as was said earlier, indeed.
+ simple commands chaining. that's how you glue together different commands in a simple way. Also Bash (not sh) have some improvements, like pipefail, so chaining is really short and expressive.
+ do not require 3rd-party programs to be installed. can be executed right away.
- god, it's full of gotchas. IFS, CDPATH.. thousands of them.
If one writing a script bigger than 100 LOC: choose Python
If one need path manipulation in script: choose Python(3)
If one need somewhat like alias but slightly complicated: choose Bash/sh
Anyway, one should try both sides to get the idea what are they capable of.
Maybe answer can be extended with packaging and IDE support points, but I'm not familiar with this sides.
As always you have to choose from turd sandwich and giant douche.
And remember, just a few years ago Perl was new hope. Where it is now.

Performance-wise bash outperforms python in the process startup time.
Here are some measurements from my core i7 laptop running Linux Mint:
Starting process Startup time
empty /bin/sh script 1.7 ms
empty /bin/bash script 2.8 ms
empty python script 11.1 ms
python script with a few libs* 110 ms
*Python loaded libs are: os, os.path, json, time, requests, threading, subprocess
This shows a huge difference however bash execution time degrades quickly if it has to do anything sensible since it usually must call external processes.
If you care about performance use bash only for:
really simple and frequently called scripts
scripts that mainly call other processes
when you need minimal friction between manual administrative actions and scripting - fast check a few commands and place them in the file.sh

Bash is primarily a batch / shell scripting language with far less support for various data types and all sorts of quirks around control structures -- not to mention compatibility issues.
Which is faster? Neither, because you are not comparing apples to apples here. If you had to sort an ascii text file and you were using tools like zcat, sort, uniq, and sed then you will smoke Python performance wise.
However, if you need a proper programming environment that supports floating point and various control flow, then Python wins hands down. If you wrote say a recursive algorithm in Bash and Python, the Python version will win in an order of magnitude or more.

I'm posting this late answer primarily because Google likes this question.
I believe the issue and context really should be about the workflow, not the tools. The overall philosophy is always "Use the right tool for the job." But before this comes one that many often forget when they get lost in the tools: "Get the job done."
When I have a problem that isn't completely defined, I almost always start with Bash. I have solved some gnarly problems in large Bash scripts that are both readable and maintainable.
But when does the problem start to exceed what Bash should be asked to do? I have some checks I use to give me warnings:
Am I wishing Bash had 2D (or higher) arrays? If yes, it's time to realize that Bash is not a great data processing language.
Am I doing more work preparing data for other utilities than I am actually running those utilities? If yes, time again to realize Bash is not a great data processing language.
Is my script simply getting too large to manage? If yes, it is important to realize that while Bash can import script libraries, it lacks a package system like other languages. It's really a "roll your own" language compared to most others. Then again, it has a enormous amount of functionality built-in (some say too much...)
The list goes on. Bottom-line, when you are working harder to keep your scripts running that you do adding features, it's time to leave Bash.
Let's assume you've decided to move your work to Python. If your Bash scripts are clean, the initial conversion is quite straightforward. There are even several converters / translators that will do the first pass for you.
The next question is: What do you give up moving to Python?
All calls to external utilities must be wrapped in something from the subprocess module (or equivalent). There are multiple ways to do this, and until 3.7 it took some effort to get it right (3.7 improved subprocess.run() to handle all common cases on its own).
Surprisingly, Python has no standard platform-independent non-blocking utility (with timeout) for polling the keyboard (stdin). The Bash read command is an awesome tool for simple user interaction. My most common use is to show a spinner until the user presses a key, while also running a polling function (with each spinner step) to make sure things are still running well. This is a harder problem than it would appear at first, so I often simply make a call to Bash: Expensive, but it does precisely what I need.
If you are developing on an embedded or memory-constrained system, Python's memory footprint can be many times larger than Bash's (depending on the task at hand). Plus, there is almost always an instance of Bash already in memory, which may not be the case for Python.
For scripts that run once and exit quickly, Python's startup time can be much longer than Bash's. But if the script contains significant calculations, Python quickly pulls ahead.
Python has the most comprehensive package system on the planet. When Bash gets even slightly complex, Python probably has a package that makes whole chunks of Bash become a single call. However, finding the right package(s) to use is the biggest and most daunting part of becoming a Pythonista. Fortunately, Google and StackExchange are your friends.

If you are looking to cobble together a quick utility with minimal effort, bash is good. For a wrapper round an application, bash is invaluable.
Anything that may have you coming back over and over to add improvements is probably (though not always) better suited to a language like Python as Bash code comprising over a 1000 lines gets very painful to maintain. Bash code is also irritating to debug when it gets long.......
Part of the problem with these kind of questions is, from my experience, that shell scripts are usually all custom tasks. There have been very few shell scripting tasks that I have come across where there is already a solution freely available.

There are 2 scenario's where Bash performance is at least equal I believe:
Scripting of command line utilities
Scripts which take only a short time to execute; where starting the Python interpreter takes more time than the operation itself
That said, I usually don't really concern myself with performance of the scripting language itself. If performance is a real issue you don't script but program (possibly in Python).

I don't know if this is accurate, but I have found that python/ruby works much better for scripts that have a lot of mathematical computations. Otherwise you have to use dc or some other "arbitrary precision calculator". It just becomes a very big pain. With python you have much more control over floats vs ints and it is much easier to perform a lot of computations and sometimes.
In particular, I would never work with a bash script to handle binary information or bytes. Instead I would use something like python (maybe) or C++ or even Node.JS.

Performance wise both can do equally the same, so the question becomes which saves more development time?
Bash relies on calling other commands, and piping them for creating new ones. This has the advantage that you can quickly create new programs just with the code borrowed from other people, no matter what programming language they used.
This also has the side effect of resisting change in sub-commands pretty well, as the interface between them is just plain text.
Additionally Bash is very permissive on how you can write on it. This means it will work well for a wider variety of context, but it also relies on the programmer having the intention of coding in a clean safe manner. Otherwise Bash won't stop you from building a mess.
Python is more structured on style, so a messy programmer won't be as messy. It will also work on operating systems outside Linux, making it instantly more appropriate if you need that kind of portability.
But it isn't as simple for calling other commands. So if your operating system is Unix most likely you will find that developing on Bash is the fastest way to develop.
When to use Bash:
It's a non graphical program, or the engine of a graphical one.
It's only for Unix.
When to use Python:
It's a graphical program.
It shall work on Windows.

Debugging a scripting language like ruby

I am basically from the world of C language programming, now delving into the world of scripting languages like Ruby and Python.
I am wondering how to do debugging.
At present the steps I follow is,
I complete a large script,
Comment everything but the portion I
want to check
Execute the script
Though it works, I am not able to debug like how I would do in, say, a VC++ environment or something like that.
My question is, is there any better way of debugging?
Note: I guess it may be a repeated question, if so, please point me to the answer.

Your sequence seems entirely backwards to me. Here's how I do it:
I write a test for the functionality I want.
I start writing the script, executing bits and verifying test results.
I review what I'd done to document and publish.
Specifically, I execute before I complete. It's way too late by then.
There are debuggers, of course, but with good tests and good design, I've almost never needed one.

Here's a screencast on ruby debugging with ruby-debug.

Seems like the problem here is that your environment (Visual Studio) doesn't support these languages, not that these languages don't support debuggers in general.
Perl, Python, and Ruby all have fully-featured debuggers; you can find other IDEs that help you, too. For Ruby, there's RubyMine; for Perl, there's Komodo. And that's just off the top of my head.

There is a nice gentle introduction to the Python debugger here

If you're working with Python then you can find a list of debugging tools here to which I just want to add Eclipse with the Pydev extension, which makes working with breakpoints etc. also very simple.

My question is, is there any better way of debugging?"
Yes.
Your approach, "1. I complete a large script, 2. Comment everything but the portion I want to check, 3. Execute the script" is not really the best way to write any software in any language (sorry, but that's the truth.)
Do not write a large anything. Ever.
Do this.
Decompose your problem into classes of objects.
For each class, write the class by
2a. Outline the class, focus on the external interface, not the implementation details.
2b. Write tests to prove that interface works.
2c. Run the tests. They'll fail, since you only outlined the class.
2d. Fix the class until it passes the test.
2e. At some points, you'll realize your class designs aren't optimal. Refactor your design, assuring your tests still pass.
Now, write your final script. It should be short. All the classes have already been tested.
3a. Outline the script. Indeed, you can usually write the script.
3b. Write some test cases that prove the script works.
3c. Runt the tests. They may pass. You're done.
3d. If the tests don't pass, fix things until they do.
Write many small things. It works out much better in the long run that writing a large thing and commenting parts of it out.

Script languages have no differences compared with other languages in the sense that you still have to break your problems into manageable pieces -- that is, functions. So, instead of testing the whole script after finishing the whole script, I prefer to test those small functions before integrating them. TDD always helps.

There's a SO question on Ruby IDEs here - and searching for "ruby IDE" offers more.
I complete a large script
That's what caught my eye: "complete", to me, means "done", "finished", "released". Whether or not you write tests before writing the functions that pass them, or whether or not you write tests at all (and I recommend that you do) you should not be writing code that can't be run (which is a test in itself) until it's become large. Ruby and Python offer a multitude of ways to write small, individually-testable (or executable) pieces of code, so that you don't have to wait for (?) days before you can run the thing.
I'm building a (Ruby) database translation/transformation script at the moment - it's up to about 1000 lines and still not done. I seldom go more than 5 minutes without running it, or at least running the part on which I'm working. When it breaks (I'm not perfect, it breaks a lot ;-p) I know where the problem must be - in the code I wrote in the last 5 minutes. Progress is pretty fast.
I'm not asserting that IDEs/debuggers have no place: some problems don't surface until a large body of code is released: it can be really useful on occasion to drop the whole thing into a debugging environment to find out what is going on. When third-party libraries and frameworks are involved it can be extremely useful to debug into their code to locate problems (which are usually - but not always - related to faulty understanding of the library function).

You can debug your Python scripts using the included pdb module. If you want a visual debugger, you can download winpdb - don't be put off by that "win" prefix, winpdb is cross-platform.

The debugging method you described is perfect for a static language like C++, but given that the language is so different, the coding methods are similarly different. One of the big very important things in a dynamic language such as Python or Ruby is the interactive toplevel (what you get by typing, say python on the command line). This means that running a part of your program is very easy.
Even if you've written a large program before testing (which is a bad idea), it is hopefully separated into many functions. So, open up your interactive toplevel, do an import thing (for whatever thing happens to be) and then you can easily start testing your functions one by one, just calling them on the toplevel.
Of course, for a more mature project, you probably want to write out an actual test suite, and most languages have a method to do that (in Python, this is doctest and nose, don't know about other languages). At first, though, when you're writing something not particularly formal, just remember a few simple rules of debugging dynamic languages:
Start small. Don't write large programs and test them. Test each function as you write it, at least cursorily.
Use the toplevel. Running small pieces of code in a language like Python is extremely lightweight: fire up the toplevel and run it. Compare with writing a complete program and the compile-running it in, say, C++. Use that fact that you can quickly change the correctness of any function.
Debuggers are handy. But often, so are print statements. If you're only running a single function, debugging with print statements isn't that inconvenient, and also frees you from dragging along an IDE.

There's a lot of good advice here, i recommend going through some best practices:
http://github.com/edgecase/ruby_koans
http://blog.rubybestpractices.com/
http://on-ruby.blogspot.com/2009/01/ruby-best-practices-mini-interview-2.html
(and read Greg Brown's book, it's superb)
You talk about large scripts. A lot of my workflow is working out logic in irb or the python shell, then capturing them into a cascade of small, single-task focused methods, with appropriate tests (not 100% coverage, more focus on edge and corner cases).
http://binstock.blogspot.com/2008/04/perfecting-oos-small-classes-and-short.html

Anyone using Python for embedded projects? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
My company is using Python for a relatively simple embedded project. Is anyone else out there using Python on embedded platforms? Overall it's working well for us, quick to develop apps, quick to debug. I like the overall "conciseness" of the language.
The only real problem I have in day to day work is that the lack of static checking vs a regular compiler can cause problems to be thrown at run-time, e.g. a simple accidental cat of a string and an int in a print statement can bring the whole application down.

We use python in quite a lot of embedded boards with ARM processors and 16 MB of RAM (running linux).
It works really well and is really easy to make custom code quickly - one of the strong points of python.
As for reliability of the code - we try to have 100% test coverage. Writing tests with python is very quick and it gives you a wonderful feeling of confidence. We use twisted trial to run the tests and report on coverage, but there are many other tools available.
In my experience python + tests is more reliable and much quicker to write than any other alternatives.
The only downsides for embedded work is that sometimes python can be slow and sometimes it uses a lot of memory (relatively speaking). This hasn't causes us a show stopping problem yet, and python is quite easy to profile for both speed and memory if it becomes a problem.
pychecker is a very useful too also which will catch quite a lot of common errors.

BTW, see this blog post: "Type inference for Python" for an interesting discussion of type inference and static typing, including links to some Guido van Rossum blog posts describing adding optional static typing to Python.
I agree with Bruce Eckel that one is better off practicing "strong testing" than relying on strong typing. I think that applies equally well to embedded development.

Personally, I've worked on some of the software that runs in the device used by BusRadio. It's an example of an embedded project built on Twisted and Python. The device is an embedded XScale processor running a debian-derived distribution, so it might not meet certain definitions of "embedded", but it is pretty dang small: it fits into the dashboard of a school bus.
There were some interesting issues with using Python with large libraries - the interpreter can take quite a while to start up and load all the code for Twisted on a really slow chip, and some things needed special-case optimizations. However, at no point was the dynamic nature of Python a problem. The software in question certainly wasn't perfect, but at least when using Twisted, a simple programming error will not "bring the whole application down". A traceback will get logged, and processing continues.
So, if you're in an embedded environment sufficiently unconstrained that you can use Python in the first place, it's no different than developing "regular" programs (games, desktop applications, web apps). You don't need static typing there, and you don't need it here either.

At my previous employer I had wanted to spend some time playing with building embedded systems in tinypy, which is a "minimalist implementation of Python in 64k of code". (But I never got to it and I no longer have time.)

Telit makes GSM/GPRS modem modules that include an embedded Python interpreter.
I haven't tried them myself, so I don't know how the Python interpreter compares or differs from a PC implementation, such as which included modules, RAM and ROM memory limits, execution speed, etc.
However, as user foresightyj pointed out in a comment, it appears that they use Python 1.5.x, which is a truly ancient version, and so I would have trouble taking them seriously. Python developers would not enjoy downgrading to such an ancient version without so many modern Python features. I would be concerned about security issues with such an old version.

I've been working on microwave telecommunication equipments based on old and slow powerpc and 16Mb of RAM.
I've been able to port the Python 2.6.1 interpreter on VxWorks, in order to have the command line interpreter available directly from the target shell, or to execute python scripts uploaded to the target flash.
We used those scripts to perform autotest on the target or execute diagnostic procedures.
Here some details on the whole procedure: HOW TO: Port Python to VxWorks

The only real problem I have in day to
day work is that the last of static
checking vs a regular compiler can
cause problems to be thrown at
run-time, e.g. a simple accidental cat
of a string and an int in a print
statement can bring the whole
application down.
Unit tests are your only safety against these things.

Indeed, Python is often used as a 'support language' while you need to write some kind of tests - i.e. I was involved in a project, which (Python based) test framework code base was (is?) almost as big as that of the main product.
Python 'agents' works on QNX, VxWorks - and most problems we have, was to port properly threading and network related parts of our code.
It might be worth to take a look OpenMoko project a lot of embedded development in Python is done there.
Things to watch-out:
- support for Python/C extension module might behave quite strangely depending on platform/OS
- most of embedded platforms offers quite out-dated versions of Python
- finally you will find out that there is a difference between 'proper' embedded software in which every bit counts, and 'modern' embedded software that is performed on >412Mhz XScale CPUs with more thatn 128MB, and then Python just don't match the hardware that you would like to target :(

We use Python here at the university for embedded applications based on the Gumstix hardware platform. Although more capable than traditional embedded systems, we find the mix of small formfactor, low (ish) power consumption and the ease in transferring code between development on desktop machines and the target hardware invaluable.
Python is also a great language to teach the students, and with the Gumstix its great they can get code working on a low power system, rather than the headache and heartbreak that comes with using dedicated languages such as NesC.

My team wrote an embedded software made out of C++ and Python. We decided to write basic classes and heavy computational routines in C++. We wrote logic in Python. Boost libraries as glue. Using boost is never easy, but the results is excellent. Fast and easy to modify. Using python to represent the custom needings, we are able to satisfy customers' needings realtime, changing the code using injection technics. Something really exciting! (ok, I'm a geek ;)
We started prototyping in python but we suddenly realized that it was clearly too slow. So we decided to structure the program in different computational layers, in order to reach the speed requirements. C++ was the best solution.
In order to use python and c++ together we had to keep a strict control on typing.

I worked for a company which used Python on an embedded product based around an Atmel AVR32 and running embedded Linux. The firmware was initially developed on a PC (due to lack of a working hardware prototype), then later moved to the embedded hardware running on the cross-compiled Python interpreter.
The ability to debug and modify source code "live" on the device was a big plus during development, and saved a lot of time. The big disadvantages were speed and memory usage of the Python interpreter.
Following the first release of production firmware we ported critical sections of code over to C/C++. The porting effort was quite straightforward and resulted in an improvement of several orders of magnitude on speed-critical code (as you would expect).
Incidently most of the design and production testing code was written in Python, mainly running inside a test harness on a PC.

In my experience, Python has been traditionally used in desktop environments more than in the embedded field. There are two reasons, related to the fact that Python is interpreted:
C/C++ languages have higher performance than Python (and this is important in embedded systems with a slow microcontroller)
C/C++ languages have more deterministic response times (and this is important in real-time embedded systems controlling something).
Of course, as embedded systems will become faster and time-to-market shorter, Python will be more adopted in the embedded sector.

I have a Python server (using Twisted) and some helper scripts running under XP Embedded, and it's been working great.

Recent developments
MicroPython is a lean and fast implementation of the Python 3 programming language that is optimised to run on a microcontroller.
The European Space Agency (ESA) is funding further development of MicroPython. It is doing so to assess the suitability of the language for space-based applications, in particular for payloads.
WiPy 1.0 & 2.0, LoPy & SiPy are wireless MicroPython platforms sold by Pycom.

Isn't the EVE Online client a showpiece of real-time, high-performance Python?

I'm using a Gatetel GT-HE910 series module which embeds the Telit modem including 3G, GPS, AD, IO and Python 2.7. This is used for a remote data aquisition application. Python is fairy slow on these modules but we only need an update every 15 minutes or in an alarm condition so they work well.
http://www.gatetel.com/#!gt-series/cscb

Blockquote
The only real problem I have in day to day work is that the last of static checking vs a regular compiler can cause problems to be thrown at run-time, e.g. a simple accidental cat of a string and an int in a print statement can bring the whole application down
To me it is a huge deal. Problems you could find at compile time and fix the problem now have to rely at run time. Not knowing the data type and having to write additional function just to check the datatype is hassle. There is no need to do that in C. How would you declare 'volatile' in python?
Blockquote
The only downsides for embedded work is that sometimes python can be slow and sometimes it uses a lot of memory (relatively speaking). This hasn't causes us a show stopping problem yet, and python is quite easy to profile for both speed and memory if it becomes a problem.
This is also huge. For Embedded sytems or RTOS time constraint is very important.
Python is not necessary quick to code. It really depends what language you are comfortable with. Honestly it takes me 1 day to write function and unnecessary object orientation stuff which I can do in 2 hours in C.
Testing is so inconvenient I have to write the code, py_compile, copy pyc in the target then run the program, then python quits complaining variable not defined or type cast error or some petty thing like that.
My suggestion is C toolchain is available for any target. C is fast, hardware oriented,challenging and fun. Stick with C for Embedded systems. No need to install configure silly python packages just to run it.