Is writing a daemon in Python a good idea?

Is writing a daemon in Python a good idea? - python

I have to write a daemon program that constantly runs in the background and performs some simple tasks. The logic is not complicated at all, however it has to run for extended periods of time and be stable.
I think C++ would be a good choice for writing this kind of application, however I'm also considering Python since it's easier to write and test something quickly in it.
The problem that I have with Python is that I'm not sure how its runtime environment is going to behave over extended periods of time. Can it eat up more and more memory because of some GC quirks? Can it crash unexpectedly? I've never written daemons in Python before, so if anyone here did, please share your experience. Thanks!

I've written a number of daemons in Python for my last company. The short answer is, it works just fine. As long as the code itself doesn't have some huge memory bomb, I've never seen any gradual degradation or memory hogging. Be mindful of anything in the global or class scopes, because they'll live on, so use del more liberally than you might normally. Otherwise, like I said, no issues I can personally report.
And in case you're wondering, they ran for months and months (let's say 6 months usually) between routine reboots with zero problems.

Yes it can leak. Yes it can crash unexpectedly. Anything can.
I'd say you're far more likely to end up accidentally leaking in an environment with manual memory management (e.g. C++) than you are with something like Python.
As for crashing unexpectedly, well, chances are an arbitrary lump of Python might be more likely to crash unexpectedly than an arbitrary lump of Java, because the latter benefits from static typing where you can catch a whole load of errors at compile time, that Python with its duck typing and other forms of flexibility.
Realistically, Python sounds a perfectly reasonable choice for what you want to do. Take a look at something like Twisted for a decent engine to build things around, or at least for an idea of structure (your question sounds like some sort of school assignment, so I'm not sure how much freedom of implementation you get)

I've written many things in C/C++ and Perl that are initiated when a LINUX box O.S. boots, launching them using the rc.d.
Also I've written a couple of java and python scripts that are started the same way I've mentioned above, but I needed a little shell-script (.sh file) to launch them and I used rc.5.
Let me tell you that your concerns about their runtime environments are completely valid, you will have to be careful about wich runlevel you'll use... (only from rc.2 to rc.5, because rc.1 and rc.6 are for the System).
If the runlevel is too low, the python runtime might not be up at the time you are launching your program and it could flop. e.g.: In a LAMP Server MySQL and Apache are started in rc.3 where the Network is already available.
I think your best shot is to make your script in python and launch it using a .sh file from rc.5.
Good luck!

Related

Why does my Python script run faster than the Shell script? [duplicate]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
Obviously Python is more user friendly, a quick search on google shows many results that say that, as Python is byte-compiled is usually faster. I even found this that claims that you can see an improvement of over 2000% on dictionary-based operations.
What is your experience on this matter? In which kind of task each one is a clear winner?

Typical mainframe flow...
Input Disk/Tape/User (runtime) --> Job Control Language (JCL) --> Output Disk/Tape/Screen/Printer
| ^
v |
`--> COBOL Program --------'
Typical Linux flow...
Input Disk/SSD/User (runtime) --> sh/bash/ksh/zsh/... ----------> Output Disk/SSD/Screen/Printer
| ^
v |
`--> Python script --------'
| ^
v |
`--> awk script -----------'
| ^
v |
`--> sed script -----------'
| ^
v |
`--> C/C++ program --------'
| ^
v |
`--- Java program ---------'
| ^
v |
: :
Shells are the glue of Linux
Linux shells like sh/ksh/bash/... provide input/output/flow-control designation facilities much like the old mainframe Job Control Language... but on steroids! They are Turing complete languages in their own right while being optimized to efficiently pass data and control to and from other executing processes written in any language the O/S supports.
Most Linux applications, regardless what language the bulk of the program is written in, depend on shell scripts and Bash has become the most common. Clicking an icon on the desktop usually runs a short Bash script. That script, either directly or indirectly, knows where all the files needed are and sets variables and command line parameters, finally calling the program. That's a shell's simplest use.
Linux as we know it however would hardly be Linux without the thousands of shell scripts that startup the system, respond to events, control execution priorities and compile, configure and run programs. Many of these are quite large and complex.
Shells provide an infrastructure that lets us use pre-built components that are linked together at run time rather than compile time. Those components are free-standing programs in their own right that can be used alone or in other combinations without recompiling. The syntax for calling them is indistinguishable from that of a Bash builtin command, and there are in fact numerous builtin commands for which there is also a stand-alone executable on the system, often having additional options.
There is no language-wide difference between Python and Bash in performance. It entirely depends on how each is coded and which external tools are called.
Any of the well known tools like awk, sed, grep, bc, dc, tr, etc. will leave doing those operations in either language in the dust. Bash then is preferred for anything without a graphical user interface since it is easier and more efficient to call and pass data back from a tool like those with Bash than Python.
Performance
It depends on which programs the Bash shell script calls and their suitability for the subtask they are given whether the overall throughput and/or responsiveness will be better or worse than the equivalent Python. To complicate matters Python, like most languages, can also call other executables, though it is more cumbersome and thus not as often used.
User Interface
One area where Python is the clear winner is user interface. That makes it an excellent language for building local or client-server applications as it natively supports GTK graphics and is far more intuitive than Bash.
Bash only understands text. Other tools must be called for a GUI and data passed back from them. A Python script is one option. Faster but less flexible options are the binaries like YAD, Zenity, and GTKDialog.
While shells like Bash work well with GUIs like Yad, GtkDialog (embedded XML-like interface to GTK+ functions), dialog, and xmessage, Python is much more capable and so better for complex GUI windows.
Summary
Building with shell scripts is like assembling a computer with off-the-shelf components the way desktop PCs are.
Building with Python, C++ or most any other language is more like building a computer by soldering the chips (libraries) and other electronic parts together the way smartphones are.
The best results are usually obtained by using a combination of languages where each can do what they do best. One developer calls this "polyglot programming".

Generally, bash works better than python only in those environments where python is not available. :)
Seriously, I have to deal with both languages daily, and will take python instantly over bash if given the choice. Alas, I am forced to use bash on certain "small" platforms because someone has (mistakenly, IMHO) decided that python is "too large" to fit.
While it is true that bash might be faster than python for some select tasks, it can never be as quick to develop with, or as easy to maintain (at least after you get past 10 lines of code or so). Bash's sole strong point wrt python or ruby or lua, etc., is its ubiquity.

Developer efficiency matters much more to me in scenarios where both bash and Python are sensible choices.
Some tasks lend themselves well to bash, and others to Python. It also isn't unusual for me to start something as a bash script and change it to Python as it evolves over several weeks.
A big advantage Python has is in corner cases around filename handling, while it has glob, shutil, subprocess, and others for common scripting needs.

When you writing scripts performance does not matter (in most cases).
If you care about performance 'Python vs Bash' is a false question.
Python:
+ easier to write
+ easier to maintain
+ easier code reuse (try to find universal error-proof way to include files with common code in sh, I dare you)
+ you can do OOP with it too!
+ easier arguments parsing. well, not easier, exactly. it still will be too wordy to my taste, but python have argparse facility built in.
- ugly ugly 'subprocess'. try to chain commands and not to cry a river how ugly your code will become. especially if you care about exit codes.
Bash:
+ ubiquity, as was said earlier, indeed.
+ simple commands chaining. that's how you glue together different commands in a simple way. Also Bash (not sh) have some improvements, like pipefail, so chaining is really short and expressive.
+ do not require 3rd-party programs to be installed. can be executed right away.
- god, it's full of gotchas. IFS, CDPATH.. thousands of them.
If one writing a script bigger than 100 LOC: choose Python
If one need path manipulation in script: choose Python(3)
If one need somewhat like alias but slightly complicated: choose Bash/sh
Anyway, one should try both sides to get the idea what are they capable of.
Maybe answer can be extended with packaging and IDE support points, but I'm not familiar with this sides.
As always you have to choose from turd sandwich and giant douche.
And remember, just a few years ago Perl was new hope. Where it is now.

Performance-wise bash outperforms python in the process startup time.
Here are some measurements from my core i7 laptop running Linux Mint:
Starting process Startup time
empty /bin/sh script 1.7 ms
empty /bin/bash script 2.8 ms
empty python script 11.1 ms
python script with a few libs* 110 ms
*Python loaded libs are: os, os.path, json, time, requests, threading, subprocess
This shows a huge difference however bash execution time degrades quickly if it has to do anything sensible since it usually must call external processes.
If you care about performance use bash only for:
really simple and frequently called scripts
scripts that mainly call other processes
when you need minimal friction between manual administrative actions and scripting - fast check a few commands and place them in the file.sh

Bash is primarily a batch / shell scripting language with far less support for various data types and all sorts of quirks around control structures -- not to mention compatibility issues.
Which is faster? Neither, because you are not comparing apples to apples here. If you had to sort an ascii text file and you were using tools like zcat, sort, uniq, and sed then you will smoke Python performance wise.
However, if you need a proper programming environment that supports floating point and various control flow, then Python wins hands down. If you wrote say a recursive algorithm in Bash and Python, the Python version will win in an order of magnitude or more.

I'm posting this late answer primarily because Google likes this question.
I believe the issue and context really should be about the workflow, not the tools. The overall philosophy is always "Use the right tool for the job." But before this comes one that many often forget when they get lost in the tools: "Get the job done."
When I have a problem that isn't completely defined, I almost always start with Bash. I have solved some gnarly problems in large Bash scripts that are both readable and maintainable.
But when does the problem start to exceed what Bash should be asked to do? I have some checks I use to give me warnings:
Am I wishing Bash had 2D (or higher) arrays? If yes, it's time to realize that Bash is not a great data processing language.
Am I doing more work preparing data for other utilities than I am actually running those utilities? If yes, time again to realize Bash is not a great data processing language.
Is my script simply getting too large to manage? If yes, it is important to realize that while Bash can import script libraries, it lacks a package system like other languages. It's really a "roll your own" language compared to most others. Then again, it has a enormous amount of functionality built-in (some say too much...)
The list goes on. Bottom-line, when you are working harder to keep your scripts running that you do adding features, it's time to leave Bash.
Let's assume you've decided to move your work to Python. If your Bash scripts are clean, the initial conversion is quite straightforward. There are even several converters / translators that will do the first pass for you.
The next question is: What do you give up moving to Python?
All calls to external utilities must be wrapped in something from the subprocess module (or equivalent). There are multiple ways to do this, and until 3.7 it took some effort to get it right (3.7 improved subprocess.run() to handle all common cases on its own).
Surprisingly, Python has no standard platform-independent non-blocking utility (with timeout) for polling the keyboard (stdin). The Bash read command is an awesome tool for simple user interaction. My most common use is to show a spinner until the user presses a key, while also running a polling function (with each spinner step) to make sure things are still running well. This is a harder problem than it would appear at first, so I often simply make a call to Bash: Expensive, but it does precisely what I need.
If you are developing on an embedded or memory-constrained system, Python's memory footprint can be many times larger than Bash's (depending on the task at hand). Plus, there is almost always an instance of Bash already in memory, which may not be the case for Python.
For scripts that run once and exit quickly, Python's startup time can be much longer than Bash's. But if the script contains significant calculations, Python quickly pulls ahead.
Python has the most comprehensive package system on the planet. When Bash gets even slightly complex, Python probably has a package that makes whole chunks of Bash become a single call. However, finding the right package(s) to use is the biggest and most daunting part of becoming a Pythonista. Fortunately, Google and StackExchange are your friends.

If you are looking to cobble together a quick utility with minimal effort, bash is good. For a wrapper round an application, bash is invaluable.
Anything that may have you coming back over and over to add improvements is probably (though not always) better suited to a language like Python as Bash code comprising over a 1000 lines gets very painful to maintain. Bash code is also irritating to debug when it gets long.......
Part of the problem with these kind of questions is, from my experience, that shell scripts are usually all custom tasks. There have been very few shell scripting tasks that I have come across where there is already a solution freely available.

There are 2 scenario's where Bash performance is at least equal I believe:
Scripting of command line utilities
Scripts which take only a short time to execute; where starting the Python interpreter takes more time than the operation itself
That said, I usually don't really concern myself with performance of the scripting language itself. If performance is a real issue you don't script but program (possibly in Python).

I don't know if this is accurate, but I have found that python/ruby works much better for scripts that have a lot of mathematical computations. Otherwise you have to use dc or some other "arbitrary precision calculator". It just becomes a very big pain. With python you have much more control over floats vs ints and it is much easier to perform a lot of computations and sometimes.
In particular, I would never work with a bash script to handle binary information or bytes. Instead I would use something like python (maybe) or C++ or even Node.JS.

Performance wise both can do equally the same, so the question becomes which saves more development time?
Bash relies on calling other commands, and piping them for creating new ones. This has the advantage that you can quickly create new programs just with the code borrowed from other people, no matter what programming language they used.
This also has the side effect of resisting change in sub-commands pretty well, as the interface between them is just plain text.
Additionally Bash is very permissive on how you can write on it. This means it will work well for a wider variety of context, but it also relies on the programmer having the intention of coding in a clean safe manner. Otherwise Bash won't stop you from building a mess.
Python is more structured on style, so a messy programmer won't be as messy. It will also work on operating systems outside Linux, making it instantly more appropriate if you need that kind of portability.
But it isn't as simple for calling other commands. So if your operating system is Unix most likely you will find that developing on Bash is the fastest way to develop.
When to use Bash:
It's a non graphical program, or the engine of a graphical one.
It's only for Unix.
When to use Python:
It's a graphical program.
It shall work on Windows.

Python vs Bash - In which kind of tasks each one outruns the other performance-wise? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
Obviously Python is more user friendly, a quick search on google shows many results that say that, as Python is byte-compiled is usually faster. I even found this that claims that you can see an improvement of over 2000% on dictionary-based operations.
What is your experience on this matter? In which kind of task each one is a clear winner?

Generally, bash works better than python only in those environments where python is not available. :)
Seriously, I have to deal with both languages daily, and will take python instantly over bash if given the choice. Alas, I am forced to use bash on certain "small" platforms because someone has (mistakenly, IMHO) decided that python is "too large" to fit.
While it is true that bash might be faster than python for some select tasks, it can never be as quick to develop with, or as easy to maintain (at least after you get past 10 lines of code or so). Bash's sole strong point wrt python or ruby or lua, etc., is its ubiquity.

Developer efficiency matters much more to me in scenarios where both bash and Python are sensible choices.
Some tasks lend themselves well to bash, and others to Python. It also isn't unusual for me to start something as a bash script and change it to Python as it evolves over several weeks.
A big advantage Python has is in corner cases around filename handling, while it has glob, shutil, subprocess, and others for common scripting needs.

Performance-wise bash outperforms python in the process startup time.
Here are some measurements from my core i7 laptop running Linux Mint:
Starting process Startup time
empty /bin/sh script 1.7 ms
empty /bin/bash script 2.8 ms
empty python script 11.1 ms
python script with a few libs* 110 ms
*Python loaded libs are: os, os.path, json, time, requests, threading, subprocess
This shows a huge difference however bash execution time degrades quickly if it has to do anything sensible since it usually must call external processes.
If you care about performance use bash only for:
really simple and frequently called scripts
scripts that mainly call other processes
when you need minimal friction between manual administrative actions and scripting - fast check a few commands and place them in the file.sh

Bash is primarily a batch / shell scripting language with far less support for various data types and all sorts of quirks around control structures -- not to mention compatibility issues.
Which is faster? Neither, because you are not comparing apples to apples here. If you had to sort an ascii text file and you were using tools like zcat, sort, uniq, and sed then you will smoke Python performance wise.
However, if you need a proper programming environment that supports floating point and various control flow, then Python wins hands down. If you wrote say a recursive algorithm in Bash and Python, the Python version will win in an order of magnitude or more.

I'm posting this late answer primarily because Google likes this question.
I believe the issue and context really should be about the workflow, not the tools. The overall philosophy is always "Use the right tool for the job." But before this comes one that many often forget when they get lost in the tools: "Get the job done."
When I have a problem that isn't completely defined, I almost always start with Bash. I have solved some gnarly problems in large Bash scripts that are both readable and maintainable.
But when does the problem start to exceed what Bash should be asked to do? I have some checks I use to give me warnings:
Am I wishing Bash had 2D (or higher) arrays? If yes, it's time to realize that Bash is not a great data processing language.
Am I doing more work preparing data for other utilities than I am actually running those utilities? If yes, time again to realize Bash is not a great data processing language.
Is my script simply getting too large to manage? If yes, it is important to realize that while Bash can import script libraries, it lacks a package system like other languages. It's really a "roll your own" language compared to most others. Then again, it has a enormous amount of functionality built-in (some say too much...)
The list goes on. Bottom-line, when you are working harder to keep your scripts running that you do adding features, it's time to leave Bash.
Let's assume you've decided to move your work to Python. If your Bash scripts are clean, the initial conversion is quite straightforward. There are even several converters / translators that will do the first pass for you.
The next question is: What do you give up moving to Python?
All calls to external utilities must be wrapped in something from the subprocess module (or equivalent). There are multiple ways to do this, and until 3.7 it took some effort to get it right (3.7 improved subprocess.run() to handle all common cases on its own).
Surprisingly, Python has no standard platform-independent non-blocking utility (with timeout) for polling the keyboard (stdin). The Bash read command is an awesome tool for simple user interaction. My most common use is to show a spinner until the user presses a key, while also running a polling function (with each spinner step) to make sure things are still running well. This is a harder problem than it would appear at first, so I often simply make a call to Bash: Expensive, but it does precisely what I need.
If you are developing on an embedded or memory-constrained system, Python's memory footprint can be many times larger than Bash's (depending on the task at hand). Plus, there is almost always an instance of Bash already in memory, which may not be the case for Python.
For scripts that run once and exit quickly, Python's startup time can be much longer than Bash's. But if the script contains significant calculations, Python quickly pulls ahead.
Python has the most comprehensive package system on the planet. When Bash gets even slightly complex, Python probably has a package that makes whole chunks of Bash become a single call. However, finding the right package(s) to use is the biggest and most daunting part of becoming a Pythonista. Fortunately, Google and StackExchange are your friends.

If you are looking to cobble together a quick utility with minimal effort, bash is good. For a wrapper round an application, bash is invaluable.
Anything that may have you coming back over and over to add improvements is probably (though not always) better suited to a language like Python as Bash code comprising over a 1000 lines gets very painful to maintain. Bash code is also irritating to debug when it gets long.......
Part of the problem with these kind of questions is, from my experience, that shell scripts are usually all custom tasks. There have been very few shell scripting tasks that I have come across where there is already a solution freely available.

There are 2 scenario's where Bash performance is at least equal I believe:
Scripting of command line utilities
Scripts which take only a short time to execute; where starting the Python interpreter takes more time than the operation itself
That said, I usually don't really concern myself with performance of the scripting language itself. If performance is a real issue you don't script but program (possibly in Python).

I don't know if this is accurate, but I have found that python/ruby works much better for scripts that have a lot of mathematical computations. Otherwise you have to use dc or some other "arbitrary precision calculator". It just becomes a very big pain. With python you have much more control over floats vs ints and it is much easier to perform a lot of computations and sometimes.
In particular, I would never work with a bash script to handle binary information or bytes. Instead I would use something like python (maybe) or C++ or even Node.JS.

Performance wise both can do equally the same, so the question becomes which saves more development time?
Bash relies on calling other commands, and piping them for creating new ones. This has the advantage that you can quickly create new programs just with the code borrowed from other people, no matter what programming language they used.
This also has the side effect of resisting change in sub-commands pretty well, as the interface between them is just plain text.
Additionally Bash is very permissive on how you can write on it. This means it will work well for a wider variety of context, but it also relies on the programmer having the intention of coding in a clean safe manner. Otherwise Bash won't stop you from building a mess.
Python is more structured on style, so a messy programmer won't be as messy. It will also work on operating systems outside Linux, making it instantly more appropriate if you need that kind of portability.
But it isn't as simple for calling other commands. So if your operating system is Unix most likely you will find that developing on Bash is the fastest way to develop.
When to use Bash:
It's a non graphical program, or the engine of a graphical one.
It's only for Unix.
When to use Python:
It's a graphical program.
It shall work on Windows.

Embedding Python Design

There are lots of tutorials/instructions on how to embed python in an application, but nothing (that I've seen) on overall design for how the embedded interpreter should be used and interact with the application.
The only idea I could think of would be to simply give the user a method (menu option, etc) of executing scripts in the program. So certain classes, functions, objects, etc. would be exported to python, some script would do something, then said script could be run from the program.
Would such a design be "safe?" Meaning is it feasible for a malicious/poorly-written script to "damage" the program and/or computer? I assume its possible depending on the functions available to the script (e.g: it could try to overwrite some important files, etc.) How might one prevent such from happening? (e.g: script certification, program design, etc.)
This is implementation specific, but is it possible/feasible to have the effects of the script stay after its done running? Meaning if a script computes something, will the result be available to the program after execution of the script has finished? I think it is possible to do if the program were setup to interact with a specific script, but the program will be released before most scripts are written; and such a setup seems like a misuse of embedding a scripting language. Is there actually cases where you would want the result of a scripts execution to be available, or is this a contrived situation that doesn't really occur?
Are there any other designs for embedding python?
What about using python in a way similar to a plugin architecture?
Thanks,
Matthew A. Todd

The only idea I could think of would be to simply give the user a method (menu option, etc) of executing scripts in the program.
Correct.
So certain classes, functions, objects, etc. would be exported to python, some script would do something, then said script could be run from the program.
Correct.
Would such a design be "safe?"
Yes. Unless your users are malicious, psychotic sociopaths. They want to make your program do useful things. They bought/downloaded the software in the first place. They think it has value.
They trusted your software. Why not trust them?
Meaning if a script computes something, will the result be available to the program after execution of the script has finished?
Programs like Apache do this all the time. You screw up the configuration ("script"), it crashes. Lesson learned? Don't screw up the configuration.

Debugging a scripting language like ruby

I am basically from the world of C language programming, now delving into the world of scripting languages like Ruby and Python.
I am wondering how to do debugging.
At present the steps I follow is,
I complete a large script,
Comment everything but the portion I
want to check
Execute the script
Though it works, I am not able to debug like how I would do in, say, a VC++ environment or something like that.
My question is, is there any better way of debugging?
Note: I guess it may be a repeated question, if so, please point me to the answer.

Your sequence seems entirely backwards to me. Here's how I do it:
I write a test for the functionality I want.
I start writing the script, executing bits and verifying test results.
I review what I'd done to document and publish.
Specifically, I execute before I complete. It's way too late by then.
There are debuggers, of course, but with good tests and good design, I've almost never needed one.

Here's a screencast on ruby debugging with ruby-debug.

Seems like the problem here is that your environment (Visual Studio) doesn't support these languages, not that these languages don't support debuggers in general.
Perl, Python, and Ruby all have fully-featured debuggers; you can find other IDEs that help you, too. For Ruby, there's RubyMine; for Perl, there's Komodo. And that's just off the top of my head.

There is a nice gentle introduction to the Python debugger here

If you're working with Python then you can find a list of debugging tools here to which I just want to add Eclipse with the Pydev extension, which makes working with breakpoints etc. also very simple.

My question is, is there any better way of debugging?"
Yes.
Your approach, "1. I complete a large script, 2. Comment everything but the portion I want to check, 3. Execute the script" is not really the best way to write any software in any language (sorry, but that's the truth.)
Do not write a large anything. Ever.
Do this.
Decompose your problem into classes of objects.
For each class, write the class by
2a. Outline the class, focus on the external interface, not the implementation details.
2b. Write tests to prove that interface works.
2c. Run the tests. They'll fail, since you only outlined the class.
2d. Fix the class until it passes the test.
2e. At some points, you'll realize your class designs aren't optimal. Refactor your design, assuring your tests still pass.
Now, write your final script. It should be short. All the classes have already been tested.
3a. Outline the script. Indeed, you can usually write the script.
3b. Write some test cases that prove the script works.
3c. Runt the tests. They may pass. You're done.
3d. If the tests don't pass, fix things until they do.
Write many small things. It works out much better in the long run that writing a large thing and commenting parts of it out.

Script languages have no differences compared with other languages in the sense that you still have to break your problems into manageable pieces -- that is, functions. So, instead of testing the whole script after finishing the whole script, I prefer to test those small functions before integrating them. TDD always helps.

There's a SO question on Ruby IDEs here - and searching for "ruby IDE" offers more.
I complete a large script
That's what caught my eye: "complete", to me, means "done", "finished", "released". Whether or not you write tests before writing the functions that pass them, or whether or not you write tests at all (and I recommend that you do) you should not be writing code that can't be run (which is a test in itself) until it's become large. Ruby and Python offer a multitude of ways to write small, individually-testable (or executable) pieces of code, so that you don't have to wait for (?) days before you can run the thing.
I'm building a (Ruby) database translation/transformation script at the moment - it's up to about 1000 lines and still not done. I seldom go more than 5 minutes without running it, or at least running the part on which I'm working. When it breaks (I'm not perfect, it breaks a lot ;-p) I know where the problem must be - in the code I wrote in the last 5 minutes. Progress is pretty fast.
I'm not asserting that IDEs/debuggers have no place: some problems don't surface until a large body of code is released: it can be really useful on occasion to drop the whole thing into a debugging environment to find out what is going on. When third-party libraries and frameworks are involved it can be extremely useful to debug into their code to locate problems (which are usually - but not always - related to faulty understanding of the library function).

You can debug your Python scripts using the included pdb module. If you want a visual debugger, you can download winpdb - don't be put off by that "win" prefix, winpdb is cross-platform.

The debugging method you described is perfect for a static language like C++, but given that the language is so different, the coding methods are similarly different. One of the big very important things in a dynamic language such as Python or Ruby is the interactive toplevel (what you get by typing, say python on the command line). This means that running a part of your program is very easy.
Even if you've written a large program before testing (which is a bad idea), it is hopefully separated into many functions. So, open up your interactive toplevel, do an import thing (for whatever thing happens to be) and then you can easily start testing your functions one by one, just calling them on the toplevel.
Of course, for a more mature project, you probably want to write out an actual test suite, and most languages have a method to do that (in Python, this is doctest and nose, don't know about other languages). At first, though, when you're writing something not particularly formal, just remember a few simple rules of debugging dynamic languages:
Start small. Don't write large programs and test them. Test each function as you write it, at least cursorily.
Use the toplevel. Running small pieces of code in a language like Python is extremely lightweight: fire up the toplevel and run it. Compare with writing a complete program and the compile-running it in, say, C++. Use that fact that you can quickly change the correctness of any function.
Debuggers are handy. But often, so are print statements. If you're only running a single function, debugging with print statements isn't that inconvenient, and also frees you from dragging along an IDE.

There's a lot of good advice here, i recommend going through some best practices:
http://github.com/edgecase/ruby_koans
http://blog.rubybestpractices.com/
http://on-ruby.blogspot.com/2009/01/ruby-best-practices-mini-interview-2.html
(and read Greg Brown's book, it's superb)
You talk about large scripts. A lot of my workflow is working out logic in irb or the python shell, then capturing them into a cascade of small, single-task focused methods, with appropriate tests (not 100% coverage, more focus on edge and corner cases).
http://binstock.blogspot.com/2008/04/perfecting-oos-small-classes-and-short.html

Anyone using Python for embedded projects? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
My company is using Python for a relatively simple embedded project. Is anyone else out there using Python on embedded platforms? Overall it's working well for us, quick to develop apps, quick to debug. I like the overall "conciseness" of the language.
The only real problem I have in day to day work is that the lack of static checking vs a regular compiler can cause problems to be thrown at run-time, e.g. a simple accidental cat of a string and an int in a print statement can bring the whole application down.

We use python in quite a lot of embedded boards with ARM processors and 16 MB of RAM (running linux).
It works really well and is really easy to make custom code quickly - one of the strong points of python.
As for reliability of the code - we try to have 100% test coverage. Writing tests with python is very quick and it gives you a wonderful feeling of confidence. We use twisted trial to run the tests and report on coverage, but there are many other tools available.
In my experience python + tests is more reliable and much quicker to write than any other alternatives.
The only downsides for embedded work is that sometimes python can be slow and sometimes it uses a lot of memory (relatively speaking). This hasn't causes us a show stopping problem yet, and python is quite easy to profile for both speed and memory if it becomes a problem.
pychecker is a very useful too also which will catch quite a lot of common errors.

BTW, see this blog post: "Type inference for Python" for an interesting discussion of type inference and static typing, including links to some Guido van Rossum blog posts describing adding optional static typing to Python.
I agree with Bruce Eckel that one is better off practicing "strong testing" than relying on strong typing. I think that applies equally well to embedded development.

Personally, I've worked on some of the software that runs in the device used by BusRadio. It's an example of an embedded project built on Twisted and Python. The device is an embedded XScale processor running a debian-derived distribution, so it might not meet certain definitions of "embedded", but it is pretty dang small: it fits into the dashboard of a school bus.
There were some interesting issues with using Python with large libraries - the interpreter can take quite a while to start up and load all the code for Twisted on a really slow chip, and some things needed special-case optimizations. However, at no point was the dynamic nature of Python a problem. The software in question certainly wasn't perfect, but at least when using Twisted, a simple programming error will not "bring the whole application down". A traceback will get logged, and processing continues.
So, if you're in an embedded environment sufficiently unconstrained that you can use Python in the first place, it's no different than developing "regular" programs (games, desktop applications, web apps). You don't need static typing there, and you don't need it here either.

At my previous employer I had wanted to spend some time playing with building embedded systems in tinypy, which is a "minimalist implementation of Python in 64k of code". (But I never got to it and I no longer have time.)

Telit makes GSM/GPRS modem modules that include an embedded Python interpreter.
I haven't tried them myself, so I don't know how the Python interpreter compares or differs from a PC implementation, such as which included modules, RAM and ROM memory limits, execution speed, etc.
However, as user foresightyj pointed out in a comment, it appears that they use Python 1.5.x, which is a truly ancient version, and so I would have trouble taking them seriously. Python developers would not enjoy downgrading to such an ancient version without so many modern Python features. I would be concerned about security issues with such an old version.

I've been working on microwave telecommunication equipments based on old and slow powerpc and 16Mb of RAM.
I've been able to port the Python 2.6.1 interpreter on VxWorks, in order to have the command line interpreter available directly from the target shell, or to execute python scripts uploaded to the target flash.
We used those scripts to perform autotest on the target or execute diagnostic procedures.
Here some details on the whole procedure: HOW TO: Port Python to VxWorks

The only real problem I have in day to
day work is that the last of static
checking vs a regular compiler can
cause problems to be thrown at
run-time, e.g. a simple accidental cat
of a string and an int in a print
statement can bring the whole
application down.
Unit tests are your only safety against these things.

Indeed, Python is often used as a 'support language' while you need to write some kind of tests - i.e. I was involved in a project, which (Python based) test framework code base was (is?) almost as big as that of the main product.
Python 'agents' works on QNX, VxWorks - and most problems we have, was to port properly threading and network related parts of our code.
It might be worth to take a look OpenMoko project a lot of embedded development in Python is done there.
Things to watch-out:
- support for Python/C extension module might behave quite strangely depending on platform/OS
- most of embedded platforms offers quite out-dated versions of Python
- finally you will find out that there is a difference between 'proper' embedded software in which every bit counts, and 'modern' embedded software that is performed on >412Mhz XScale CPUs with more thatn 128MB, and then Python just don't match the hardware that you would like to target :(

We use Python here at the university for embedded applications based on the Gumstix hardware platform. Although more capable than traditional embedded systems, we find the mix of small formfactor, low (ish) power consumption and the ease in transferring code between development on desktop machines and the target hardware invaluable.
Python is also a great language to teach the students, and with the Gumstix its great they can get code working on a low power system, rather than the headache and heartbreak that comes with using dedicated languages such as NesC.

My team wrote an embedded software made out of C++ and Python. We decided to write basic classes and heavy computational routines in C++. We wrote logic in Python. Boost libraries as glue. Using boost is never easy, but the results is excellent. Fast and easy to modify. Using python to represent the custom needings, we are able to satisfy customers' needings realtime, changing the code using injection technics. Something really exciting! (ok, I'm a geek ;)
We started prototyping in python but we suddenly realized that it was clearly too slow. So we decided to structure the program in different computational layers, in order to reach the speed requirements. C++ was the best solution.
In order to use python and c++ together we had to keep a strict control on typing.

I worked for a company which used Python on an embedded product based around an Atmel AVR32 and running embedded Linux. The firmware was initially developed on a PC (due to lack of a working hardware prototype), then later moved to the embedded hardware running on the cross-compiled Python interpreter.
The ability to debug and modify source code "live" on the device was a big plus during development, and saved a lot of time. The big disadvantages were speed and memory usage of the Python interpreter.
Following the first release of production firmware we ported critical sections of code over to C/C++. The porting effort was quite straightforward and resulted in an improvement of several orders of magnitude on speed-critical code (as you would expect).
Incidently most of the design and production testing code was written in Python, mainly running inside a test harness on a PC.

In my experience, Python has been traditionally used in desktop environments more than in the embedded field. There are two reasons, related to the fact that Python is interpreted:
C/C++ languages have higher performance than Python (and this is important in embedded systems with a slow microcontroller)
C/C++ languages have more deterministic response times (and this is important in real-time embedded systems controlling something).
Of course, as embedded systems will become faster and time-to-market shorter, Python will be more adopted in the embedded sector.

I have a Python server (using Twisted) and some helper scripts running under XP Embedded, and it's been working great.

Recent developments
MicroPython is a lean and fast implementation of the Python 3 programming language that is optimised to run on a microcontroller.
The European Space Agency (ESA) is funding further development of MicroPython. It is doing so to assess the suitability of the language for space-based applications, in particular for payloads.
WiPy 1.0 & 2.0, LoPy & SiPy are wireless MicroPython platforms sold by Pycom.

Isn't the EVE Online client a showpiece of real-time, high-performance Python?

I'm using a Gatetel GT-HE910 series module which embeds the Telit modem including 3G, GPS, AD, IO and Python 2.7. This is used for a remote data aquisition application. Python is fairy slow on these modules but we only need an update every 15 minutes or in an alarm condition so they work well.
http://www.gatetel.com/#!gt-series/cscb

Blockquote
The only real problem I have in day to day work is that the last of static checking vs a regular compiler can cause problems to be thrown at run-time, e.g. a simple accidental cat of a string and an int in a print statement can bring the whole application down
To me it is a huge deal. Problems you could find at compile time and fix the problem now have to rely at run time. Not knowing the data type and having to write additional function just to check the datatype is hassle. There is no need to do that in C. How would you declare 'volatile' in python?
Blockquote
The only downsides for embedded work is that sometimes python can be slow and sometimes it uses a lot of memory (relatively speaking). This hasn't causes us a show stopping problem yet, and python is quite easy to profile for both speed and memory if it becomes a problem.
This is also huge. For Embedded sytems or RTOS time constraint is very important.
Python is not necessary quick to code. It really depends what language you are comfortable with. Honestly it takes me 1 day to write function and unnecessary object orientation stuff which I can do in 2 hours in C.
Testing is so inconvenient I have to write the code, py_compile, copy pyc in the target then run the program, then python quits complaining variable not defined or type cast error or some petty thing like that.
My suggestion is C toolchain is available for any target. C is fast, hardware oriented,challenging and fun. Stick with C for Embedded systems. No need to install configure silly python packages just to run it.

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.