Develop Reference

Python exit with error: Use sysexits.h or errno.h codes?

I'm trying to make my code play nicely with the other children on Linux systems, and I want to use exit codes that make sense to the shell.
In the specific case I'm starting with, when attempting to write to a directory that does not exist, I find two candidate codes:
#define EX_CANTCREAT 73 /* can't create (user) output file */
in sysexits.h and (effectively):
errno.ENOENT = 2 # No such file or directory
in Python's errno module.
Is one more appropriate than the other for a sys.exit()?
I note that Python provides errno and there is a corresponding errno.h but I'm not seeing anything Pythonic prebuilt for sysexits.h...
EDIT: I was trying to determine if I should be using more specific exit codes and if so, from which set of predefined codes. The answers below give a pretty definite "No": Stick to 0 for success and 1 for failure as exit codes. Use the errno codes for stderr messages, and, for Linux at least, just stay away from sysexits.h... I think.

As a rule (specifically, precedent set by UNIX), a program only uses a few exit codes: 0 for success; optionally, one for each other valid outcome that is not an error (e.g. for grep -- if no matches were found); and one for any error. The code for "any error" is usually the highest. Some programs use special codes for some errors, but these are few and far between.
(A number of exit codes (in UNIX, 128+, in Windows, a few NTSTATUS values that look like large negative numbers) are used if the OS terminates the process -- it's thus not useful to generate them in the program, to avoid confusion.)
Any more specific information about the error is supposed to be printed on stderr.
sysexits.h seems to be specific to BSD and OSX (which is derived from BSD) and is not a part of the POSIX standard.
The motivation for a "catch-all" error exit code is that without additional information, the receiving process cannot do anything intelligent about the error anyway. stderr is a much better way to pass such information.

Python program still running but PID can't be found

I am running a detached child program in background from my parent program. After i exited the parent program, i would expect the child program to continuing running and logging into OUTPUT_PATH. And indeed, i can see the log file updating. However as i was trying to find the PID from ps aux i can't find it. can anyone explain this behavior? what am i doing wrong?
shellCommand = "nohup python PYTHON_PROGRAM ARGS >OUTPUT_PATH 2>&1 &"
subprocess.Popen(shellCommand, shell=True, preexec_fn=os.setpgrp)

OK, this is getting too big for comments. By running ps -fwp $(pgrep -f PYTHON_PROGRAM), we've found the process now. :) But its PID does not match the one reported by Popen.pid. This would be down to shell instance that was called since you've used shell=True. First fork was to call shell, the second was for your script. Actually, this is documented in the link mention above:
Note that if you set the shell argument to True, this is the process ID of the spawned shell.
But see the NOTE bellow.
Which brings us to the "more orthodox way". Where we're entering possibly contested territory, different people, different ideas. Not as much the first one perhaps as it would be in line with the documentation to suggest not to use shell=True unless you really need to.
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names). If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.
There is also another section on (security) implications of not heeding the recommendation.
So, compiling a list of arguments to run nohup with your script and handling the output redirection already through keyword arguments (stdout, stderr) of Popen would seem like a good course of action and would also get you a consistent PID.
This last step might attract most controversy: but you can actually daemonize process by means of python interfaces to corresponding syscalls. Well documented example would seem to grow in github (reached over one hop from a link in the PEP mentioned bellow).
or there is a library referred to from the PEP-3143 on the topic.
NOTE: That bit does not appear to be always true (calling of sh yes, but two PIDs no). At least on my system I've observed sh to exec the program called via -c (in itself) without forking. From few quick runs and traces this was the case at least if I did not mess with the stdin/-out/-err (i.e. no pipes or redirections), did not force subshell (...), or did not chain commands over ;. (The latter two are kind of obvious, former is as well once you realize how redirections are implemented). So at least for my shell I would dare to extrapolate and say that: It seems it does not fork, unless it must. Or even more simplified (and hence not entirely correct) statement would be: simple stuff won't fork.

Advantages of subprocess over os.system

I have recently came across a few posts on stack overflow saying that subprocess is much better than os.system, however I am having difficulty finding the exact advantages.
Some examples of things I have run into:
https://docs.python.org/3/library/os.html#os.system
"The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function."
No idea in what ways it is more powerful though, I know it is easier in many ways to use subprocess but is it actually more powerful in some way?
Another example is:
https://stackoverflow.com/a/89243/3339122
The advantage of subprocess vs system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).
This post which has 2600+ votes. Again could not find any elaboration on what was meant by better error handling or real status code.
Top comment on that post is:
Can't see why you'd use os.system even for quick/dirty/one-time. subprocess seems so much better.
Again, I understand it makes some things slightly easier, but I hardly can understand why for example:
subprocess.call("netsh interface set interface \"Wi-Fi\" enable", shell=True)
is any better than
os.system("netsh interface set interface \"Wi-Fi\" enabled")
Can anyone explain some reasons it is so much better?

First of all, you are cutting out the middleman; subprocess.call by default avoids spawning a shell that examines your command, and directly spawns the requested process. This is important because, besides the efficiency side of the matter, you don't have much control over the default shell behavior, and it actually typically works against you regarding escaping.
In particular, do not do this:
subprocess.call('netsh interface set interface "Wi-Fi" enable')
since
If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.
Instead, you'll do:
subprocess.call(["netsh", "interface", "set", "interface", "Wi-Fi", "enable"])
Notice that here all the escaping nightmares are gone. subprocess handles escaping (if the OS wants arguments as a single string - such as Windows) or passes the separated arguments straight to the relevant syscall (execvp on UNIX).
Compare this with having to handle the escaping yourself, especially in a cross-platform way (cmd doesn't escape in the same way as POSIX sh), especially with the shell in the middle messing with your stuff (trust me, you don't want to know what unholy mess is to provide a 100% safe escaping for your command when calling cmd /k).
Also, when using subprocess without the shell in the middle you are sure you are getting correct return codes. If there's a failure launching the process you get a Python exception, if you get a return code it's actually the return code of the launched program. With os.system you have no way to know if the return code you get comes from the launched command (which is generally the default behavior if the shell manages to launch it) or it is some error from the shell (if it didn't manage to launch it).
Besides arguments splitting/escaping and return code, you have way better control over the launched process. Even with subprocess.call (which is the most basic utility function over subprocess functionalities) you can redirect stdin, stdout and stderr, possibly communicating with the launched process. check_call is similar and it avoids the risk of ignoring a failure exit code. check_output covers the common use case of check_call + capturing all the program output into a string variable.
Once you get past call & friends (which is blocking just as os.system), there are way more powerful functionalities - in particular, the Popen object allows you to work with the launched process asynchronously. You can start it, possibly talk with it through the redirected streams, check if it is running from time to time while doing other stuff, waiting for it to complete, sending signals to it and killing it - all stuff that is way besides the mere synchronous "start process with default stdin/stdout/stderr through the shell and wait it to finish" that os.system provides.
So, to sum it up, with subprocess:
even at the most basic level (call & friends), you:
avoid escaping problems by passing a Python list of arguments;
avoid the shell messing with your command line;
either you have an exception or the true exit code of the process you launched; no confusion about program/shell exit code;
have the possibility to capture stdout and in general redirect the standard streams;
when you use Popen:
you aren't restricted to a synchronous interface, but you can actually do other stuff while the subprocess run;
you can control the subprocess (check if it is running, communicate with it, kill it).
Given that subprocess does way more than os.system can do - and in a safer, more flexible (if you need it) way - there's just no reason to use system instead.

There are many reasons, but the main reason is mentioned directly in the docstring:
>>> os.system.__doc__
'Execute the command in a subshell.'
For almost all cases where you need a subprocess, it is undesirable to spawn a subshell. This is unnecessary and wasteful, it adds an extra layer of complexity, and introduces several new vulnerabilities and failure modes. Using subprocess module cuts out the middleman.

Python: why is this print automatically flushing to screen without flush()?

I was reading about std.flush() in python. And I found this example a lot.
import sys,time
for i in range(10):
print i,
#sys.stdout.flush()
time.sleep(1)
It is often said that it makes a difference with/without the "sys.stdout.flush()".
However, when I called this script from command prompt, it didn't make a difference in my case. Both printed numbers to the screen in real time.
I used python 2.7.5 in windows.
Why is that happening?
p.s. In another example which printed the output through subprocess.PIPE instead of to the screen directly, I did observe a difference of the buffering.
What am I missing?

Using flush will generally guarantee that flushing is done but assuming the reverse relationship is a logical fallacy, akin to:
Dogs are animals.
This is an animal.
Therefore this is a dog.
In other words, not using flush does not guarantee flushing will not happen.
Interestingly enough, using Python 2.7.8 under Cygwin in Win81, I see the opposite behaviour - everything is batched up until the end. It may be different with Windows-native Python, it may also be different from within IDLE.

See stdio buffering. In brief:
Default Buffering modes:
stdin is always buffered
stderr is always unbuffered
if stdout is a terminal then buffering is automatically set to line buffered, else it is set to buffered
For me, the example you gave prints:
In cmd:
all the numbers upon exit in Cygwin's python
one by one in Win32 python
In mintty:
both all upon exit
both one by one with -u option
sys.stdout.isatty() returns False!
So, it looks like msvcrt's stdout is unbuffered when it points to a terminal. A test with a simple C program shows the same behaviour.

Python subprocess stderr redirection, redirect warnings, still raise on control flow stoppers?

I did basic due diligence and couldn't find a good answer to this anywhere.
I want to call subprocess.Popen in a way that they will still raise a Python exception when control flow is interrupted, but I want to redirect stderr at the same time.
The use case is for a command line client that shouldn't output warnings, but whose internal logic still wants to know about subprocess problems.
As an example, this silently redirects all errors:
subprocess.Popen(command, stderr=subprocess.PIPE)
Calling a Python module as a subprocess with contents:
raise(Exception("AVAST!"))
Doesn't raise anything.
It would be great if it redirected all error text, but still raised on anything that halted control flow prematurely. Do I need to use return code and hope that all subprocesses I call have implemented this correctly?
The best thing I've thought of so far is manually parsing the redirected errors, which is a pretty poor implementation in my mind.
Is there a clean canonical way to do this?

There's no way to pass exceptions across a text pipe like stderr, because all you can pass across a text pipe is text.
But there are a few options:
Make sure all of your children exit with non-zero status on exception (which is the default, if you don't do anything) and don't do so in any other cases (unless there are cases you want to treat the same as an exception).
Parse for exceptions in stderr.
Create some other communication channel for the parent and children to share.
Don't use subprocess. For example, if the only reason you're running Python scripts via subprocess is to get core parallelism (or memory space isolation), it may be a lot easier to use multiprocessing or concurrent.futures, which have already built the machinery to propagate exceptions for you.
From your comment:
My use case is calling a bunch of non-Python third party things. If return codes are how the standard library module propagates errors, I'm happy enough using them.
No, the Python standard library propagates errors using Exceptions. And so should you.
Return codes are how non-Python third party things propagate errors. (Actually, how they propagate both errors and unexpected signals, but… don't worry about that.) That's limited to 7 bits worth of data, and the meanings aren't really standardized, so it's not as good. But it's all you have in the POSIX child process model, so that's what generic programs use.
What you probably want to do is the same thing subprocess.check_call does—if the return code is not zero, raise an exception. (In fact, if you're not doing anything asynchronous, ask yourself whether you can use check_call in the first place, instead of using a Popen object explicitly.)
For example, if you were doing this:
output, errors = p.communicate()
Change it to this:
output, errors = p.communicate()
if p.returncode:
raise subprocess.CalledProcessError(p.returncode, 'description')
(The description is usually the subprocess path or name.)
Then, the rest of your code can just handle exceptions.