I have recently came across a few posts on stack overflow saying that subprocess is much better than os.system, however I am having difficulty finding the exact advantages.
Some examples of things I have run into:
https://docs.python.org/3/library/os.html#os.system
"The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function."
No idea in what ways it is more powerful though, I know it is easier in many ways to use subprocess but is it actually more powerful in some way?
Another example is:
https://stackoverflow.com/a/89243/3339122
The advantage of subprocess vs system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).
This post which has 2600+ votes. Again could not find any elaboration on what was meant by better error handling or real status code.
Top comment on that post is:
Can't see why you'd use os.system even for quick/dirty/one-time. subprocess seems so much better.
Again, I understand it makes some things slightly easier, but I hardly can understand why for example:
subprocess.call("netsh interface set interface \"Wi-Fi\" enable", shell=True)
is any better than
os.system("netsh interface set interface \"Wi-Fi\" enabled")
Can anyone explain some reasons it is so much better?
First of all, you are cutting out the middleman; subprocess.call by default avoids spawning a shell that examines your command, and directly spawns the requested process. This is important because, besides the efficiency side of the matter, you don't have much control over the default shell behavior, and it actually typically works against you regarding escaping.
In particular, do not do this:
subprocess.call('netsh interface set interface "Wi-Fi" enable')
since
If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.
Instead, you'll do:
subprocess.call(["netsh", "interface", "set", "interface", "Wi-Fi", "enable"])
Notice that here all the escaping nightmares are gone. subprocess handles escaping (if the OS wants arguments as a single string - such as Windows) or passes the separated arguments straight to the relevant syscall (execvp on UNIX).
Compare this with having to handle the escaping yourself, especially in a cross-platform way (cmd doesn't escape in the same way as POSIX sh), especially with the shell in the middle messing with your stuff (trust me, you don't want to know what unholy mess is to provide a 100% safe escaping for your command when calling cmd /k).
Also, when using subprocess without the shell in the middle you are sure you are getting correct return codes. If there's a failure launching the process you get a Python exception, if you get a return code it's actually the return code of the launched program. With os.system you have no way to know if the return code you get comes from the launched command (which is generally the default behavior if the shell manages to launch it) or it is some error from the shell (if it didn't manage to launch it).
Besides arguments splitting/escaping and return code, you have way better control over the launched process. Even with subprocess.call (which is the most basic utility function over subprocess functionalities) you can redirect stdin, stdout and stderr, possibly communicating with the launched process. check_call is similar and it avoids the risk of ignoring a failure exit code. check_output covers the common use case of check_call + capturing all the program output into a string variable.
Once you get past call & friends (which is blocking just as os.system), there are way more powerful functionalities - in particular, the Popen object allows you to work with the launched process asynchronously. You can start it, possibly talk with it through the redirected streams, check if it is running from time to time while doing other stuff, waiting for it to complete, sending signals to it and killing it - all stuff that is way besides the mere synchronous "start process with default stdin/stdout/stderr through the shell and wait it to finish" that os.system provides.
So, to sum it up, with subprocess:
even at the most basic level (call & friends), you:
avoid escaping problems by passing a Python list of arguments;
avoid the shell messing with your command line;
either you have an exception or the true exit code of the process you launched; no confusion about program/shell exit code;
have the possibility to capture stdout and in general redirect the standard streams;
when you use Popen:
you aren't restricted to a synchronous interface, but you can actually do other stuff while the subprocess run;
you can control the subprocess (check if it is running, communicate with it, kill it).
Given that subprocess does way more than os.system can do - and in a safer, more flexible (if you need it) way - there's just no reason to use system instead.
There are many reasons, but the main reason is mentioned directly in the docstring:
>>> os.system.__doc__
'Execute the command in a subshell.'
For almost all cases where you need a subprocess, it is undesirable to spawn a subshell. This is unnecessary and wasteful, it adds an extra layer of complexity, and introduces several new vulnerabilities and failure modes. Using subprocess module cuts out the middleman.
Related
I've been using Google Colab for a few weeks now and I've been wondering what the difference is between the two following commands (for example):
!ffmpeg ...
subprocess.Popen(['ffmpeg', ...
I was wondering because I ran into some issues when I started either of the commands above and then tried to stop execution midway. Both of them cancel on KeyboardInterrupt but I noticed that after that the runtime needs a factory reset because it somehow got stuck. Checking ps aux in the Linux console listed a process [ffmpeg] <defunct> which somehow still was running or at least blocking some ressources as it seemed.
I then did some research and came across some similar posts asking questions on how to terminate a subprocess correctly (1, 2, 3). Based on those posts I generally came to the conclusion that using the subprocess.Popen(..) variant obviously provides more flexibility when it comes to handling the subprocess: Defining different stdout procedures or reacting to different returncode etc. But I'm still unsure on what the first command above using the ! as prefix exactly does under the hood.
Using the first command is much easier and requires way less code to start this process. And assuming I don't need a lot of logic handling the process flow it would be a nice way to execute something like ffmpeg - if I were able to terminate it as expected. Even following the answers from the other posts using the 2nd command never got me to a point where I could terminate the process fully once started (even when using shell=False, process.kill() or process.wait() etc.). This got me frustrated, because restarting and re-initializing the Colab instance itself can take several minutes every time.
So, finally, I'd like to understand in more general terms what the difference is and was hoping that someone could enlighten me. Thanks!
! commands are executed by the notebook (or more specifically by the ipython interpreter), and are not valid Python commands. If the code you are writing needs to work outside of the notebook environment, you cannot use ! commands.
As you correctly note, you are unable to interact with the subprocess you launch via !; so it's also less flexible than an explicit subprocess call, though similar in this regard to subprocess.call
Like the documentation mentions, you should generally avoid the bare subprocess.Popen unless you specifically need the detailed flexibility it offers, at the price of having to duplicate the higher-level functionality which subprocess.run et al. already implement. The code to run a command and wait for it to finish is simply
subprocess.check_call(['ffmpeg', ... ])
with variations for capturing its output (check_output) and the more modern run which can easily replace all three of the legacy high-level calls, albeit with some added verbosity.
I am using a 3rd-party python module which is normally called through terminal commands. When called through terminal commands it has a verbose option which prints to terminal in real time.
I then have another python program which calls the 3rd-party program through subprocess. Unfortunately, when called through subprocess the terminal output no longer flushes, and is only returned on completion (the process takes many hours so I would like real-time progress).
I can see the source code of the 3rd-party module and it does not set printing to be flushed such as print('example', flush=True). Is there a way to force the flushing through my module without editing the 3rd-party source code? Furthermore, can I send this output to a log file (again in real time)?
Thanks for any help.
The issue is most likely that many programs work differently if run interactively in a terminal or as part of a pipe line (i.e. called using subprocess). It has very little to do with Python itself, but more with the Unix/Linux architecture.
As you have noted, it is possible to force a program to flush stdout even when run in a pipe line, but it requires changes to the source code, by manually applying stdout.flush calls.
Another way to print to screen, is to "trick" the program to think it is working with an interactive terminal, using a so called pseudo-terminal. There is a supporting module for this in the Python standard library, namely pty. Using, that, you will not explicitly call subprocess.run (or Popen or ...). Instead you have to use the pty.spawn call:
def prout(fd):
data = os.read(fd, 1024)
while(data):
print(data.decode(), end="")
data = os.read(fd, 1024)
pty.spawn("./callee.py", prout)
As can be seen, this requires a special function for handling stdout. Here above, I just print it to the terminal, but of course it is possible to do other thing with the text as well (such as log or parse...)
Another way to trick the program, is to use an external program, called unbuffer. Unbuffer will take your script as input, and make the program think (as for the pty call) that is called from a terminal. This is arguably simpler if unbuffer is installed or you are allowed to install it on your system (it is part of the expect package). All you have to do then, is to change your subprocess call as
p=subprocess.Popen(["unbuffer", "./callee.py"], stdout=subprocess.PIPE)
and then of course handle the output as usual, e.g. with some code like
for line in p.stdout:
print(line.decode(), end="")
print(p.communicate()[0].decode(), end="")
or similar. But this last part I think you have already covered, as you seem to be doing something with the output.
I am trying to open chromium to an authorization URL with a specific launch parameter. I was not able to find the solution to this launch parameter using the webbrowser library, so I moved onto os.system
browser_cmd = "chromium-browser --password-store=basic " + auth_url
os.system(browser_cmd)
This works up until the "&" in the URL. So chromium opens without bothering me with keyring nonsense, but only opens the URL until the first &. Is there a way of handling the URL and maintaining its integrity?
This is because & is special to the shell. The canonical way to run a subprocess from within Python 3 is:
import subprocess
subprocess.run(['chromium-browser', '--password-store=basic', auth_url],
check=True)
print('chromium-browser exited successfully')
This is exactly why you shouldn't use os.system for anything non-trivial. As the docs for that function say:
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.
If you follow that link, you'll see that you can write this as:
subprocess.call(['chromium-browser', '--password-store=basic', auth_url])
Because you're passing a list of arguments, rather than trying to put them together into a string that you can smuggle through the shell, you don't have to worry about quoting, or anything else.
By the way, you probably want to use run rather than call here, but for some reason the recipes still haven't been updated in the docs as of 3.7, and I didn't want to add confusion by showing something that doesn't match… Anyway, you should read the at least the quick-start "Using" section at the top of the docs.
If you really want to use os.system anyway for some reason, you will need to quote and/or escape the auth_url argument. Assuming you don't care about Windows, the best way to do this is with the shlex module:
browser_cmd = "chromium-browser --password-store=basic " + shlex.quote(auth_url)
os.system(browser_cmd)
If you do care about Windows, you can add posix=False to the quote call. The effects of the posix flag are documented under the shlex constructor. The default True value means it follows POSIX rules as closely as possible, which means it should be able to handle anything that could possibly be handled, as long as your shell is strictly compatible with sh (as, e.g., bash is, but Windows cmd is definitely not, and even tcsh or fish may not be). With False, it uses "compatibility mode". For simple examples like yours, it should work for most shells without fiddling, but if you need to get more complicated, you need to read Improved Compatibility with Shells. (And, for Windows cmd or PowerShell, there's a limit to how far you can push things.)
I am running a detached child program in background from my parent program. After i exited the parent program, i would expect the child program to continuing running and logging into OUTPUT_PATH. And indeed, i can see the log file updating. However as i was trying to find the PID from ps aux i can't find it. can anyone explain this behavior? what am i doing wrong?
shellCommand = "nohup python PYTHON_PROGRAM ARGS >OUTPUT_PATH 2>&1 &"
subprocess.Popen(shellCommand, shell=True, preexec_fn=os.setpgrp)
OK, this is getting too big for comments. By running ps -fwp $(pgrep -f PYTHON_PROGRAM), we've found the process now. :) But its PID does not match the one reported by Popen.pid. This would be down to shell instance that was called since you've used shell=True. First fork was to call shell, the second was for your script. Actually, this is documented in the link mention above:
Note that if you set the shell argument to True, this is the process ID of the spawned shell.
But see the NOTE bellow.
Which brings us to the "more orthodox way". Where we're entering possibly contested territory, different people, different ideas. Not as much the first one perhaps as it would be in line with the documentation to suggest not to use shell=True unless you really need to.
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names). If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.
There is also another section on (security) implications of not heeding the recommendation.
So, compiling a list of arguments to run nohup with your script and handling the output redirection already through keyword arguments (stdout, stderr) of Popen would seem like a good course of action and would also get you a consistent PID.
This last step might attract most controversy: but you can actually daemonize process by means of python interfaces to corresponding syscalls. Well documented example would seem to grow in github (reached over one hop from a link in the PEP mentioned bellow).
or there is a library referred to from the PEP-3143 on the topic.
NOTE: That bit does not appear to be always true (calling of sh yes, but two PIDs no). At least on my system I've observed sh to exec the program called via -c (in itself) without forking. From few quick runs and traces this was the case at least if I did not mess with the stdin/-out/-err (i.e. no pipes or redirections), did not force subshell (...), or did not chain commands over ;. (The latter two are kind of obvious, former is as well once you realize how redirections are implemented). So at least for my shell I would dare to extrapolate and say that: It seems it does not fork, unless it must. Or even more simplified (and hence not entirely correct) statement would be: simple stuff won't fork.
I did basic due diligence and couldn't find a good answer to this anywhere.
I want to call subprocess.Popen in a way that they will still raise a Python exception when control flow is interrupted, but I want to redirect stderr at the same time.
The use case is for a command line client that shouldn't output warnings, but whose internal logic still wants to know about subprocess problems.
As an example, this silently redirects all errors:
subprocess.Popen(command, stderr=subprocess.PIPE)
Calling a Python module as a subprocess with contents:
raise(Exception("AVAST!"))
Doesn't raise anything.
It would be great if it redirected all error text, but still raised on anything that halted control flow prematurely. Do I need to use return code and hope that all subprocesses I call have implemented this correctly?
The best thing I've thought of so far is manually parsing the redirected errors, which is a pretty poor implementation in my mind.
Is there a clean canonical way to do this?
There's no way to pass exceptions across a text pipe like stderr, because all you can pass across a text pipe is text.
But there are a few options:
Make sure all of your children exit with non-zero status on exception (which is the default, if you don't do anything) and don't do so in any other cases (unless there are cases you want to treat the same as an exception).
Parse for exceptions in stderr.
Create some other communication channel for the parent and children to share.
Don't use subprocess. For example, if the only reason you're running Python scripts via subprocess is to get core parallelism (or memory space isolation), it may be a lot easier to use multiprocessing or concurrent.futures, which have already built the machinery to propagate exceptions for you.
From your comment:
My use case is calling a bunch of non-Python third party things. If return codes are how the standard library module propagates errors, I'm happy enough using them.
No, the Python standard library propagates errors using Exceptions. And so should you.
Return codes are how non-Python third party things propagate errors. (Actually, how they propagate both errors and unexpected signals, but… don't worry about that.) That's limited to 7 bits worth of data, and the meanings aren't really standardized, so it's not as good. But it's all you have in the POSIX child process model, so that's what generic programs use.
What you probably want to do is the same thing subprocess.check_call does—if the return code is not zero, raise an exception. (In fact, if you're not doing anything asynchronous, ask yourself whether you can use check_call in the first place, instead of using a Popen object explicitly.)
For example, if you were doing this:
output, errors = p.communicate()
Change it to this:
output, errors = p.communicate()
if p.returncode:
raise subprocess.CalledProcessError(p.returncode, 'description')
(The description is usually the subprocess path or name.)
Then, the rest of your code can just handle exceptions.