First of all, I read as many related questions to subprocess.check_output() as I could find, but still struggle to identify the problem.
If I execute kill -l 1 in the shell, I get the corresponding signal name for 1, which is HUP. I need the same behaviour in my python script, so I use:
>>> subprocess.check_output(['kill', '-l', '1'])
b'HUP INT QUIT ILL TRAP ABRT BUS FPE KILL USR1 SEGV USR2 PIPE ALRM TERM STKFLT\nCHLD CONT STOP TSTP TTIN TTOU URG XCPU XFSZ VTALRM PROF WINCH POLL PWR SYS\n'
The subprocess seems to ignore the '1' in the argument list and instead executes kill -l.
I tried different versions, the argument as a list or string, with shell optione True and False, but none seem to work.
Any ideas what could be the reason? Using python3.4 on a Ubuntu14.04.
Thanks!
Possible cause: The kill command in your shell is executing a shell built-in (most shells have one, because you need to be able to kill without a process launch when you have runaway fork bombs and the like), whereas check_output (not executing within a shell by default) is running the kill executable found in your PATH (often /bin/kill, but not always, running type -P kill in bash will tell you where that executable is).
Odds are, the built-in supports the switches you're looking for, the executable does not. In bash, try running type -P kill, then explicitly running the /full/path/to/kill -l 1 to see if the kill check_output is finding actually supports that invocation. Often there are subtle differences between different implementations of kill.
The best solution to this is probably to avoid expensive and pointless subprocess launches and check the Python definitions for the signals. For example, in Python 3.5, it's trivial to construct a mapping from the signals known by Python to their Python names:
import signal
sigdict = {sig.value: sig.name for sig in signal.Signals}
print(sigdict[1])
CTRL_BREAK_EVENT # <-- The output on my Windows box. On your machine, it would probably be SIGHUP
In older Python where the names aren't enums, you can use similar code using dir of the module, filtering for names whose __module__ is signal and whose values are integers to construct the mapping.
Related
I have a Python 3.9 script that starts another process in a new console. The 'other process' keeps running even after the original one has completed.
This is what I have on Windows:
# startup.py script
# =================
import sys, subprocess
if __name__ == '__main__':
print('start startup.py script')
arguments = ['python', 'other_process.py']
arguments.extend(sys.argv[1:])
subprocess.Popen(
arguments,
creationflags = subprocess.CREATE_NEW_CONSOLE,
)
print('end startup.py script')
It works great. Below you see the original console on the left, in which I invoke startup.py. I also pass it a --help flag, which is then simply passed to the other_process.py script.
On the right, you see the other_process.py script running. Please note that the original startup.py script has already finished, while the other_process.py script is still running. That's exactly what I need:
The subprocess.CREATE_NEW_CONSOLE parameter doesn't work on Linux. I've heard that setting shell=True would have a similar effect, but it doesn't spawn a new console.
How can I get the same effect on Linux?
Unix doesn’t provide this option/service, but you can run a terminal emulator:
subprocess.Popen(["gnome-terminal","--"]+arguments)
There isn’t a standard means of finding which terminal emulator to use (or even which are available), unfortunately. Checking shutil.which for a few common ones might be the right idea; from Wikipedia’s list, I’d recommend gnome-terminal, konsole, and xterm. You still then have to deal with the slightly different syntax to run a command in each.
I am making a virtual assistant that can start several programs using subprocess.Popen("path/to/app.exe"). But when I exit the python program, all of processes are killed. I want the processes (the applications started with Popen) to be independent and remain alive after main process is killed.
I have tried adding start_new_session=True as argument in subprocess.Popen() as some posts have suggested, but it's still not working.
I don't think showing the code is necessary, but still, here you go.
app_path = r'C:\Users\myusername\AppData\Local\Discord\app-1.0.9001\discord.exe'
subprocess.Popen(app_path) # also tried adding start_new_session=True as argument
Since you're on Windows, you can call the start command, which exists for this very purpose: to run another program independently of the one that starts it.
The start command is provided by the command-line interpreter cmd.exe. It is not an executable: there is no start.exe. It is a "shell command" (in Linux terminology), which is why shell=True must be passed when creating the subprocess.
You won't be able to communicate with the subprocess started in this way, that is, not via the pipe mechanism provided by the subprocess module. So instead of Popen, you may just use the convenience function run:
from subprocess import run
app = 'notepad'
run(['start', app], shell=True)
The example starts the Notepad editor (instead of Discord in the question) in order to make it easier to reproduce.
In cases where the full path to the app contains spaces, we can either call start like so
app = r'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe'
run(f'start "" "{app}"', shell=True)
using the Edge browser in this example, or pass the directory separately:
folder = r'C:\Program Files (x86)\Microsoft\Edge\Application'
app = 'msedge.exe'
run(['start', '/d', folder, app], shell=True)
This is needed because start treats a single argument as the window title if that argument is in quotes. And only if not does it treat it as the command. See "Can I use the start command with spaces in the path?" (on SuperUser) for more details.
Answered here: https://stackoverflow.com/a/34718600/4355695
subprocess.Popen(full_command, shell=True, close_fds=True)
(In my linux system I have to put shell=True if I'm passing a full command as string instead of a split-up array of arguments. In windows it may differ, idk)
I am using Python's subprocess module to call some Linux command line functions. The documentation explains the shell=True argument as
If shell is True, the specified command will be executed through the shell
There are two examples, which seem the same to me from a descriptive viewpoint (i.e. both of them call some command-line command), but one of them uses shell=True and the other does not
>>> subprocess.call(["ls", "-l"])
0
>>> subprocess.call("exit 1", shell=True)
1
My question is:
What does running the command with shell=False do, in contrast to shell=True?
I was under the impression that subprocess.call and check_call and check_output all must execute the argument through the shell. In other words, how can it possibly not execute the argument through the shell?
It would also be helpful to get some examples of:
Things that can be done with shell=True that can't be done with
shell=False and why they can't be done.
Vice versa (although it seems that there are no such examples)
Things for which it does not matter whether shell=True or False and why it doesn't matter
UNIX programs start each other with the following three calls, or derivatives/equivalents thereto:
fork() - Create a new copy of yourself.
exec() - Replace yourself with a different program (do this if you're the copy!).
wait() - Wait for another process to finish (optional, if not running in background).
Thus, with shell=False, you do just that (as Python-syntax pseudocode below -- exclude the wait() if not a blocking invocation such as subprocess.call()):
pid = fork()
if pid == 0: # we're the child process, not the parent
execlp("ls", "ls", "-l", NUL);
else:
retval = wait(pid) # we're the parent; wait for the child to exit & get its exit status
whereas with shell=True, you do this:
pid = fork()
if pid == 0:
execlp("sh", "sh", "-c", "ls -l", NUL);
else:
retval = wait(pid)
Note that with shell=False, the command we executed was ls, whereas with shell=True, the command we executed was sh.
That is to say:
subprocess.Popen(foo, shell=True)
is exactly the same as:
subprocess.Popen(
["sh", "-c"] + ([foo] if isinstance(foo, basestring) else foo),
shell=False)
That is to say, you execute a copy of /bin/sh, and direct that copy of /bin/sh to parse the string into an argument list and execute ls -l itself.
So, why would you use shell=True?
You're invoking a shell builtin.
For instance, the exit command is actually part of the shell itself, rather than an external command. That said, this is a fairly small set of commands, and it's rare for them to be useful in the context of a shell instance that only exists for the duration of a single subprocess.call() invocation.
You have some code with shell constructs (ie. redirections) that would be difficult to emulate without it.
If, for instance, your command is cat one two >three, the syntax >three is a redirection: It's not an argument to cat, but an instruction to the shell to set stdout=open('three', 'w') when running the command ['cat', 'one', 'two']. If you don't want to deal with redirections and pipelines yourself, you need a shell to do it.
A slightly trickier case is cat foo bar | baz. To do that without a shell, you need to start both sides of the pipeline yourself: p1 = Popen(['cat', 'foo', 'bar'], stdout=PIPE), p2=Popen(['baz'], stdin=p1.stdout).
You don't give a damn about security bugs.
...okay, that's a little bit too strong, but not by much. Using shell=True is dangerous. You can't do this: Popen('cat -- %s' % (filename,), shell=True) without a shell injection vulnerability: If your code were ever invoked with a filename containing $(rm -rf ~), you'd have a very bad day. On the other hand, ['cat', '--', filename] is safe with all possible filenames: The filename is purely data, not parsed as source code by a shell or anything else.
It is possible to write safe scripts in shell, but you need to be careful about it. Consider the following:
filenames = ['file1', 'file2'] # these can be user-provided
subprocess.Popen(['cat -- "$#" | baz', '_'] + filenames, shell=True)
That code is safe (well -- as safe as letting a user read any file they want ever is), because it's passing your filenames out-of-band from your script code -- but it's safe only because the string being passed to the shell is fixed and hardcoded, and the parameterized content is external variables (the filenames list). And even then, it's "safe" only to a point -- a bug like Shellshock that triggers on shell initialization would impact it as much as anything else.
I was under the impression that subprocess.call and check_call and check_output all must execute the argument through the shell.
No, subprocess is perfectly capable of starting a program directly (via an operating system call). It does not need a shell
Things that can be done with shell=True that can't be done with shell=False
You can use shell=False for any command that simply runs some executable optionally with some specified arguments.
You must use shell=True if your command uses shell features. This includes pipelines, |, or redirections or that contains compound statements combined with ; or && or || etc.
Thus, one can use shell=False for a command like grep string file. But, a command like grep string file | xargs something will, because of the | require shell=True.
Because the shell has power features that python programmers do not always find intuitive, it is considered better practice to use shell=False unless you really truly need the shell feature. As an example, pipelines are not really truly needed because they can also be done using subprocess' PIPE feature.
Let's say I want to read RAM usage from /proc/meminfo. There are two basic ways to do this that I can think of.
Use a shell command
output = subprocess.check_output('cat /proc/meminfo', shell=True)
# or output = subprocess.check_output(['cat', '/proc/meminfo'])
lines = output.splitlines()
Use open()
with open('/proc/meminfo') as meminfo:
output = meminfo.read()
lines = output.splitlines()
My question is what is the difference between the two methods? Is there a significant performance difference? My assumption is that using open() is the preferred method, since using a shell command is a bit hackish and may be system dependent, but I can't find any information on this so I thought I'd ask.
...so, let's look at what output = subprocess.check_output('cat /proc/meminfo', shell=True) does:
Creates a FIFO pair with mkfifo(), and spawns a shell running sh -c 'cat /proc/meminfo' writing to the input end of the FIFO (while the Python interpreter itself watches for output on the other end, either using the select() call or blocking IO operations). This means opening /bin/sh, opening all the libraries it depends on, etc.
The shell parses those arguments as code. This can be dangerous if, instead of opening /proc/meminfo. you're instead opening /tmp/$(rm -rf ~)/pwned.txt.
The shell forks a subprocess (optionally; shells may have an implicit exec), which then uses the execve system call to invoke /bin/cat with an argv of ['cat', '/proc/meminfo'] -- meaning that /bin/cat is again loaded as an executable, with its dynamic libraries, with all the performance overhead that implies.
/bin/cat then opens /proc/meminfo, reads from it, and writes to its stdout
The shell, if it did not use the implicit-exec optimization, waits for the /bin/cat executable to finish and exit using a wait()-family syscall.
The Python interpreter reads from the remote end of the FIFO until it provides an EOF (which will not happen until after cat has closed its output pipeline, potentially by exiting), and then uses a wait()-family call to retrieve information on how the shell it spawned exited, checking that exit status to determine whether an error occurred.
Now, let's look at what open('/proc/meminfo').read() does:
Opens the file using the open() syscall.
Reads the file using the read() syscall.
Drops the reference count on the file, allowing it to be closed (either immediately or on a future garbage collection pass) with the close() syscall.
One of these things is much, much, much more efficient and generally sensible than the other.
I am testing sorting algorithms and therefore I would like to compine in my Python code, the linux command "time", because it takes some interesting arguments and for example the call of quicksort.
from subprocess import Popen
import quicksort
import rand
time=Popen("time quicksort.main(rand.main())")
This is tottaly wrong, but it is the closest I managed to get. I haven't grasped the idea of subprocess class, is it possible to combine method calls with linux commands, or only add commands in python like "grep..." and send the output to a variable??
If you use Popen from subprocess you need to do a lot of things differently.
I believe what you are looking for is check_output, another function belonging to the subprocess module.
But in order to further your understanding, since you are sort-of close, here is what you need to change to get it to work:
The command string "time quicksort.main(rand.main())" is not going to mean anything to bash. That is python. BUT in the case that it was valid bash language, it would need to be split on word boundaries (like bash WOULD normally do) so you would make it into a list:
['time', '...','...']
The only time you can pass Popen a command STRING (not a list) is when you set shell=True in the keywords to Popen.
But let's just leave shell at False, do some word-splitting for bash, and pass in a list. On to the next part.
Popen returns something you can communicate to/at/with. Not the result of the process' stdout. Use subprocess.PIPE for stdin and stdout keywords to Popen.
Once you have made a Popen object as described, you can call it's communicate method.
The result is two things, stdout and stderr.
You're after the first one. One use case for Popen is for when you need to keep errors and output seperate. Obviously this isn't turning out to be the best option for inline but oh well. Lets deal with stdout.
sdtout will probably need to be decoded:
stdout.decode()
or perhaps even have newlines stripped as well:
stdout.decode().rstrip()
So as you can see, Popen does not fit the use case you have in mind. There is no need to use subprocess and make system calls in order to time python. Look into timeit.