Develop Reference

Using subprocess to run a Python script with shell=True

I have a Python script foo.py that I want to invoke with a second Python script using the subprocess module. The code in foo.py is provided by a user, and it could be malicious. I have set some security features in place (e.g., resource limits, change UID) prior to running the code to minimize the damage that the user can do.
I'd like to invoke the script with shell=True kwarg in the subprocess.run command because I am setting resource limits at the UID level. It seems like these limits do not apply if I do not add this kwarg. My code looks something like this:
import os
import subprocess
def enable_extra_security() -> None:
# set a different user ID for security reasons
os.setuid(1234)
# limit CPU time usage
resource.setrlimit(resource.RLIMIT_CPU, (5, 5))
# limit file size creation
resource.setrlimit(resource.RLIMIT_FSIZE, (100_000, 100_000))
# prevent forking
resource.setrlimit(resource.RLIMIT_NPROC, (5, 5))
if __name__ == '__main__':
# this script is run as root.
pid = os.fork()
if pid == 0:
# child process
enable_extra_security()
res = subprocess.run(['/usr/bin/python', 'foo.py'], shell=True, capture_output=True)
# ...
# child does some processing based on the returncode and output.
# ...
else:
# parent process
os.waitpid(pid, 0)
# ...
# ...
When I invoke enable_extra_security(), it only applies the additional security features to the child process. Specifically, the limits are only imposed on user ID 1234, which is only set for the child process. If I run subprocess.run without shell=True, the security features vanish. Unfortunately, the code above does not work: the subprocess.run command will actually just start a Python console and hang (i.e., it does not seem to pass in "foo.py" as an argument).
Interestingly, the code does work for C and C++ code (if I were to replace /usr/bin/python with /usr/bin/gcc and instead put a .c or .cpp file, it would work just fine). Why is this happening, and how can I solve my problem?

Run subprocess at high prio

I need to run a jar file from my python script that has to meet some realtime requirements (it's a programmer).
What I currently do is
import subprocess
process = subprocess.Popen(shlex.split("java programmer.jar"), stdout=subprocess.PIPE)
but the programmer stalls on a regular basis what doesn't happen if I start it by hand using start /high java programmer.jar. So I'd like to somehow also set the priority of the subprocess directly from my python script.
But if use the command above, I get a FileNotFoundError, so Google told me that I have to use the shell=True flag. That solved my problem but created a new one as I now have my jar running in another shell so I cannot parse the stdout anymore what I have to do to take some actions.
So is there a way to either run subprocess directly at a higher priority or to redirect the console output of the shell it creates when I run
process = subprocess.Popen(shlex.split("java programmer.jar"), stdout=subprocess.PIPE, shell=True)
This has to run on Windows

The kernel schedules a thread at one of 32 priority levels. The base priority depends on the thread priority relative to the process priority class:
IDLE_PRIORITY_CLASS = 0x00000040
BELOW_NORMAL_PRIORITY_CLASS = 0x00004000
NORMAL_PRIORITY_CLASS = 0x00000020
ABOVE_NORMAL_PRIORITY_CLASS = 0x00008000
HIGH_PRIORITY_CLASS = 0x00000080
REALTIME_PRIORITY_CLASS = 0x00000100
The realtime-priority class should only be used when absolutely necessary and only for short-lived operations. Even the high-priority class should be used sparingly and with care.
The initial priority class of a process can be set via the dwCreationFlags parameter of CreateProcessW. Python's subprocess.Popen supports this as its creationflags parameter. For convenience, the above priority-class constants were added in 3.7, but of course we can use these flag values in previous versions of Python.

Python subprocess.check_output() seems to ignore arguments

First of all, I read as many related questions to subprocess.check_output() as I could find, but still struggle to identify the problem.
If I execute kill -l 1 in the shell, I get the corresponding signal name for 1, which is HUP. I need the same behaviour in my python script, so I use:
>>> subprocess.check_output(['kill', '-l', '1'])
b'HUP INT QUIT ILL TRAP ABRT BUS FPE KILL USR1 SEGV USR2 PIPE ALRM TERM STKFLT\nCHLD CONT STOP TSTP TTIN TTOU URG XCPU XFSZ VTALRM PROF WINCH POLL PWR SYS\n'
The subprocess seems to ignore the '1' in the argument list and instead executes kill -l.
I tried different versions, the argument as a list or string, with shell optione True and False, but none seem to work.
Any ideas what could be the reason? Using python3.4 on a Ubuntu14.04.
Thanks!

Possible cause: The kill command in your shell is executing a shell built-in (most shells have one, because you need to be able to kill without a process launch when you have runaway fork bombs and the like), whereas check_output (not executing within a shell by default) is running the kill executable found in your PATH (often /bin/kill, but not always, running type -P kill in bash will tell you where that executable is).
Odds are, the built-in supports the switches you're looking for, the executable does not. In bash, try running type -P kill, then explicitly running the /full/path/to/kill -l 1 to see if the kill check_output is finding actually supports that invocation. Often there are subtle differences between different implementations of kill.
The best solution to this is probably to avoid expensive and pointless subprocess launches and check the Python definitions for the signals. For example, in Python 3.5, it's trivial to construct a mapping from the signals known by Python to their Python names:
import signal
sigdict = {sig.value: sig.name for sig in signal.Signals}
print(sigdict[1])
CTRL_BREAK_EVENT # <-- The output on my Windows box. On your machine, it would probably be SIGHUP
In older Python where the names aren't enums, you can use similar code using dir of the module, filtering for names whose __module__ is signal and whose values are integers to construct the mapping.

After starting process, how to get parent's PID in the child?

In Python, I start a new process via Popen(), which works fine. Now in the child process I want to find the parent's process ID.
What is the best way to achieve this, maybe I can pass the PID via the Popen constructor, but how? Or is there a better way to do so?
PS: If possible I would prefere a solution using only standard libraries.

You can use os.getppid():
os.getppid()
Return the parent’s process id.
Note: this works only on Unix, not on Windows. On Windows you can use os.getpid() in the parent process and pass the pid as argument to the process you start with Popen.
Windows support for os.getppid was added in Python 3.2.

Use psutil (here)
import psutil, os
psutil.Process(os.getpid()).ppid()
works both for Unix & Windows (even if os.getppid() doesn't exist on this platform)

ppid() is a member method, not variable, of Process, so the above needs to be changed to include the parenthesis.
Source: psutil documentation

If you don't want to use psutil (e.g. because your environment makes installing dependencies and IT request), here's how you can do it manually on Linux.
def get_parent_process_id(pid: int) -> int:
# Read /proc/<pid>/status and look for the line `PPid:\t120517\n`
with open(f"/proc/{pid}/status", encoding="ascii") as f:
for line in f.readlines():
if line.startswith("PPid:\t"):
return int(line[6:])
raise Exception(f"No PPid line found in /proc/{pid}/status")

Python: How to determine subprocess children have all finished running

I am trying to detect when an installation program finishes executing from within a Python script. Specifically, the application is the Oracle 10gR2 Database. Currently I am using the subprocess module with Popen. Ideally, I would simply use the wait() method to wait for the installation to finish executing, however, the documented command actually spawns child processes to handle the actual installation. Here is some sample code of the failing code:
import subprocess
OUI_DATABASE_10GR2_SUBPROCESS = ['sudo',
'-u',
'oracle',
os.path.join(DATABASE_10GR2_TMP_PATH,
'database',
'runInstaller'),
'-ignoreSysPrereqs',
'-silent',
'-noconfig',
'-responseFile '+ORACLE_DATABASE_10GR2_SILENT_RESPONSE]
oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
oracle_subprocess.wait()
There is a similar question here: Killing a subprocess including its children from python, but the selected answer does not address the children issue, instead it instructs the user to call directly the application to wait for. I am looking for a specific solution that will wait for all children of the subprocess. What if there are an unknown number of subprocesses? I will select the answer that addresses the issue of waiting for all children subprocesses to finish.
More clarity on failure: The child processes continue executing after the wait() command since that command only waits for the top level process (in this case it is 'sudo'). Here is a simple diagram of the known child processes in this problem:
Python subprocess module -> Sudo -> runInstaller -> java -> (unknown)

Ok, here is a trick that will work only under Unix. It is similar to one of the answers to this question: Ensuring subprocesses are dead on exiting Python program. The idea is to create a new process group. You can then wait for all processes in the group to terminate.
pid = os.fork()
if pid == 0:
os.setpgrp()
oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
oracle_subprocess.wait()
os._exit(0)
else:
os.waitpid(-pid)
I have not tested this. It creates an extra subprocess to be the leader of the process group, but avoiding that is (I think) quite a bit more complicated.
I found this web page to be helpful as well. http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

You can just use os.waitpid with the the pid set to -1, this will wait for all the subprocess of the current process until they finish:
import os
import sys
import subprocess
proc = subprocess.Popen([sys.executable,
'-c',
'import subprocess;'
'subprocess.Popen("sleep 5", shell=True).wait()'])
pid, status = os.waitpid(-1, 0)
print pid, status
This is the result of pstree <pid> of different subprocess forked:
python───python───sh───sleep
Hope this can help :)

Check out the following link http://www.oracle-wiki.net/startdocsruninstaller which details a flag you can use for the runInstaller command.
This flag is definitely available for 11gR2, but I have not got a 10g database to try out this flag for the runInstaller packaged with that version.
Regards

Everywhere I look seems to say it's not possible to solve this in the general case. I've whipped up a library called 'pidmon' that combines some answers for Windows and Linux and might do what you need.
I'm planning to clean this up and put it on github, possibly called 'pidmon' or something like that. I'll post a link if/when I get it up.
EDIT: It's available at http://github.com/dbarnett/python-pidmon.
I made a special waitpid function that accepts a graft_func argument so that you can loosely define what sort of processes you want to wait for when they're not direct children:
import pidmon
pidmon.waitpid(oracle_subprocess.pid, recursive=True,
graft_func=(lambda p: p.name == '???' and p.parent.pid == ???))
or, as a shotgun approach, to just wait for any processes started since the call to waitpid to stop again, do:
import pidmon
pidmon.waitpid(oracle_subprocess.pid, graft_func=(lambda p: True))
Note that this is still barely tested on Windows and seems very slow on Windows (but did I mention it's on github where it's easy to fork?). This should at least get you started, and if it works at all for you, I have plenty of ideas on how to optimize it.