In Python, I start a new process via Popen(), which works fine. Now in the child process I want to find the parent's process ID.
What is the best way to achieve this, maybe I can pass the PID via the Popen constructor, but how? Or is there a better way to do so?
PS: If possible I would prefere a solution using only standard libraries.
You can use os.getppid():
os.getppid()
Return the parent’s process id.
Note: this works only on Unix, not on Windows. On Windows you can use os.getpid() in the parent process and pass the pid as argument to the process you start with Popen.
Windows support for os.getppid was added in Python 3.2.
Use psutil (here)
import psutil, os
psutil.Process(os.getpid()).ppid()
works both for Unix & Windows (even if os.getppid() doesn't exist on this platform)
ppid() is a member method, not variable, of Process, so the above needs to be changed to include the parenthesis.
Source: psutil documentation
If you don't want to use psutil (e.g. because your environment makes installing dependencies and IT request), here's how you can do it manually on Linux.
def get_parent_process_id(pid: int) -> int:
# Read /proc/<pid>/status and look for the line `PPid:\t120517\n`
with open(f"/proc/{pid}/status", encoding="ascii") as f:
for line in f.readlines():
if line.startswith("PPid:\t"):
return int(line[6:])
raise Exception(f"No PPid line found in /proc/{pid}/status")
Related
I'm looking to implement the following command in python.
su ${USER} -c "whoami"
which simply create child process which runs under different privilege, and return the result of whoami back to the parent process.
As bash works, It should be done using os.fork() while the result is retrieved to parent process once the Chile process terminate its work.
def runWhoamiOtherUser(self):
pid = os.fork()
if pid == 0:
os.setuid(getpwnam(os.environ['USER'])[2])
x = getpass.getuser()
return x
pid, status = os.waitpid(pid, 0)
Is this the right way to do it ?
It has been added as a new feature to the subprocess module in Python 3.9, to the subprocess.Popen constructor:
New in version 3.9.
If user is not None, the setreuid() system call will be made in the
child process prior to the execution of the subprocess. If the
provided value is a string, it will be looked up via pwd.getpwnam()
and the value in pw_uid will be used. If the value is an integer, it
will be passed verbatim. (POSIX only)
https://docs.python.org/3.9/library/subprocess.html
Before 3.9, probably best to use that same setreuid() call. Same facilities are there for changing the group too, if you prefer that to user. The code was added in this commit, if you wanna check the specifics of the setting and maybe copy-paste the relevant part to your program: https://github.com/python/cpython/commit/2b2ead74382513d0bb9ef34504e283a71e6a706f
I know that I can create a new directory with the os module. But I was trying to create a new directory with the subprocess module as follows:
p=subprocess.Popen("mkdir extractions", shell=True)
os.chdir("extractions")
When the script executes, I notice that the directory extractions is created but the next os.chdir call fails saying the directory extractions does not exist. I know that I am missing something in terms of using subprocess that makes the next line unaware of the directory created. Please help!
You probably want to call p.wait() to wait for the mkdir to complete, before calling os.chdir. Or even better, use (stdout, stderr) = p.communicate(), and check the result.
Why dont you use os.mkdir("extractions")?
You could even use subprocess.call("mkdir extractions")
Both of those methods will work
After Popen, you have to do something like communicate()
p1 = subprocess.Popen('mkdir extractions', shell=True)
p1.communicate()
However, this is the same as just using subprocess.call("mkdir extractions", shell=True).
>>> import os
>>> import subprocess
>>> p=subprocess.Popen("mkdir extractions", shell=True)
>>> os.chdir("extractions")
This worked for me (on OSX). What OS are you running? Have you tried os.popen?
It will be faster to call os.mkdir, and—while the difference is unlikely to actually have any noticeable effect on your system—you are generating the overhead of creating a whole new process and running a whole separate program, and then (assuming you take the advice of any of the other [correct] answers) waiting for it to notify to tell you that it's finished instead of just calling a function in the kernel.
I am trying to detect when an installation program finishes executing from within a Python script. Specifically, the application is the Oracle 10gR2 Database. Currently I am using the subprocess module with Popen. Ideally, I would simply use the wait() method to wait for the installation to finish executing, however, the documented command actually spawns child processes to handle the actual installation. Here is some sample code of the failing code:
import subprocess
OUI_DATABASE_10GR2_SUBPROCESS = ['sudo',
'-u',
'oracle',
os.path.join(DATABASE_10GR2_TMP_PATH,
'database',
'runInstaller'),
'-ignoreSysPrereqs',
'-silent',
'-noconfig',
'-responseFile '+ORACLE_DATABASE_10GR2_SILENT_RESPONSE]
oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
oracle_subprocess.wait()
There is a similar question here: Killing a subprocess including its children from python, but the selected answer does not address the children issue, instead it instructs the user to call directly the application to wait for. I am looking for a specific solution that will wait for all children of the subprocess. What if there are an unknown number of subprocesses? I will select the answer that addresses the issue of waiting for all children subprocesses to finish.
More clarity on failure: The child processes continue executing after the wait() command since that command only waits for the top level process (in this case it is 'sudo'). Here is a simple diagram of the known child processes in this problem:
Python subprocess module -> Sudo -> runInstaller -> java -> (unknown)
Ok, here is a trick that will work only under Unix. It is similar to one of the answers to this question: Ensuring subprocesses are dead on exiting Python program. The idea is to create a new process group. You can then wait for all processes in the group to terminate.
pid = os.fork()
if pid == 0:
os.setpgrp()
oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
oracle_subprocess.wait()
os._exit(0)
else:
os.waitpid(-pid)
I have not tested this. It creates an extra subprocess to be the leader of the process group, but avoiding that is (I think) quite a bit more complicated.
I found this web page to be helpful as well. http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/
You can just use os.waitpid with the the pid set to -1, this will wait for all the subprocess of the current process until they finish:
import os
import sys
import subprocess
proc = subprocess.Popen([sys.executable,
'-c',
'import subprocess;'
'subprocess.Popen("sleep 5", shell=True).wait()'])
pid, status = os.waitpid(-1, 0)
print pid, status
This is the result of pstree <pid> of different subprocess forked:
python───python───sh───sleep
Hope this can help :)
Check out the following link http://www.oracle-wiki.net/startdocsruninstaller which details a flag you can use for the runInstaller command.
This flag is definitely available for 11gR2, but I have not got a 10g database to try out this flag for the runInstaller packaged with that version.
Regards
Everywhere I look seems to say it's not possible to solve this in the general case. I've whipped up a library called 'pidmon' that combines some answers for Windows and Linux and might do what you need.
I'm planning to clean this up and put it on github, possibly called 'pidmon' or something like that. I'll post a link if/when I get it up.
EDIT: It's available at http://github.com/dbarnett/python-pidmon.
I made a special waitpid function that accepts a graft_func argument so that you can loosely define what sort of processes you want to wait for when they're not direct children:
import pidmon
pidmon.waitpid(oracle_subprocess.pid, recursive=True,
graft_func=(lambda p: p.name == '???' and p.parent.pid == ???))
or, as a shotgun approach, to just wait for any processes started since the call to waitpid to stop again, do:
import pidmon
pidmon.waitpid(oracle_subprocess.pid, graft_func=(lambda p: True))
Note that this is still barely tested on Windows and seems very slow on Windows (but did I mention it's on github where it's easy to fork?). This should at least get you started, and if it works at all for you, I have plenty of ideas on how to optimize it.
I know about os.nice() it works perfect for parent process, but I need to do renice of my child subprocesses. I found way to do this, but it seems to be not very handy and too excessive:
os.system("renice -n %d %d" % ( new_nice, suprocess.pid ) )
And it isn't return resulting nice level after renicing.
Is there more clean way to renice subprocesses in python?
Use the preexec_fn parameter of subprocess.Popen:
If preexec_fn is set to a callable object, this object will be called in the child process just before the child is executed. (Unix only)
Example:
>>> Popen(["nice"]).communicate()
0
(None, None)
>>> Popen(["nice"], preexec_fn=lambda : os.nice(10)).communicate()
10
(None, None)
>>> Popen(["nice"], preexec_fn=lambda : os.nice(20)).communicate()
19
(None, None)
You should use subprocess.Popen instead of os.system, so you can access any results printed to sys.stdout. IIRC, os.system only gives you access to the return value, which is probably '0' and not the nice level.
renice is usually implemented by set/getpriority , which doesn't seem to have made it into the python os or posix module(yet?). So calling the renice system command seems like your best bet now.
As an alternative, you could os.nice the parent before you create a child process - which will inherit its parents nice value - and os.nice back again after you've created the child process.
without proper rights you can renice only in one way
I created a python script with a CLI in the past. You can find it here: https://github.com/jedie/python-code-snippets/blob/master/CodeSnippets/reniceall.py
renice is usually implemented by set/getpriority , which doesn't seem to have made it into the python os or posix module(yet?). So calling the renice system command seems like your best bet now.
Expanding Daniel's comment about ctypes:
from ctypes import cdll
libc = cdll.LoadLibrary("libc.so.6")
for pid in pids:
print("old priority for PID", pid, "is", libc.getpriority(0, pid))
libc.setpriority(0, pid, 20)
print("new priority for PID", pid, "is", libc.getpriority(0, pid))
Result:
old priority for PID 9721 is 0
new priority for PID 9721 is 19
If you want to change the nice of the subprocess without changing the parent, you can start a function which runs the subprocess command with the threading library.
import threading, subprocess
def run():
os.nice(5)
subprocess.run(['sha256sum', '/dev/urandom'])
thread = threading.Thread(target=run)
thread.start()
I want to call a process via a python program, however, this process need some specific environment variables that are set by another process. How can I get the first process environment variables to pass them to the second?
This is what the program look like:
import subprocess
subprocess.call(['proc1']) # this set env. variables for proc2
subprocess.call(['proc2']) # this must have env. variables set by proc1 to work
but the to process don't share the same environment. Note that these programs aren't mine (the first is big and ugly .bat file and the second a proprietary soft) so I can't modify them (ok, I can extract all that I need from the .bat but it's very combersome).
N.B.: I am using Windows, but I prefer a cross-platform solution (but my problem wouldn't happen on a Unix-like ...)
Here's an example of how you can extract environment variables from a batch or cmd file without creating a wrapper script. Enjoy.
from __future__ import print_function
import sys
import subprocess
import itertools
def validate_pair(ob):
try:
if not (len(ob) == 2):
print("Unexpected result:", ob, file=sys.stderr)
raise ValueError
except:
return False
return True
def consume(iter):
try:
while True: next(iter)
except StopIteration:
pass
def get_environment_from_batch_command(env_cmd, initial=None):
"""
Take a command (either a single command or list of arguments)
and return the environment created after running that command.
Note that if the command must be a batch file or .cmd file, or the
changes to the environment will not be captured.
If initial is supplied, it is used as the initial environment passed
to the child process.
"""
if not isinstance(env_cmd, (list, tuple)):
env_cmd = [env_cmd]
# construct the command that will alter the environment
env_cmd = subprocess.list2cmdline(env_cmd)
# create a tag so we can tell in the output when the proc is done
tag = 'Done running command'
# construct a cmd.exe command to do accomplish this
cmd = 'cmd.exe /s /c "{env_cmd} && echo "{tag}" && set"'.format(**vars())
# launch the process
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=initial)
# parse the output sent to stdout
lines = proc.stdout
# consume whatever output occurs until the tag is reached
consume(itertools.takewhile(lambda l: tag not in l, lines))
# define a way to handle each KEY=VALUE line
handle_line = lambda l: l.rstrip().split('=',1)
# parse key/values into pairs
pairs = map(handle_line, lines)
# make sure the pairs are valid
valid_pairs = filter(validate_pair, pairs)
# construct a dictionary of the pairs
result = dict(valid_pairs)
# let the process finish
proc.communicate()
return result
So to answer your question, you would create a .py file that does the following:
env = get_environment_from_batch_command('proc1')
subprocess.Popen('proc2', env=env)
As you say, processes don't share the environment - so what you literally ask is not possible, not only in Python, but with any programming language.
What you can do is to put the environment variables in a file, or in a pipe, and either
have the parent process read them, and pass them to proc2 before proc2 is created, or
have proc2 read them, and set them locally
The latter would require cooperation from proc2; the former requires that the variables become known before proc2 is started.
Since you're apparently in Windows, you need a Windows answer.
Create a wrapper batch file, eg. "run_program.bat", and run both programs:
#echo off
call proc1.bat
proc2
The script will run and set its environment variables. Both scripts run in the same interpreter (cmd.exe instance), so the variables prog1.bat sets will be set when prog2 is executed.
Not terribly pretty, but it'll work.
(Unix people, you can do the same thing in a bash script: "source file.sh".)
You can use Process in psutil to get the environment variables for that Process.
If you want to implement it yourself, you can refer to the internal implementation of psutil. It adapts to some operating system.
Currently supported operating systems are:
AIX
FreeBSD, OpenBSD, NetBSD
Linux
macOS
Sun Solaris
Windows
Eg: In Linux platform, you can find one pid 7877 environment variables in file /proc/7877/environ, just open with rt mode to read it.
Of course the best way to do this is to:
import os
from typing import Dict
from psutil import Process
process = Process(pid=os.getpid())
process_env: Dict = process.environ()
print(process_env)
You can find other platform implementation in source code
Hope I can help you.
The Python standard module multiprocessing have a Queues system that allow you to pass pickle-able object to be passed through processes. Also processes can exchange messages (a pickled object) using os.pipe. Remember that resources (e.g : database connection) and handle (e.g : file handles) can't be pickled.
You may find this link interesting :
Communication between processes with multiprocessing
Also the PyMOTw about multiprocessing worth mentioning :
multiprocessing Basics
sorry for my spelling
Two things spring to mind: (1) make the processes share the same environment, by combining them somehow into the same process, or (2) have the first process produce output that contains the relevant environment variables, that way Python can read it and construct the environment for the second process. I think (though I'm not 100% sure) that there isn't any way to get the environment from a subprocess as you're hoping to do.
Environment is inherited from the parent process. Set the environment you need in the main script, not a subprocess (child).