I don't want to delete the temp file until the subprocess execution completes and hence, I invoke the subprocess script as:
import os
import tempfile
import subprocess
def main():
with tempfile.NamedTemporaryFile("w", delete=False) as temp:
temp.write("Hello World")
temp.flush()
print(f"Temp file is: {temp.name}")
args = ["python3",
os.path.dirname(__file__) + "/hello_world.py",
"--temp-file", temp.name]
subprocess.Popen(args)
return
main()
hello_world.py
import argparse
import sys
def print_hello():
print("Hello World")
return
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="""Test case""")
parser.add_argument('--temp-file',
required=True,
help='For test')
args = parser.parse_args()
print(args)
print_hello()
sys.exit(0)
I was hoping the temp file to be deleted once subprocess execution finishes.
Do I need to manually delete the temp file in this case?
Calling subprocess.Popen() starts the process but does not wait for it to finish.
If you want to wait for the process to finish before exiting the with block, you can use subprocess.run() instead.
Edit: Per your comment, you don't want to wait for the process to finish. Since you are creating the file with delete=False, it won't be deleted when the file pointer is closed at the end of the with block, so you will need to manually delete the path, either in the parent or child process.
Related
If I use print() function in subprocess, then subprocess will terminate as soone as the main process terminated.
The following programs terminate at the same time.
# main.py
import time
from subprocess import Popen
if __name__ == '__main__':
proc = Popen(['python', 'sub.py'])
# sub.py
for i in range(10):
time.sleep(1)
print(i)
However if I comment the print() in sub.py, then sub process continues after main terminates.
Also, If I redirect it's stdout in main.py (see following) , the sub process continues as well.
# main.py
import time
from subprocess import Popen
if __name__ == '__main__':
with open('a.txt", 'w') as out:
proc = Popen(['python', 'sub.py'],stdout=out)
I run this python file to spawn a process:
import os
import pwd
import subprocess
import sys
p = subprocess.Popen(['python', 'process_script.py'],
cwd="/execute",
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
process_script.py looks like this:
import time
import random
import string
import helper
#
def run():
while True:
filename = "/execute/" + "".join([random.choice(string.ascii_letters) for j in range(8)]) + ".txt"
helper.execute(f"echo foo > {filename}")
time.sleep(10)
#
run()
[EDIT] In fact ps shows no other processess, so it looks like the thread terminates... but how and why?
If I run process_script.py directly, the files are created.
in Popen child process dies when the parent exits you can add p.wait() at the end of your first script to prevent parent from exiting.
also this link is useful check it out!
subprocess gets killed even with nohup
I have a method which executes a command using subprocess,
I want to call that method "n" no of times and wait for all "n" subprocess to complete
for example:
import subprocess
class mysubprocess():
def child_process(self,directory):
self.process=subprocess.Popen('ls',cwd=directory)
def execute(self):
directory=['/home/suresh/Documents','/home/suresh/Downloads']
for i in directory:
print(i)
self.child_process(directory)
self.process.wait()
def main():
myobject=mysubprocess()
myobject.execute()
if __name__=='main':
main()
You need to store references to the Popen objects to call wait methods of them later.
(The code in the question overwrites the Popen object with the last Popen object, and waits only the last sub-process.)
import subprocess
class mysubprocess():
def execute(self, directory_list):
procs = []
for d in directory:
print(d)
procs.append(subprocess.Popen('ls', cwd=d)) # <---
for proc in procs:
proc.wait()
def main():
myobject = mysubprocess()
myobject.execute(['/home/suresh/Documents','/home/suresh/Downloads'])
if __name__ == '__main__':
main()
Other issues
The code is passing the entire list (directory) instead of item.
The last if statement should compare __name__ with '__main__'.
I have a simple python program:
test.py:
import time
for i in range(100000):
print i
time.sleep(0.5)
I want to use another program that executes the above one in order to read the last line output while the above program is counting.
import subprocess
process = subprocess.Popen("test",stdout=PIPE)
sleep(20) # sleeps an arbitrary time
print stdout.readlines()[-1]
The problem is that process.stdout.readlines() waits until test.py finishes execution.
Is there any way to read the last line that has been writen in the output while the program is executing?
You could use collections.deque to save only the last specified number of lines:
#!/usr/bin/env python
import collections
import subprocess
import time
import threading
def read_output(process, append):
for line in iter(process.stdout.readline, ""):
append(line)
def main():
process = subprocess.Popen(["program"], stdout=subprocess.PIPE)
# save last `number_of_lines` lines of the process output
number_of_lines = 1
q = collections.deque(maxlen=number_of_lines)
t = threading.Thread(target=read_output, args=(process, q.append))
t.daemon = True
t.start()
#
time.sleep(20)
# print saved lines
print ''.join(q),
# process is still running
# uncomment if you don't want to wait for the process to complete
##process.terminate() # if it doesn't terminate; use process.kill()
process.wait()
if __name__=="__main__":
main()
See other tail-like solutions that print only the portion of the output
See here if your child program uses a block-buffering (instead of line-bufferring) for its stdout while running non-interactively.
Fairly trivial with sh.py:
import sh
def process_line(line):
print line
process = sh.python("test.py", _out=process_line)
process.wait()
can I use Popen from python subprocess to close started process? For example, from popen I run some application. In some part of my code I have to close that ran app.
For example, from console in Linux I do:
./some_bin
... It works and logs stdout here ...
Ctrl + C and it breaks
I need something like Ctrl + C but in my program code.
from subprocess import Popen
process = Popen(['slow', 'running', 'program'])
while process.poll():
if raw_input() == 'Kill':
if process.poll(): process.kill()
kill() will kill a process. See more here: Python subprocess module
Use the subprocess module.
import subprocess
# all arguments must be passed one at a time inside a list
# they must all be string elements
arguments = ["sleep", "3600"] # first argument is the program's name
process = subprocess.Popen(arguments)
# do whatever you want
process.terminate()
Some time ago I needed a 'gentle' shutdown for a process by sending CTRL+C in Windows console.
Here's what I have:
import win32api
import win32con
import subprocess
import time
import shlex
cmdline = 'cmd.exe /k "timeout 60"'
args = shlex.split(cmdline)
myprocess = subprocess.Popen(args)
pid = myprocess.pid
print(myprocess, pid)
time.sleep(5)
win32api.GenerateConsoleCtrlEvent(win32con.CTRL_C_EVENT, pid)
# ^^^^^^^^^^^^^^^^^^^^ instead of myprocess.terminate()