I'm testing a certain exe file, and I want to implement a way for my script to determine that it has entered an infinite loop.
This is my current code:
import subprocess
import os
import sys
runs = 1000 # Default run is 1000
if len(sys.argv)>1: # If I want to change the num of runs
runs = int(sys.argv[1])
FNULL = open(os.devnull, 'w')
logfile = open('logfile', 'w')
args = "exe" # Exe to test
succ = 0
fail = 0
for i in range (0,runs):
if subprocess.call(args,stdout = logfile, stderr = FNULL) == 100:
succ += 1 # If returned 100 then success
else:
fail += 1 # Else Failed
break # Break on failure
open('logfile', 'w').close() # Empties the file
print "Succ: %d , Fail: %d" % (succ, fail)
Lets say I define an infinite loop as my exe running for longer than 5 seconds.
How would i go about implementing this?
Thanks for any help, including tips on the current code!
Start a threading.Timer that will kill the process after 5 seconds and report back that the deed was done. You'll need to create and wait for the process in different steps, so use the Popen object instead of call. I created a test program that uses sleep to simulate your inifinite list.
import subprocess
import os
import sys
import threading
def on_timeout(proc, status_dict):
"""Kill process on timeout and note as status_dict['timeout']=True"""
# a container used to pass status back to calling thread
status_dict['timeout'] = True
print("timed out")
proc.kill()
runs = 1000 # Default run is 1000
if len(sys.argv)>1: # If I want to change the num of runs
runs = int(sys.argv[1])
FNULL = open(os.devnull, 'w')
logfile = open('logfile', 'w')
# replacing example with a running program. This is a simple python
# we can call from the command line.
# args = "exe" # Exe to test
test_script = "import time;import sys;time.sleep(%d);sys.exit(100)"
succ = 0
fail = 0
for i in range (0,runs):
# set by timer
status_dict = {'timeout':False}
# test prog sleeps i seconds
args = ["python", "-c", test_script % i]
proc = subprocess.Popen(args,
stdout = logfile, stderr = FNULL)
# trigger timout and kill process in 5 seconds
timer = threading.Timer(5, on_timeout, (proc, status_dict))
timer.start()
proc.wait()
# in case we didn't hit timeout
timer.cancel()
print status_dict
if not status_dict['timeout'] and proc.returncode == 100:
succ += 1 # If returned 100 then success
else:
fail += 1 # Else Failed
break # Break on failure
open('logfile', 'w').close() # Empties the file
print "Succ: %d , Fail: %d" % (succ, fail)
In python3.3 timeout was added to subprocess.call. If you are using python3.3 then you can just change your subprocess.call to have timeout as argument:
subprocess.call(args,stdout = logfile, stderr = FNULL, timeout=5)
If you are using python2.7, you can either use subprocess32 package or you need to write some extra code to handle the timeout.
If you install subprocess32 module you can use the above method of subprocess.call with timeout as argument.
Else, this code can help you achieve the same functionality:
from subprocess import Popen
timeout=5 #5 seconds
p = Popen(args, shell = True, stdout = logfile, stderr = FNULL)
while (p.poll() is None and timeout > 0):
time.sleep(1)
timeout-=1
if timeout <= 0:
p.terminate() #Timeout
This worked for me
import subprocess
import time
for _ in range(999999): # or whatever
p1 = subprocess.Popen(['exe_file', 'arg1', 'arg2'])
time.sleep(0.1) # wait a small amount of time, hopefully the process ends fast
return_code = p1.poll()
if return_code is not None: # we're happy, the process ended fast
... # do something to celebrate
else: # we're sad. let the process run for at most 5 seconds
time.sleep(5)
p1.terminate() # this kills the process. try p1.kill() too...
p1.wait() # this cleans up the process registry.
Disclaimer: this is linux code. Things could be different on windows, but you can check this out first, and then read some stuff here https://docs.python.org/2/library/subprocess.html about the subprocess library and differences between linux and windows.
Related
I'm trying to run "docker-compose pull" from inside a Python automation script and to incrementally display the same output that Docker command would print if it was run directly from the shell. This command prints a line for each Docker image found in the system, incrementally updates each line with the Docker image's download progress (a percentage) and replaces this percentage with a "done" when the download has completed. I first tried getting the command output with subprocess.poll() and (blocking) readline() calls:
import shlex
import subprocess
def run(command, shell=False):
p = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)
while True:
# print one output line
output_line = p.stdout.readline().decode('utf8')
error_output_line = p.stderr.readline().decode('utf8')
if output_line:
print(output_line.strip())
if error_output_line:
print(error_output_line.strip())
# check if process finished
return_code = p.poll()
if return_code is not None and output_line == '' and error_output_line == '':
break
if return_code > 0:
print("%s failed, error code %d" % (command, return_code))
run("docker-compose pull")
The code gets stuck in the first (blocking) readline() call. Then I tried to do the same without blocking:
import select
import shlex
import subprocess
import sys
import time
def run(command, shell=False):
p = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)
io_poller = select.poll()
io_poller.register(p.stdout.fileno(), select.POLLIN)
io_poller.register(p.stderr.fileno(), select.POLLIN)
while True:
# poll IO for output
io_events_list = []
while not io_events_list:
time.sleep(1)
io_events_list = io_poller.poll(0)
# print new output
for event in io_events_list:
# must be tested because non-registered events (eg POLLHUP) can also be returned
if event[1] & select.POLLIN:
if event[0] == p.stdout.fileno():
output_str = p.stdout.read(1).decode('utf8')
print(output_str, end="")
if event[0] == p.stderr.fileno():
error_output_str = p.stderr.read(1).decode('utf8')
print(error_output_str, end="")
# check if process finished
# when subprocess finishes, iopoller.poll(0) returns a list with 2 select.POLLHUP events
# (one for stdout, one for stderr) and does not enter in the inner loop
return_code = p.poll()
if return_code is not None:
break
if return_code > 0:
print("%s failed, error code %d" % (command, return_code))
run("docker-compose pull")
This works, but only the final lines (with "done" at the end) are printed to the screen, when all Docker images downloads have been completed.
Both methods work fine with a command with simpler output such as "ls". Maybe the problem is related with how this Docker command prints incrementally to screen, overwriting already written lines ? Is there a safe way to incrementally show the exact output of a command in the command line when running it via a Python script?
EDIT: 2nd code block was corrected
Always openSTDIN as a pipe, and if you are not using it, close it immediately.
p.stdout.read() will block until the pipe is closed, so your polling code does nothing useful here. It needs modifications.
I suggest not to use shell=True
Instead of *.readline(), try with *.read(1) and wait for "\n"
Of course you can do what you want in Python, the question is how. Because, a child process might have different ideas about how its output should look like, that's when trouble starts. E.g. the process might want explicitly a terminal at the other end, not your process. Or a lot of such simple nonsense. Also, a buffering may also cause problems. You can try starting Python in unbuffered mode to check. (/usr/bin/python -U)
If nothing works, then use pexpect automation library instead of subprocess.
I have found a solution, based on the first code block of my question:
def run(command,shell=False):
p = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)
while True:
# read one char at a time
output_line = p.stderr.read(1).decode("utf8")
if output_line != "":
print(output_line,end="")
else:
# check if process finished
return_code = p.poll()
if return_code is not None:
if return_code > 0:
raise Exception("Command %s failed" % command)
break
return return_code
Notice that docker-compose uses stderr to print its progress instead of stdout. #Dalen has explained that some applications do it when they want that their results are pipeable somewhere, for instance a file, but also want to be able to show their progress.
I'm trying to integrate ESA'2 sen2cor python-script into my workflow. To do this I create a subprocess with which I call the "L2A_Process.bat" file, which in turn calls the "L2A_Process.py" script.
I want to launch the sen2cor-script with a timeout since it gets stuck and freezes from time to time, so as to automatically re-launch it if it failed.
To launch it and catch a timeout I successfully used the following script:
import os, subprocess
from signal import CTRL_BREAK_EVENT
timeout = 3600 #1hour
l1c_safe_path = "path/to/my/input/file.SAFE" #product that I want to process
command = ["L2A_process.bat", l1c_safe_path]
p = subprocess.Popen(command,shell=False, creationflags=subprocess.CREATE_NEW_PROCESS_GROUP)
try:
p.wait(timeout)
except subprocess.TimeoutExpired:
os.kill(p.pid, CTRL_BREAK_EVENT)
This is as far as I got. It results in the sen2cor-script being paused giving the following output:
Terminate batch job (Y/N)
I'd like to know how I can properly terminate my subprocess "p" with all it own child-subprocesses (i.e. "L2A_Process.py").
Some observations:
This script needs to run on Windows;
I've tried to kill the subprocess without the creationflag I've used in the example above: this results in the subprocess being killed but the "L2A_Process.py" script deteaches an keeps running (which is the core of my problem);
I cannot use a CTRL_C_EVENT since I want to re-launch the failed "L2A_Process.py" in a loop until it succeeds.
This code works for me to monitor Sen2cor status while converting L1C to L2A for Sentinel 2 data. The Sen2cor process is time and cpu consuming so be patient. It took half an hour to create L2A with DEM, DDV, etc. Hope it helps
from subprocess import Popen, PIPE
import os
pathtoprodS1C = "path_to_L1C_product" // safe file
outdirS2A = "output_dir" // where L2A files will be placed
pathtoL2Process = "path_to_L2A_Process" //if not in path
pathtoGIPP = "path_to_your_GIPP/L2A_GIPP.xml"
procName = "./L2A_Process"
os.chdir(pathtoL2Process)
import shlex
pcall = "./{} {} --output_dir {} --tif --GIP_L2A {}".format(procName,
pathtoprodS1C,
pathtoprodS2A,
pathtoGIPP)
args = shlex.split(pcall)
print(args)
try:
p = Popen(args, stdout=PIPE)
eut = p.stdout.readline()
while eut:
eut = p.stdout.readline()
print(eut)
finally:
print('Sen2Cor is Done')
exit()
I see that there are several solutions for capturing a command output in realtime when invoked from python. I have a case like this.
run_command.py
import time
for i in range(10):
print "Count = ", i
time.sleep(1)
check_run_command.py - this one tries to capture the run_command.py output in realtime.
import subprocess
def run_command(cmd):
p = subprocess.Popen(
cmd,
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE
)
while True:
line = p.stdout.readline()
if line == '':
break
print(line.strip())
if __name__ == "__main__":
run_command("python run_command.py".split())
$ python check_run_command.py
(Waits 10 secs) then prints the following
Count = 0
Count = 1
....
Count = 9
I am not sure why I can't capture the output in realtime in this case. I tried multiple solutions in other threads for the same problem, but didn't help. Is the sleep in run_command.py has anything to do with this.
I tried running ls commands, but can't figure out if the output is printed in realtime or after the process completes, because the command itself completes quickly. Hence I added one that has sleep.
I copied a partial Python script from a benchmark scripts and tried to execute it after making modifications. The problem is, it is hanging once it is started.
The Python script is:
# !/usr/bin/python
# =============================
# initialize & configure
# =============================
#import shlex
#import fnmatch
import os
import subprocess
import resource
import os, sys, cPickle, time, threading, signal
from errno import EEXIST
from os.path import join
from subprocess import Popen
#from domain import Record
from threading import Timer
def measure(commandline):
# r,w = os.pipe()
forkedPid = os.fork()
if forkedPid: # read pickled measurements from the pipe
print "Parent proc start and the child pid: ", forkedPid
# os.close(w); rPipe = os.fdopen(r); r = cPickle.Unpickler(rPipe)
# measurements = r.load()
# rPipe.close()
os.waitpid(forkedPid,0)
print "Parent proc end"
return "------------"
else:
# Sample thread will be destroyed when the forked process _exits
class Sample(threading.Thread):
def __init__(self,program):
threading.Thread.__init__(self)
self.setDaemon(1)
self.timedout = False
self.p = program
self.maxMem = 0
self.childpids = None
self.start()
def run(self):
try:
remaining = maxtime
delay=0.01
print "Start run deman thread: ", remaining, "delay", delay
while remaining > 0:
res = resource.getrusage(resource.RUSAGE_CHILDREN)
time.sleep(delay)
remaining -= delay
print "\n res: ",res, " and remaining is: ", remaining
else:
self.timedout = True
os.kill(self.p, signal.SIGKILL)
except OSError, (e,err):
print "Error ", err
try:
print "Child proc ", commandline
# only write pickles to the pipe
# os.close(r); wPipe = os.fdopen(w, 'w'); w = cPickle.Pickler(wPipe)
start = time.time()
# print "commandLine: ", commandline, " remaining: ",remaining
# spawn the program in a separate process
p = Popen(commandline,stdout=outFile,stderr=errFile,stdin=inFile, shell=True)
# start a thread to sample the program's resident memory use
t = Sample( program = p.pid )
print "Child ......"
# wait for program exit status and resource usage
rusage = os.wait3(0)
print 'rusage: ', rusage
elapsed = time.time() - start
# m.userSysTime = rusage[2][0] + rusage[2][1]
# m.maxMem = t.rusage
# m.cpuLoad = "%"
# m.elapsed = elapsed
print "Child proc end"
except KeyboardInterrupt:
print "keyBordInterrupt..."
os.kill(p.pid, signal.SIGKILL)
except ZeroDivisionError, (e,err):
print " error ZeroDiv: "
except (OSError,ValueError), (e,err):
print " error ", e
finally:
print "Here is finally section. "
#w.dump(m)
# wPipe.close()
# Sample thread will be destroyed when the forked process _exits
os._exit(0)
if __name__ == '__main__':
print "Start now...."
#measure("jruby \/tmp/test.rb")
When I use ps -ef | grep MyAccount, I find the interpreter will hang on the first `import instruction:
MyAccount 16934 16933 0 12:08 pts/19 00:00:00 import os
/tmp/test.rb is a one-line Ruby script (puts "hello"), and it should not caused any problem as it has been commented out.
I ran ps -ef | grep MyAccount for three mintues, and this one is ALWAYS there. Also, there is not any output in the console, where I expected to see Start now.....
It's because your code is not interpreted as a Python script but as a Bash script.
The first line should be
#!/usr/bin/python
instead of
# !/usr/bin/python
You can skip this line and simply run your program using
python <your_script.py>
instead of
./<your_script.py>
I also think you read the ps results incorrectly.
The last column of ps -ef is the name of the command, definatelly not a line of the script, and the first one, MyAccount, is the name of the user.
So this ps output means that process import os is hanging. There is an application called import in some Linux distributions, see man import. If you try to execute it from the shell:
$import os
it simply waits for input forever (until interupted), and that is what happened to you.
I'm testing out a way to print out stdout from several subprocesses in Python 2.7. What I have setup is a main process that spawns, at the moment, three subprocesses and spits out their output. Each subprocess is a for-loop that goes to sleep for some random amount of time, and when it wakes up, says "Slept for X seconds".
The problem I'm seeing is that the printing out seems synchronous. Say subprocess A sleeps for 1 second, subprocess B sleeps for 3 seconds, and subprocess C sleeps for 10 seconds. The main process stops for the full 10 seconds when it's trying to see if subprocess C has something, even though the other two have probably slept and printed something out. This is to simulate if a subprocess truly has nothing to output for a longer period of time than the other two.
I need a solution which works on Windows.
My code is as follows:
main_process.py
import sys
import subprocess
logfile = open('logfile.txt', 'w')
processes = [
subprocess.Popen('python subproc_1.py', stdout=subprocess.PIPE, bufsize=1),
subprocess.Popen('python subproc_2.py', stdout=subprocess.PIPE, bufsize=1),
subprocess.Popen('python subproc_3.py', stdout=subprocess.PIPE, bufsize=1),
]
while True:
line = processes[0].stdout.readline()
if line != '':
sys.stdout.write(line)
logfile.write(line)
line = processes[1].stdout.readline()
if line != '':
sys.stdout.write(line)
logfile.write(line)
line = processes[2].stdout.readline()
if line != '':
sys.stdout.write(line)
logfile.write(line)
#If everyone is dead, break
if processes[0].poll() is not None and \
processes[1].poll() is not None and \
processes[2].poll() is not None:
break
processes[0].wait()
processes[1].wait()
print 'Done'
subproc_1.py/subproc_2.py/subproc_3.py
import time, sys, random
sleep_time = random.random() * 3
for x in range(0, 20):
print "[PROC1] Slept for {0} seconds".format(sleep_time)
sys.stdout.flush()
time.sleep(sleep_time)
sleep_time = random.random() * 3 #this is different for each subprocess.
Update: Solution
Taking the answer below along with this question, this is this should work.
import sys
import subprocess
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty # for Python 3.x
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
if __name__ == '__main__':
logfile = open('logfile.txt', 'w')
processes = [
subprocess.Popen('python subproc_1.py', stdout=subprocess.PIPE, bufsize=1),
subprocess.Popen('python subproc_2.py', stdout=subprocess.PIPE, bufsize=1),
subprocess.Popen('python subproc_3.py', stdout=subprocess.PIPE, bufsize=1),
]
q = Queue()
threads = []
for p in processes:
threads.append(Thread(target=enqueue_output, args=(p.stdout, q)))
for t in threads:
t.daemon = True
t.start()
while True:
try:
line = q.get_nowait()
except Empty:
pass
else:
sys.stdout.write(line)
logfile.write(line)
logfile.flush()
#break when all processes are done.
if all(p.poll() is not None for p in processes):
break
print 'All processes done'
I'm not sure if I need any cleanup code at the end of the while loop. If anyone has comments about it, please add them.
And each subproc script looks similar to this (I edited for the sake of making a better example):
import datetime, time, sys, random
for x in range(0, 20):
sleep_time = random.random() * 3
time.sleep(sleep_time)
timestamp = datetime.datetime.fromtimestamp(time.time()).strftime('%H%M%S.%f')
print "[{0}][PROC1] Slept for {1} seconds".format(timestamp, sleep_time)
sys.stdout.flush()
print "[{0}][PROC1] Done".format(timestamp)
sys.stdout.flush()
Your problem comes from the fact that readline() is a blocking function; if you call it on a file object and there isn't a line waiting to be read, the call won't return until there is a line of output. So what you have now will read repeatedly from subprocesses 1, 2, and 3 in that order, pausing at each until output is ready.
(Edit: The OP clarified that they're on Windows, which makes the below inapplicable. )
If you want to read from whichever output stream is ready, you need to check on the status of the streams in non-blocking fashion, using the select module, and then attempt reads only on those that are ready. select provides various ways of doing this, but for the sake of example we'll use select.select(). After starting your subprocesses, you'll have something like:
streams = [p.stdout for p in processes]
def output(s):
for f in [sys.stdout, logfile]:
f.write(s)
f.flush()
while True:
rstreams, _, _ = select.select(streams, [], [])
for stream in rstreams:
line = stream.readline()
output(line)
if all(p.poll() is not None for p in processes):
break
for stream in streams:
output(stream.read())
What select() does, when called with three lists of file objects (or file descriptors), is return three subsets of its arguments, which are the streams that are ready for reading, are ready for writing, or have an error condition. Thus on each iteration of the loop we check to see which output streams are ready to read, and iterate over just those. Then we repeat. (Note that it's important here that you're line-buffering the output; the above code assumes that if a stream is ready for reading there's at least one full line ready to be read. If you specify different buffering the above can block.)
A further problem with your original code: When you exit the loop after poll() reports all subprocesses to have exited, you might not have read all their output. So you need to do a last sweep over the streams to read any remaining output.
Note: The example code I gave doesn't try all that hard to capture the subprocesses' output in exactly the order in which it becomes available (which is impossible to do perfectly, but can be approximated more closely than the above manages to do). It also lacks other refinements (for example, in the main loop it'll continue to select on the stdout of every subprocess, even after some have already terminated, which is harmless, but inefficient). It's just meant to illustrate a basic technique of non-blocking IO.