I have this script, it's purpose is to call an other script while with different parameters and print the output as it would be print if I called it myself :
import subprocess
def run_this(command):
print(f"running {command}")
p = subprocess.Popen(command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
retcode = p.poll()
line = p.stdout.readline()
if line:
yield line
if retcode is not None:
print(f"retcode : {retcode}")
p.stdout.close()
break
def build_command(pruned_model, prompt):
return f'python scripts/stable_txt2img.py --ddim_eta 0.0 --n_samples 1 --n_iter 4 --scale 7.0 ' \
+ f'--ddim_steps 50 --ckpt "{pruned_model}" ' \
+ f'--prompt "{prompt}" --seed 6514689'
pruned_model = r"C:\checkout2\Stable-diffusion\checkpoints\last-pruned.ckpt"
prompts = [
"a person in space",
"a person on a boat"
]
for prompt in prompts:
print("iteration")
command = build_command(pruned_model, prompt)
run_this(command)
print("done")
however the output is this :
iteration
iteration
done
Process finished with exit code 0
how is this possible? there is a print at the start of the run_this() function.
Thanks.
ps : you can pass any command to run_this(), it will never go into the function. for example, this will never print 'running toto'
import subprocess
def run_this(command):
print(f"running {command}")
p = subprocess.Popen(command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
retcode = p.poll()
line = p.stdout.readline()
if line:
yield line
if retcode is not None:
print(f"retcode : {retcode}")
p.stdout.close()
break
print("start")
run_this("toto")
print("done")
Your run_this is a generator function. Calling it doesn't actually run anything. It just creates a generator iterator. Iterating over the iterator would run the code.
Related
Consider the following snippet that runs three different subprocesses one after the other with subprocess.run (and notably all with defaulted kwargs):
import subprocess
p1 = subprocess.run(args1)
if p1.returncode != 0:
error()
p2 = subprocess.run(args2)
if p2.returncode != 0:
error()
p3 = subprocess.run(args3)
if p3.returncode != 0:
error()
How can we rewrite this so that the subprocesses are run in parallel to each other?
With Popen right? What does that exactly look like?
For reference, the implementation of subprocess.run is essentially:
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
if _mswindows:
exc.stdout, exc.stderr = process.communicate()
else:
process.wait()
raise
except:
process.kill()
raise
retcode = process.poll()
return CompletedProcess(process.args, retcode, stdout, stderr)
So something like...
with Popen(args1) as p1:
with Popen(args2) as p2:
with Popen(args3) as p3:
try:
p1.communicate(None, timeout=None)
p2.communicate(None, timeout=None)
p3.communicate(None, timeout=None)
except:
p1.kill()
p2.kill()
p3.kill()
raise
if p1.poll() != 0 or p2.poll() != 0 or p3.poll() != 0:
error()
Is that along the right lines?
I would just use multiprocessing to accomplish your mission but ensuring that your invocation of subprocess.run uses capture_output=True so that the output from the 3 commands running in parallel are not interlaced:
import multiprocessing
import subprocess
def runner(args):
p = subprocess.run(args, capture_output=True, text=True)
if p.returncode != 0:
raise Exception(r'Return code was {p.returncode}.')
return p.stdout, p.stderr
def main():
args1 = ['git', 'status']
args2 = ['git', 'log', '-3']
args3 = ['git', 'branch']
args = [args1, args2, args3]
with multiprocessing.Pool(3) as pool:
results = [pool.apply_async(runner, args=(arg,)) for arg in args]
for result in results:
try:
out, err = result.get()
print(out, end='')
except Exception as e: # runner completed with an Exception
print(e)
if __name__ == '__main__': # required for Windows
main()
Update
With just subprocess we have something like:
import subprocess
args1 = ['git', 'status']
args2 = ['git', 'log', '-3']
args3 = ['git', 'branch']
p1 = subprocess.Popen(args1)
p2 = subprocess.Popen(args2)
p3 = subprocess.Popen(args3)
p1.communicate()
rc1 = p1.returncode
p2.communicate()
rc2 = p2.returncode
p3.communicate()
rc3 = p3.returncode
But, for whatever reason on my Windows platform I never saw the output from the third subprocess command ('git branch'), so there must be some limitation there. Also, if the command you were running required input from stdin before proceeding, that input would have to be provided to the communicate method. But the communicate method would not complete until the entire subprocess has completed and you would get no parallelism, so as a general solution this is not really very good. In the multiprocessing code, there is no problem with having stdin input to communicate.
Update 2
When I recode it as follows, I now get all the expected output. I am not sure why it makes a difference, however. According to the documentation, Popen.communicate:
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate and set the returncode attribute. The optional input argument should be data to be sent to the child process, or None, if no data should be sent to the child. If streams were opened in text mode, input must be a string. Otherwise, it must be bytes.
So the call should be waiting for the process to terminate. Nevertheless, my preceding comment about the situation where the command you are executing requiring stdin input (via a pipe) would not run in parallel without using multiprocessing.
import subprocess
args1 = ['git', 'status']
args2 = ['git', 'log', '-3']
args3 = ['git', 'branch']
with subprocess.Popen(args1) as p1:
with subprocess.Popen(args2) as p2:
with subprocess.Popen(args3) as p3:
p1.communicate()
rc1 = p1.returncode
p2.communicate()
rc2 = p2.returncode
p3.communicate()
rc3 = p3.returncode
I've got a small script that's trying to execute an external command. But for some reason, the function that I made to execute the command is being completely skipped over! No errors seem to be raised, it just doesn't execute. I've got a few debug print statements inside it to verify that the function gets entered, but they never print. And I've got a print statement outside of it to verify that the script isn't dying. So what gives?
from xml.etree import ElementTree as et
import subprocess
pomFileLocation = "pom.xml"
uiAutomationCommand = "mvn clean install"
revertPomFileCommand = "git checkout pom.xml"
profileToSetToDefault = "smoketest"
def modifyxml( datafile, value ):
print( "modifying " + datafile )
tree = et.parse( datafile )
rootNodes = tree.getroot()
for node in rootNodes:
if "profiles" in node.tag:
for profile in node.iter():
foundIt = False
for param in profile.iter():
if "id" in param.tag and profileToSetToDefault in param.text:
foundIt = True
break
if foundIt == True:
for param in profile.iter():
if "activation" in param.tag:
for child in param.iter():
if "activeByDefault" in child.tag:
child.text = value
tree.write( datafile )
return
def runExternalCommand( comm ):
print( "running command " + comm )
p = subprocess.Popen( comm, bufsize=-1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE ).communicate()[0]
print( str(p) )
while( True ):
print( "still running" )
retcode = p.poll()
line = p.stdout.readline()
yield line
if( retcode is not None ):
print("Exiting")
break
return
if __name__ == '__main__':
modifyxml( pomFileLocation, "true" )
#runExternalCommand( uiAutomationCommand )
runExternalCommand( revertPomFileCommand )
print( "finished" )
runExternalCommand uses yield, so if you want it to execute all the way to the end, you ought to call it like for something in runExternalCommand(revertPomFileCommand):. Or just delete the yield line, since you don't seem to need it anyway.
def runExternalCommand( comm ):
print( "running command " + comm )
p = subprocess.Popen( comm, bufsize=-1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE ).communicate()[0]
print( str(p) )
while( True ):
print( "still running" )
retcode = p.poll()
line = p.stdout.readline()
yield line
if( retcode is not None ):
print("Exiting")
break
return
if __name__ == '__main__':
modifyxml( pomFileLocation, "true" )
#runExternalCommand( uiAutomationCommand )
for line in runExternalCommand( revertPomFileCommand ):
pass
print( "finished" )
Or
def runExternalCommand( comm ):
print( "running command " + comm )
p = subprocess.Popen( comm, bufsize=-1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE ).communicate()[0]
print( str(p) )
while( True ):
print( "still running" )
retcode = p.poll()
line = p.stdout.readline()
if( retcode is not None ):
print("Exiting")
break
return
if __name__ == '__main__':
modifyxml( pomFileLocation, "true" )
#runExternalCommand( uiAutomationCommand )
runExternalCommand( revertPomFileCommand )
print( "finished" )
As #Kevin said, the main (but not the only) issue is that runExternalCommand is a generator. To consume it, you could run: print(list(runExternalCommand(revertPomFileCommand))).
Though the function runExternalCommand() is broken: there is no point to call p.stdout.readline() after .communicate() returns (the latter waits for the child process to finish and returns the whole output at once).
It is not clear what result you want to get e.g., to run the git command and to store its output in a variable, you could use subprocess.check_output():
from subprocess import check_output, STDOUT
output = check_output("git checkout pom.xml".split(),
stderr=STDOUT, universal_newlines=True)
To discard child's stdout/stderr instead of saving it, use subprocess.check_call():
from subprocess import check_call, DEVNULL, STDOUT
check_call("git checkout pom.xml".split(),
stdout=DEVNULL, stderr=STDOUT)
For the code example, to read output while the child process is still running, see Constantly print Subprocess output while process is running.
I am starting a subprocess via python and display the stdout (progress) in a Progress bar:
def rv(args):
p = subprocess.Popen(["linkto.exe"]+[x for x in args], stdout=subprocess.PIPE)
while True:
line = p.stdout.readline()
if line != "":
progressStr=re.search(r"([0-9]+.[0-9]+%)", line.rstrip())
if progressStr == None:
print line.rstrip()
else:
progressInt=int(float(re.sub("[^0123456789\.]", "", progressStr.group())))
print progressInt
else:
break
As you see, progressInt is my cleaned up version of the stdout with integer values for the progress % - it works fine so far. However, depending on my input the stdout may vary because the subprocess may spawn another process after the primary one.
How could I drop all lines of my stdout after progressInt hits 100 for the first time?
I managed to find a solution via re.search. There was a small difference in the stdout of process1 (writes "Info:") and process2 (writes "Info [32]:").
def rv(args):
p = subprocess.Popen(["C:/Program Files/Tweak/RV-4.2.3-64/bin/rvio_hw.exe"]+[x for x in args], stdout=subprocess.PIPE)
for line in iter(p.stdout.readline,""):
noFFMpeg=re.search(r"INFO: (.*)", line.rstrip())
if noFFMpeg is not None:
progressStr=re.search(r"([0-9]+.[0-9]+%)", noFFMpeg.group())
if progressStr is not None:
progressInt=int(float(re.sub("[^0123456789\.]", "", progressStr.group())))
self.prog_QProgressBar.setValue(progressInt)
QtGui.QApplication.processEvents()
print progressStr.group()
When i run the python script ( BootScript.py ) on the shell it runs properly but when i try to run it through another script( automation.py ) it gets stuck
//automation.py
#!/usr/bin/env python
import sys
import optparse
import subprocess
global flag
failcount = 0
def incrfailcount():
global failcount
failcount += 1
def readfile():
fp = open('BootStrap.log','r')
print "press any key"
#_input()
for l in fp.readlines() :
if "BOOTSCRIPT SCORE IS: 3010" in l :
#import pdb
#pdb.set_trace()
global flag
flag = 0
fp.close()
parser = optparse.OptionParser()
parser.add_option('-c', '--count', dest='counter', help='no of time reboot Should Happen')
(options, args) = parser.parse_args()
#counter = 1
if options.counter is None:
counter = 1
else :
counter = options.counter
count = 0
output = ""
mylist = [ ' --cfgfile="BDXT0_PO_0.cfg"' , ' --cfgfile="BDXT0_PO_OVR_0.cfg"' ,' --scbypass' , ' --dmipy="C:\\sfd\\jg\\kdg\\dmi_pcie_po.py"', ' --fusestr="IA_CORE_DISABLE=0y111111111111111111111110"' , ' --fusestr="HT_DIS=1"' , ' --earbreakpy="C:\\dvfdfv\\dskf\\lsvcd\\config_restart.py"']
logfile = open('BootStrap.log', 'w')
#if out.__contains__('3010') :
#break
for i in range(int(counter)):
global flag
flag = 1
logfile = open('BootStrap.log', 'w')
proc = subprocess.Popen(['python' ,'bdxBootScript.py', mylist ], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in proc.stdout:
sys.stdout.write(line)
logfile.write(line)
proc.wait()
count = count + 1
print "file closing "
logfile.close()
readfile()
#global flag
if flag :
incrfailcount()
continue
if flag :
print "Error Occured in %d th iteration" %count
else :
print "Every thing Went well"
if failcount >= 0 :
print "Script failed %d times of total run %d " %(failcount, count)
I am trying to automate BootScript.py
**What the Program Does ?**
Here it runs BootScript.py which arguments . the output of the bootscript.py is checked for specific line (BOOTSCRIPT SCORE IS: 3010)
If present it is asumed as to sucess else failure , this script is run for counter number of times
**What i want?**
This script gets stuck for a long time , i want it to execute with out beeing sstuck , as though i am running the bootscript manually
There are several issues e.g., Popen(['python' ,'bdxBootScript.py', mylist ]) should raise an exception because you should use Popen(['python' ,'bdxBootScript.py'] + mylist) instead. If you don't see the exception then either the code is not run e.g., counter==0 or (worse) you suppress exceptions up the stack (don't do it, at the very least you should log unexpected errors).
If bdxBootScript.py doesn't produce much output then for line in proc.stdout: may appear to do nothing for some time, to fix it pass -u flag to python to make its output unbuffered and use iter(p.stdout.readline, b'') to workaround the "hidden read-ahead buffer" bug for pipes in Python 2:
import os
import sys
from subprocess import Popen, PIPE, STDOUT
with open(os.devnull, 'rb', 0) as DEVNULL:
proc = Popen([sys.executable, '-u', 'bdxBootScript.py'] + mylist,
stdin=DEVNULL, stdout=PIPE, stderr=STDOUT, bufsize=1)
for line in iter(proc.stdout.readline, b''):
sys.stdout.write(line)
sys.stdout.flush()
logfile.write(line)
logfile.flush() # make the line available in the log immediately
proc.stdout.close()
rc = proc.wait()
Here is my code:
def cmdoutput(cmd1, flag):
finish = time.time() + 50
p = subprocess.Popen(cmd1, stdin=subprocess.PIPE, stdout = subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
while p.poll() is None:
time.sleep(1)
if finish < time.time():
os.kill(p.pid, signal.SIGTERM)
print "timed out and killed child, collecting what output exists so far"
if (flag == "1"):#To enable container
out, err = p.communicate(input='container\nzone1')
else:
out, err = p.communicate()
print (out)
return out
When I run this script, I get
Attribute Error: 'module' object has no attribute 'kill'.
What's wrong with my code?
I think you have your own os.py.
Put print os.__file__ before os.kill(...) line, and you will see what's going on.
UPDATE
os.kill is only available in unix in jython
Instead of os.kill(...), use p.kill().
UPDATE
p.kill() not work. (At least in Windows + Jython 2.5.2, 2.5.3).
p.pid is None.
http://bugs.jython.org/issue1898
Change your code as follow. Change CPYTHON_EXECUTABLE_PATH, CMDOUTPUT_SCRIPT_PATH.
CPYTHON_EXECUTABLE_PATH = r'c:\python27\python.exe' # Change path to python.exe
CMDOUTPUT_SCRIPT_PATH = r'c:\users\falsetru\cmdoutput.py' # Change path to the script
def cmdoutput(cmd1, flag):
return subprocess.check_output([CPYTHON_EXECUTABLE_PATH, CMDOUTPUT_SCRIPT_PATH, flag])
Save following code as cmdoutput.py
import subprocess
import sys
def cmdoutput(cmd1, flag):
finish = time.time() + 50
p = subprocess.Popen(cmd1, stdin=subprocess.PIPE, stdout = subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
while p.poll() is None:
time.sleep(1)
if finish < time.time():
p.kill()
return '<<timeout>>'
if flag == "1":
out, err = p.communicate('container\nzone1')
else:
out, err = p.communicate()
return out
if __name__ == '__main__':
cmd, flag = sys.argv[1:3]
print(cmdoutput(cmd, flag))