Develop Reference

How to execute a non-blocking script in python and get its return code?

I am trying to execute a non-blocking bash script from python and to get its return code. Here is my function so far:
def run_bash_script(script_fullname, logfile):
my_cmd = ". " + script_fullname + " >" + logfile +" 2>&1"
p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)
print(p.returncode)
As you can see, all the output is redirected into a log file, which I can monitor while the bash process is running.
However, the last command just returns 'None' instead of a useful exit code.
What am I doing wrong here?

You should use p.wait() rather than os.waitpid(). os.waitpid() is a low level api and it knows nothing about the Popen object so it could not touch p.

How to intercept output to windows 10 cmd.exe and modify to add color?

I'm invoking another program from the command line to create visual studio solutions and build them. This program outputs the results of those commands.
I want to print warning lines that are output in yellow text rather than the default grey and error lines in red.
Let's assume that my cmd.exe console has already been modified to support rendering ascii2 escape codes to color output.
I've done quite a bit of searching for solutions, but most of the things I've found are made for linux/osx. I did find a script that given regex as input, could replace text using the specified rules.
regex script
Is it possible for me to run this script in the background, but still connected to the cmd.exe, such that it will run on all the text that is output to the cmd.exe, to run the regex search and replace before the text is displayed in the cmd.exe window? I could put this into a batch file or python script.
I wanted to lay out the specific application, but to make this question potentially more generic, how do I apply an existing script/program to a running cmd.exe prompt in the background, such that the user can still run commands in the cmd prompt, but have the background program apply to the commands run by the user?
I'm open to trying powershell if there are no other performant viable solutions that exist.
The regular expression to detect if a line is an error just searches for the word error
"\berror\b"
It's the same search for a warning
"\bwarning\b"

Edit: Adding the better solution first. This solution sets up a Pipe so it can receive the output from the external program, then prints the colorized result in realtime.
#Python 2
from subprocess import Popen, PIPE
def invoke(command):
process = Popen(command, stdout=PIPE, bufsize=1)
with process.stdout:
#b'' is byte. Per various other SO posts, we use this method to
#iterate to workaround bugs in Python 2
for line in iter(process.stdout.readline, b''):
line = line.rstrip()
if not line:
continue
line = line.decode()
if "error" in line:
print (bcolors.FAIL + line + bcolors.ENDC)
elif "warning" in line:
print (bcolors.WARNING + line + bcolors.ENDC)
else:
print (line)
error_code = process.wait()
return error_code
To accomplish this, I pipped the output of the build command to a file. I then wrote this python script to install a required dependency, loop through the file contents, then print the data with appropriate coloring.
I will now look into a solution which colors the output in real time, as this solution requires the user to wait for the build to complete before seeing the colored output.
#Python 2
import pip
def install(package):
if hasattr(pip, 'main'):
pip.main(['install', package])
else:
pip._internal.main(['install', package])
class bcolors:
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
def print_text():
install('colorama')
try:
import colorama
colorama.init()
except:
print ("could not import colorama")
if len(sys.argv) != 2:
print ("usage: python pretty_print \"file_name\"")
return 0
else:
file_name = sys.argv[1]
with open(sys.argv[1], "r") as readfile:
for line in readfile:
line = line.rstrip()
if not line:
continue
if "error" in line:
print (bcolors.FAIL + line + bcolors.ENDC)
elif "warning" in line:
print (bcolors.WARNING + line + bcolors.ENDC)
else:
print (line)
return 0
if __name__ == "__main__":
ret = print_text()
sys.exit(ret)

Exit if the called python script encounters an error

I have a central python script that calls various other python scripts and looks like this:
os.system("python " + script1 + args1)
os.system("python " + script2 + args2)
os.system("python " + script3 + args3)
Now, I want to exit from my central script if any of the sub-scripts encounter an error.
What is happening with current code is that let's say script1 encounters an error. The console will display that error and then central script will move onto calling script2 and so on.
I want to display the encountered error and immediately exit my central code.
What is the best way to do this?

Overall this is a terrible way to execute a series of commands from within Python. However here's a minimal way to handle it:
#!python
import os, system
for script, args in some_tuple_of_commands:
exit_code = os.system("python " + script + args)
if exit_code > 0:
print("Error %d running 'python %s %s'" % (
exit_code, script, args), file=sys.stderr)
sys.exit(exit_code)
But, honestly this is all horrible. It's almost always a bad idea to concatenate strings and pass them to your shell for execution from within any programming language.
Look at the subprocess module for much more sane handling of subprocesses in Python.
Also consider trying the sh or the pexpect third party modules depending on what you're trying to do with input or output.

You can try subprocess
import subprocess,sys
try:
output = subprocess.check_output("python test.py", shell=True)
print(output)
except ValueError as e:
print e
sys.exit(0)
print("hello world")

I don't know if it's ideal for you but enclosing these commands in a function seems a good idea to me:
I am using the fact that when a process exits with error os.system(process) returns 256 else it returns 0 as an output respectively.
def runscripts():
if os.system("python " + script1 + args1):return(-1); #Returns -1 if script1 fails and exits.
if os.system("python " + script2 + args2):return(-2); #Returns -2 and exits
if os.system("python " + script3 + args3):return(-3); #Pretty obvious
return(0)
runscripts()
#or if you want to exit the main program
if runscripts():sys.exit(0)

Invoking the operating system like that is a security breach waiting to happen. One should use the subprocess module, because it is more powerful and does not invoke a shell (unless you specifically tell it to). In general, avoid invoking shell whenever possible (see this post).
You can do it like this:
import subprocess
import sys
# create a list of commands
# each command to subprocess.run must be a list of arguments, e.g.
# ["python", "echo.py", "hello"]
cmds = [("python " + script + " " + args).split()
for script, args in [(script1, args1), (script2, args2), (script3,
args3)]]
def captured_run(arglist):
"""Run a subprocess and return the output and returncode."""
proc = subprocess.run( # PIPE captures the output
arglist, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return proc.stdout, proc.stderr, proc.returncode
for cmd in cmds:
stdout, stderr, rc = captured_run(cmd)
# do whatever with stdout, stderr (note that they are bytestrings)
if rc != 0:
sys.exit(rc)
If you don't care about the output, just remove the subprocess.PIPE stuff and return only the returncode from the function. You may also want to add a timeout to the execution, see the subprocess docs linked above for how to do that.

Save output of bash command into variable inside if..else statement

I have following function:
def check_process_running(pid_name):
if subprocess.call(["pgrep", pid_name]):
print pid_name + " is not running"
else:
print pid_name + " is running and has PID="
check_process_running(sys.argv[1])
if I run the script it gives me:
$ ./test.py firefox
22977
firefox is running and has PID=
I need to get pid_num to work with the process further. I've learnt that if I want to create variable with above pid of value 22977 I can use:
tempvar = subprocess.Popen(['pgrep', sys.argv[1]], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
pid_num = tempvar.stdout.read()
print pid_num
22977
Is there solution where construction of tempvar is not needed, where the pid is picked up and saved into variable pid_num within the if..else statement as it is in my function? Or what is the most straight forward way to create the pid_num variable with just one call into the shell using subprocess and keep the function as simple as it is now?
EDIT:
With bellow solution I was able to reconstruct the statement, keep it simple and have pid_num to work with the process further:
def check_process_running(pid_name):
pid_num = subprocess.Popen(['pgrep', sys.argv[1]], stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate()[0]
if pid_num:
print pid_name + " is running and has PID=" + pid_num
else:
print pid_name + " is not running"

pid_number = subprocess.Popen(['pgrep', sys.argv[1]], stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate()[0]
maybe? or probably better
pid_number = subprocess.check_output(['pgrep', sys.argv[1]])

Python/Batch: Use Python to press any key to continue

I am using a python script to automate a process involving batch files. These are batch files that are used for other applications and I am not allowed to edit them.
At the end of the batch file, it prompts the following:
"Press any key to continue ..."
How do I use python to recognize when this prompt appears, and how do I respond to it? I want to be able to close the file so I can run the next batch file.
Currently I have found the following solution, but it's terrible and makes me feel dirty inside:
#Run the batch file with parameter DIABFile
subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile])
#Sit here like an idiot until I'm confident the batch file is finished
time.sleep(4)
#Press any key
virtual_keystrokes.press('enter')
Any ideas?
Attempt #1
p = subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile],
bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
while p.poll() is None:
line = p.stdout.readline()
print(line)
if line.startswith('Press any key to continue'):
p.communicate('\r\n')
Resulted in the following output and error:
b'\r\n'
Traceback (most recent call last):
File "C:\workspace\Perform_QAC_Check\Perform_QAC_Check.py", line 341, in <module>
main()
File "C:\workspace\Perform_QAC_Check\Perform_QAC_Check.py", line 321, in main
run_setup_builderenv(sandboxPath, DIABFile)
File "C:\workspace\Perform_QAC_Check\Perform_QAC_Check.py", line 126, in run_setup_builderenv
if line.startswith('Press any key to continue'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
The process tried to write to a nonexistent pipe.
The part that seemed weirdest to me was that the startswith first arg must be bytes or a tuple of bytes, not str. I looked up the documentation and it definitely should be a string? tutorial of startswith
So I looked online and found this little bit.
The error message seems to be a bug in Python, as it is exactly the other way around. But still, no problems here, add after line #75 in indian.py
try:
line = line.decode()
except AttributeError:
pass
And so I did.
Attempt #2
p = subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile],
bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
while p.poll() is None:
line = p.stdout.readline()
print(line)
try:
line = line.decode()
if line.startswith('Press any key to continue'):
p.communicate('\r\n')
except AttributeError:
pass
Resulted in the following output:
b'\r\n'
b'Build Environment is created.\r\n'
b'\r\n'
b'Please Refer to the directory: C:/directory\r\n'
b'\r\n'
And then it hangs there... That is the last output before the "Please press any key to continue" should show up, but it never does.
Notes
I have since taken the second script and asked it to find "Please Refer", which it does. Unfortunately, then the script hangs again at the line:
p.communicate('\r\n')
Ending the program, again, prints the error:
The process tried to write to a nonexistent pipe.
Which I believe is related to this bug.
I can't imagine what I'm trying to do is THAT out of the ordinary. Since this is seemingly a little more complicated than expected I would like to say I am using XP and Python version 3.3.

Something like the following should work:
p = subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile],
bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
while p.poll() is None:
line = p.stdout.readline()
if line.startswith('Press any key to continue'):
p.communicate('\r\n')

You could parse the output of the subprocess and match on the "Press any key to continue" phrase to continue on.
See this thread: read subprocess stdout line by line especially what he posted as Update2
It might look like this:
import subprocess
proc = subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile],stdout=subprocess.PIPE)
for line in iter(proc.stdout.readline,''):
if (line.rstrip() == "Press any key to..":
break;

As stated in the OP, none of the solutions were solving the problem. So at the end the solution from Bryce solved the problem for me:
subprocess.call([path + '\\' + batchFile, path + '\\' + DIABFile], stdin=subprocess.DEVNULL)

The solution from this post worked for me:
Try to execute cmd.exe /c YourCmdFile < nul
YourCmdFile - full path to your batch script