Develop Reference

Command works on Command Prompt but it does not work when called with subprocess.run() or os.system() in python

Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files

When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')

How to execute a function at the end of the subprocess Popen process

Run a series of perl scripts using subprocess and specifically popen, these scripts are run on the windos cmd, these scripts (at the end of each) generate a report file. I need that as soon as the n_script execution process ends I want to execute a function created by me.
def callPerlScript(command, path):
arg = command.split(" ")
path = path + arg[0]
arg.pop(0)
perl_script = subprocess.Popen(["C:\\Perl64\\bin\\perl.exe",path] + arg, stdout=sys.stdout)
perl_script.communicate()
#now i want to execute my function after end of process
This function will be called n_times inside a for loop by changing the value of the path and maybe some arguments
path = "location/where/I/saved/the/various/scripts/"
list = [] #name of each script to execute
for i in list:
callPerlScript(i, path)

Running PowerShell Script from Python

I'm trying to run a Powershell Script ( check below )
First remark, my Powershell script, when running with Powershell works fine, giving the expected result ( closing all open folders ) , but I have to call this from a Python script, therefore I tried using subprocess.Popen, but I'm having the error :
"io.UnsupportedOperation: fileno"
Tried several different things already, but the solutions suggested aren't working.
I'm trying to call the following Powershell Script :
$shell = New-Object -ComObject Shell.Application
$shell.Windows() | Format-Table LocationName, LocationURL
$window = $shell.Windows()
$window | ForEach-Object { $_.Quit() }
The idea is to close all the open folders, and directly running with Powershell works as expected.
Then I tried to call this script from Python :
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'])
or
p = subprocess.Popen(['C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'])
and they don't return errors, a cmd window opens, but nothing happens.
Then I tried the following :
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'], stdout=sys.stdout)
But I have the following error ( check below )
File "<pyshell#15>", line 1, in <module>
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\FernanP\\Desktop\\TEST.ps1'], stdout=sys.stdout)
File "C:\Users\FernanP\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 667, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Users\FernanP\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 922, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
io.UnsupportedOperation: fileno
I looked into similar Questions here, and people said about the Pillow version, but I'm currently using 6.1.0, so it should not be a problem.
Therefore, either an equivalent to the powershell script directly in Python would be enough, or a way to deal with this issue.
Hope I informed sufficiently.
KR

Here I have created my own function to run any powershell script with its parameters
import subprocess # IMPORT FOR SUB PROCESS . RUN METHOD
POWERSHELL_PATH = "powershell.exe" # POWERSHELL EXE PATH
ps_script_path = "C:\\PowershellScripts\\FTP_UPLOAD.PS1" # YOUR POWERSHELL FILE PATH
class Utility: # SHARED CLASS TO USE IN OUR PROJECT
#staticmethod # STATIC METHOD DEFINITION
def run_ftp_upload_powershell_script(script_path, *params): # SCRIPT PATH = POWERSHELL SCRIPT PATH, PARAM = POWERSHELL SCRIPT PARAMETERS ( IF ANY )
commandline_options = [POWERSHELL_PATH, '-ExecutionPolicy', 'Unrestricted', script_path] # ADD POWERSHELL EXE AND EXECUTION POLICY TO COMMAND VARIABLE
for param in params: # LOOP FOR EACH PARAMETER FROM ARRAY
commandline_options.append("'" + param + "'") # APPEND YOUR FOR POWERSHELL SCRIPT
process_result = subprocess.run(commandline_options, stdout = subprocess.PIPE, stderr = subprocess.PIPE, universal_newlines = True) # CALL PROCESS
print(process_result.returncode) # PRINT RETURN CODE OF PROCESS 0 = SUCCESS, NON-ZERO = FAIL
print(process_result.stdout) # PRINT STANDARD OUTPUT FROM POWERSHELL
print(process_result.stderr) # PRINT STANDARD ERROR FROM POWERSHELL ( IF ANY OTHERWISE ITS NULL|NONE )
if process_result.returncode == 0: # COMPARING RESULT
Message = "Success !"
else:
Message = "Error Occurred !"
return Message # RETURN MESSAGE

You can use subprocess.run and need PIPE and shell. The following code worked for me:
import subprocess
result = subprocess.run([r'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe', r'C:\Users\(correct subfolders)\TEST.ps1'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
print(result)
Printing the result can give you the return value like if the command was successfully executed. If you want to extract the result value, you can do,
print(result.stdout.decode('utf-8'))

Redirecting Output From a Program to a File with Python: Specific Bug

I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')

I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code

running command before other finished in python

I asked already and few people gave good advises but there were to many unknowns for me as I am beginner. Therefore I decided to ask for help again without giving bad code.
I need a script which will execute copy files to directory while the other is still running.
Basically I run first command, it generates files (until user press enter) and then those files are gone (automatically removed).
What I would like to have is to copying those files (without have to press "Enter" as well).
I made in bash however I would like to achieve this on python. Please see below:
while kill -0 $! 2>/dev/null;do
cp -v /tmp/directory/* /tmp/
done

If first script is purely command line : it should be fully manageable with a python script.
General architecture :
python scripts starts first one with subprocess module
reads output from first script until it gets the message asking for pressing enter
copies all files from source directory to destination directory
sends \r into first script input
waits first script terminates
exits
General requirements :
first script must be purely CLI one
first script must write to standart output/error and read from standard input - if it reads/writes to physical terminal (/dev/tty on Unix/Linux or con: on Dos/Windows), it won't work
the end of processing must be identifiable in standard output/error
if the two above requirement were no met, the only way would be to wait a define amount of time
Optional operation :
if there are other interactions in first script (read and/or write), it will be necessary to add the redirections in the script, it is certainly feasible, but will be a little harder
Configuration :
the command to be run
the string (from command output) that indicates first program has finished processing
the source directory
the destination directory
a pattern for file name to be copied
if time defined and no identifiable string in output : the delay to wait before copying
A script like that should be simple to write and test and able to manage the first script as you want.
Edit : here is an example of such a script, still without timeout management.
import subprocess
import os
import shutil
import re
# default values for command execution - to be configured at installation
defCommand = "test.bat"
defEnd = "Appuyez"
defSource = "."
defDest = ".."
# BEWARE : pattern is in regex format !
defPattern="x.*\.txt"
class Launcher(object):
'''
Helper to launch a command, wait for a defined string from stderr or stdout
of the command, copy files from a source folder to a destination folder,
and write a newline to the stdin of the command.
Limits : use blocking IO without timeout'''
def __init__(self, command=defCommand, end=defEnd, source=defSource,
dest=defDest, pattern = defPattern):
self.command = command
self.end = end
self.source = source
self.dest = dest
self.pattern = pattern
def start(self):
'Actualy starts the command and copies the files'
found = False
pipes = os.pipe() # use explicit pipes to mix stdout and stderr
rx = re.compile(self.pattern)
cmd = subprocess.Popen(self.command, shell=True, stdin=subprocess.PIPE,
stdout=pipes[1], stderr=pipes[1])
os.close(pipes[1])
while True:
txt = os.read(pipes[0], 1024)
#print(txt) # for debug
if str(txt).find(self.end) != -1:
found = True
break
# only try to copy files if end string found
if found:
for file in os.listdir(self.source):
if rx.match(file):
shutil.copy(os.path.join(self.source, file), self.dest)
print("Copied : %s" % (file,))
# copy done : write the newline to command input
cmd.stdin.write(b"\n")
cmd.stdin.close()
try:
cmd.wait()
print("Command terminated with %d status" % (cmd.returncode,))
except:
print("Calling terminate ...")
cmd.terminate()
os.close(pipes[0])
# allows to use the file either as an imported module or directly as a script
if __name__ == '__main__':
# parse optional parameters
import argparse
parser = argparse.ArgumentParser(description='Launch a command and copy files')
parser.add_argument('--command', '-c', nargs = 1, default = defCommand,
help="full text of the command to launch")
parser.add_argument('--endString', '-e', nargs = 1, default = defEnd,
dest="end",
help="string that denotes that command has finished processing")
parser.add_argument('--source', '-s', nargs = 1, default = defSource,
help="source folder")
parser.add_argument('--dest', '-d', nargs = 1, default = defDest,
help = "destination folder")
parser.add_argument('--pattern', '-p', nargs = 1, default = defPattern,
help = "pattern (regex format) for files to be copied")
args = parser.parse_args()
# create and start a Launcher ...
launcher = Launcher(args.command, args.end, args.source, args.dest,
args.pattern)
launcher.start()