Develop Reference

Command works on Command Prompt but it does not work when called with subprocess.run() or os.system() in python

Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files

When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')

How to make python script over windbg to make it's use easier?

import sys
import subprocess
command = 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64 -y ' + sys.argv[1] + ' -i ' + sys.argv[2] + ' -z ' + sys.argv[3] + ' -c "!analyze" '
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
I tried this code, I am trying to take input of crash dump name and exe location and then I have to display user understandable crash analysis ouput.How to do that using python scripting? Is it easier with cpp scripting?

take input of crash dump name and exe location and then I have to display user understandable crash analysis ouput.
It seems you want to parse the text output of the !analyze command. You can do that, but you should be aware that this command can have a lot of different output.
Let me assume you're analyzing a user mode crash dump. In such a case, I would first run a few simpler commands to check whether you got a legit dump. You may try the following commands:
|| to check the dump type (should be "user")
| to get the name of the executable (should match your application)
lmvm <app> to check the version number of your executable
If everything is fine, you can go on:
.exr -1: distinguish between a crash and a hang. A 80000003 breakpoint is more likely a hang or nothing at all.
This may help you decide if you should run !analyze or !analyze -hang.
How to do that using Python scripting?
[...] \Windows Kits\10\Debuggers\x64 -y ' + [...]
This path contains backslashes, so you want to escape them or use an r-string like r"C:\Program Files (x86)\Windows Kits\10\...".
You should probably start an executable here to make it work. cdb.exe is the command line version of WinDbg.
command.split()
This will not only split the arguments, but also the path to the exectuable. Thus subprocess.popen() will try to an application called C:\Program which does not exist.
This could fail even more often, depending on the arguments with spaces in sys.argv[].
I suggest that you pass the options as they are:
command = r'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe'
arguments = [command]
arguments.extend(['-y', sys.argv[1]]) # Symbol path
arguments.extend(['-i', sys.argv[2]]) # Image path
arguments.extend(['-z', sys.argv[3]]) # Dump file
arguments.extend(['-c', '!analyze']) # Command(s) for analysis
process = subprocess.Popen(arguments, stdout=subprocess.PIPE)
Note that there's no split() involved, which could split in wrong position.
Side note: -i may not work as expected. If you receive the crash dump from clients, they may have a different version than the one you have on disk. Set up a proper symbol server to mitigate this.
Is it easier with CPP scripting?
It will be different, not easier.
Working example
This is a Python code that considers the above. It's still a bit hacky because of the delays etc. but there's no real indicator other than time and output for deciding when a command finished. This succeeds with Python 3.8 on a crash dump of Windows Explorer.
import subprocess
import threading
import time
import re
class ReaderThread(threading.Thread):
def __init__(self, stream):
super().__init__()
self.buffer_lock = threading.Lock()
self.stream = stream # underlying stream for reading
self.output = "" # holds console output which can be retrieved by getoutput()
def run(self):
"""
Reads one from the stream line by lines and caches the result.
:return: when the underlying stream was closed.
"""
while True:
line = self.stream.readline() # readline() will block and wait for \r\n
if len(line) == 0: # this will only apply if the stream was closed. Otherwise there is always \r\n
break
with self.buffer_lock:
self.output += line
def getoutput(self, timeout=0.1):
"""
Get the console output that has been cached until now.
If there's still output incoming, it will continue waiting in 1/10 of a second until no new
output has been detected.
:return:
"""
temp = ""
while True:
time.sleep(timeout)
if self.output == temp:
break # no new output for 100 ms, assume it's complete
else:
temp = self.output
with self.buffer_lock:
temp = self.output
self.output = ""
return temp
command = r'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe'
arguments = [command]
arguments.extend(['-y', "srv*D:\debug\symbols*https://msdl.microsoft.com/download/symbols"]) # Symbol path, may use sys.argv[1]
# arguments.extend(['-i', sys.argv[2]]) # Image path
arguments.extend(['-z', sys.argv[3]]) # Dump file
arguments.extend(['-c', ".echo LOADING DONE"])
process = subprocess.Popen(arguments, stdout=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
reader = ReaderThread(process.stdout)
reader.start()
result = ""
while not re.search("LOADING DONE", result):
result = reader.getoutput() # ignore initial output
def dbg(command):
process.stdin.write(command+"\r\n")
process.stdin.flush()
return reader.getoutput()
result = dbg("||")
if "User mini" not in result:
raise Exception("Not a user mode dump")
else:
print("Yay, it's a user mode dump")
result = dbg("|")
if "explorer" not in result:
raise Exception("Not an explorer crash")
else:
print("Yay, it's an Explorer crash")
result = dbg("lm vm explorer")
if re.search(r"^\s*File version:\s*10\.0\..*$", result, re.M):
print("That's a recent version for which we should analyze crashes")
else:
raise Exception("That user should update to a newer version before we spend effort on old bugs")
dbg("q")

if you don't want to use windbg which is a gui
use cdb.exe it is console mode windbg it will output all the results to terminal
here is a demo
F:\>cdb -c "!analyze -v;qq" -z testdmp.dmp | grep -iE "bucket|owner"
DEFAULT_BUCKET_ID: BREAKPOINT
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
FOLLOWUP_NAME: MachineOwner
BUCKET_ID: BREAKPOINT_ntdll!LdrpDoDebuggerBreak+30
BUCKET_ID_IMAGE_STR: ntdll.dll
BUCKET_ID_MODULE_STR: ntdll
BUCKET_ID_FUNCTION_STR: LdrpDoDebuggerBreak
BUCKET_ID_OFFSET: 30
BUCKET_ID_MODTIMEDATESTAMP: c1bb301
BUCKET_ID_MODCHECKSUM: 1f647b
BUCKET_ID_MODVER_STR: 10.0.18362.778
BUCKET_ID_PREFIX_STR: BREAKPOINT_
FAILURE_BUCKET_ID: BREAKPOINT_80000003_ntdll.dll!LdrpDoDebuggerBreak
Followup: MachineOwner
grep is a general purpose string parser
it is built-in in Linux
it is available for windows in several places
if in 32 bit you can use it from gnuwin32 package / Cygwin
if in 64 bit you can find it in git
you can use the native findstr.exe also
:\>dir /b f:\git\usr\bin\gr*
grep.exe
groups.exe
or in msys / mingw / Cygwin / wsl / third party clones /
:\>dir /b /s *grep*.exe
F:\git\mingw64\bin\x86_64-w64-mingw32-agrep.exe
F:\git\mingw64\libexec\git-core\git-grep.exe
F:\git\usr\bin\grep.exe
F:\git\usr\bin\msggrep.exe
F:\msys64\mingw64\bin\msggrep.exe
F:\msys64\mingw64\bin\pcregrep.exe
F:\msys64\mingw64\bin\x86_64-w64-mingw32-agrep.exe
F:\msys64\usr\bin\grep.exe
F:\msys64\usr\bin\grepdiff.exe
F:\msys64\usr\bin\msggrep.exe
F:\msys64\usr\bin\pcregrep.exe
or you can write your own simple string parser in python / JavaScript / typescript / c / c++ / ruby / rust / whatever
here is a sample python word lookup and repeat script
import sys
for line in sys.stdin:
if "BUCKET" in line:
print(line)
lets check this out
:\>dir /b *.py
pyfi.py
:\>cat pyfi.py
import sys
for line in sys.stdin:
if "BUCKET" in line:
print(line)
:\>cdb -c "!analyze -v ;qq" -z f:\testdmp.dmp | python pyfi.py
DEFAULT_BUCKET_ID: BREAKPOINT
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
BUCKET_ID: BREAKPOINT_ntdll!LdrpDoDebuggerBreak+30
BUCKET_ID_IMAGE_STR: ntdll.dll
BUCKET_ID_MODULE_STR: ntdll
BUCKET_ID_FUNCTION_STR: LdrpDoDebuggerBreak
BUCKET_ID_OFFSET: 30
BUCKET_ID_MODTIMEDATESTAMP: c1bb301
BUCKET_ID_MODCHECKSUM: 1f647b
BUCKET_ID_MODVER_STR: 10.0.18362.778
BUCKET_ID_PREFIX_STR: BREAKPOINT_
FAILURE_BUCKET_ID: BREAKPOINT_80000003_ntdll.dll!LdrpDoDebuggerBreak

Running PowerShell Script from Python

I'm trying to run a Powershell Script ( check below )
First remark, my Powershell script, when running with Powershell works fine, giving the expected result ( closing all open folders ) , but I have to call this from a Python script, therefore I tried using subprocess.Popen, but I'm having the error :
"io.UnsupportedOperation: fileno"
Tried several different things already, but the solutions suggested aren't working.
I'm trying to call the following Powershell Script :
$shell = New-Object -ComObject Shell.Application
$shell.Windows() | Format-Table LocationName, LocationURL
$window = $shell.Windows()
$window | ForEach-Object { $_.Quit() }
The idea is to close all the open folders, and directly running with Powershell works as expected.
Then I tried to call this script from Python :
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'])
or
p = subprocess.Popen(['C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'])
and they don't return errors, a cmd window opens, but nothing happens.
Then I tried the following :
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'], stdout=sys.stdout)
But I have the following error ( check below )
File "<pyshell#15>", line 1, in <module>
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\FernanP\\Desktop\\TEST.ps1'], stdout=sys.stdout)
File "C:\Users\FernanP\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 667, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Users\FernanP\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 922, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
io.UnsupportedOperation: fileno
I looked into similar Questions here, and people said about the Pillow version, but I'm currently using 6.1.0, so it should not be a problem.
Therefore, either an equivalent to the powershell script directly in Python would be enough, or a way to deal with this issue.
Hope I informed sufficiently.
KR

Here I have created my own function to run any powershell script with its parameters
import subprocess # IMPORT FOR SUB PROCESS . RUN METHOD
POWERSHELL_PATH = "powershell.exe" # POWERSHELL EXE PATH
ps_script_path = "C:\\PowershellScripts\\FTP_UPLOAD.PS1" # YOUR POWERSHELL FILE PATH
class Utility: # SHARED CLASS TO USE IN OUR PROJECT
#staticmethod # STATIC METHOD DEFINITION
def run_ftp_upload_powershell_script(script_path, *params): # SCRIPT PATH = POWERSHELL SCRIPT PATH, PARAM = POWERSHELL SCRIPT PARAMETERS ( IF ANY )
commandline_options = [POWERSHELL_PATH, '-ExecutionPolicy', 'Unrestricted', script_path] # ADD POWERSHELL EXE AND EXECUTION POLICY TO COMMAND VARIABLE
for param in params: # LOOP FOR EACH PARAMETER FROM ARRAY
commandline_options.append("'" + param + "'") # APPEND YOUR FOR POWERSHELL SCRIPT
process_result = subprocess.run(commandline_options, stdout = subprocess.PIPE, stderr = subprocess.PIPE, universal_newlines = True) # CALL PROCESS
print(process_result.returncode) # PRINT RETURN CODE OF PROCESS 0 = SUCCESS, NON-ZERO = FAIL
print(process_result.stdout) # PRINT STANDARD OUTPUT FROM POWERSHELL
print(process_result.stderr) # PRINT STANDARD ERROR FROM POWERSHELL ( IF ANY OTHERWISE ITS NULL|NONE )
if process_result.returncode == 0: # COMPARING RESULT
Message = "Success !"
else:
Message = "Error Occurred !"
return Message # RETURN MESSAGE

You can use subprocess.run and need PIPE and shell. The following code worked for me:
import subprocess
result = subprocess.run([r'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe', r'C:\Users\(correct subfolders)\TEST.ps1'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
print(result)
Printing the result can give you the return value like if the command was successfully executed. If you want to extract the result value, you can do,
print(result.stdout.decode('utf-8'))

Redirecting Output From a Program to a File with Python: Specific Bug

I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')

I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code

python popen special characters not parsing file

I am having issues with non ASCII characters.
I am using python 2.7.3
python -V
Python 2.7.3
I installed http://pymediainfo.readthedocs.org/en/latest/
via
easy_install pymediainfo
imported as below
from pymediainfo import MediaInfo
media_info = MediaInfo.parse(os.path.join(path, to, file))
using this with ascii characters is fine
for debugging I printed the 'command' from:
/usr/local/lib/python2.7/dist-packages/pymediainfo-1.3.5-py2.7.egg/pymediainfo/__init__.py
the class that 'runs' the mediainfo command in /usr/local/lib/python2.7/dist-packages/pymediainfo-1.3.5-py2.7.egg/pymediainfo/init.py
ENV_DICT = {
"PATH": "/usr/local/bin/:/usr/bin/",
"LD_LIBRARY_PATH": "/usr/local/lib/:/usr/lib/"}
#staticmethod
def parse(filename, environment=ENV_DICT):
command = ["mediainfo", "-f", "--Output=XML", filename]
print command
print repr(command)
fileno_out, fname_out = mkstemp(suffix=".xml", prefix="media-")
fileno_err, fname_err = mkstemp(suffix=".err", prefix="media-")
fp_out = os.fdopen(fileno_out, 'r+b')
fp_err = os.fdopen(fileno_err, 'r+b')
p = Popen(command, stdout=fp_out, stderr=fp_err, env=environment)
p.wait()
fp_out.seek(0)
xml_dom = MediaInfo.parse_xml_data_into_dom(fp_out.read())
fp_out.close()
fp_err.close()
return MediaInfo(xml_dom)
both print and print repr() display:
['mediainfo', '-f', '--Output=XML', "/mnt/path/Long 73\xc2\xb0 58' W.avi"]
the filename is:
Long 73° 58' W.avi
Looking at a UTF-8 table \xc2\xb0 corresponds to °
I am aware this might just be just the console not interpreting the encoding as it should but the output of mediainfo is just
<?xml version="1.0" encoding="UTF-8"?>
<Mediainfo version="0.7.58">
</Mediainfo>
which means "file not found"
os.path.isfile(os.path.join(path, to, file))
returns true for these files
and in bash using
mediainfo -f --Output=XML "/path/to/file"
works
i have goggled and searched around and cannot see the answer.
Any ideas?
Thanks
UPDATE
I used this new test script
#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys
import os
import subprocess as sub
root = "/mnt/path"
for rootfldr in sorted(os.listdir(root)):
if os.path.isfile(os.path.join(root, rootfldr)):
command = ['mediainfo', '-f', '--Output=XML', rootfldr]
aa = sub.Popen(command, stdout=sub.PIPE, stderr=sub.PIPE, stdin=sub.PIPE)
result = aa.communicate()[0]
print rootfldr
print result
And the results were fine (some of then had non ASCII chars).
I then decided to change (from the pymedinfo parse function):
p = Popen(command, stdout=fp_out, stderr=fp_err, env=environment)
to
p = Popen(command, stdout=fp_out, stderr=fp_err)
and the problem was solved
I am guessing that something is missing and or wrong from
ENV_DICT = {
"PATH": "/usr/local/bin/:/usr/bin/",
"LD_LIBRARY_PATH": "/usr/local/lib/:/usr/lib/"}

The command looks ok. The filename is passed as a bytestring that contains text encoded using utf-8. If your filesystem uses utf-8 then it is correct filename:
>>> print "/mnt/path/Long 73\xc2\xb0 58' W.avi".decode('utf-8')
/mnt/path/Long 73° 58' W.avi
It might be a bug in pymediainfo. Try passing the environment argument explicitely as a workaround e.g., environment=os.environ.